Road environment reconstruction and obstacle prediction method and device, equipment and medium

Abstract

The embodiment of the invention relates to a road environment reconstruction and obstacle prediction method and device, equipment and a medium. The method comprises the following steps: acquiring obstacle motion information, road environment information and map information; the obstacle movement information, the road environment information and the map information are fused to obtain fusion information, and the fusion information comprises relevance characteristics among the obstacle movement information, the road environment information and the map information; road environment reconstruction operation is carried out according to the fusion information and the map information, road environment reconstruction information is obtained, and road environment reconstruction intermediate data generated in the road environment reconstruction operation process is obtained; and generating obstacle prediction information according to the fusion information, the obstacle motion information and the road environment reconstruction intermediate information, thereby solving the technical problem of low prediction efficiency caused by the fact that obstacle prediction can only be performed after road environment reconstruction is completed in the prior art. The technical effect of accurately and efficiently reconstructing the road environment and predicting the obstacle is achieved.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a method, device, equipment and medium for road environment reconstruction and obstacle prediction. Background Art

[0002] With the advancement and development of autonomous driving technology, predicting the future travel information of vehicles is a key technology. In order to improve the accuracy of the acquired driving environment, the existing technology abandons high-precision maps in the process of autonomous driving and chooses to obtain the driving environment in real time and dynamically to achieve the reconstruction of the road topology environment and the prediction of the future travel information of obstacles.

[0003] In the related art, usually the environment topology is reconstructed based on the real-time driving environment, and then obstacle prediction is performed based on the result data of the environment topology reconstruction. However, the prediction of obstacle motion information in the related art needs to be performed after the environment topology is reconstructed, which consumes a lot of operating resources and time.

[0004] This shows that related technologies have technical problems such as low efficiency in environmental topology reconstruction and obstacle motion information prediction in autonomous driving scenarios. Summary of the invention

[0005] The present application provides a road environment reconstruction and obstacle prediction method, device, equipment and medium to solve the technical problem of low efficiency of environment topology reconstruction and obstacle motion trajectory prediction in autonomous driving scenarios in the prior art.

[0006] In a first aspect, the present application provides a road environment reconstruction and obstacle prediction method, comprising: obtaining obstacle motion information, road environment information and map information; fusing the obstacle motion information, the road environment information and the map information to obtain fused information, wherein the fused information includes correlation characteristics among the obstacle motion information, the road environment information and the map information; performing a road environment reconstruction operation according to the fused information and the map information to obtain road environment reconstruction information, and obtaining road environment reconstruction intermediate data generated during the road environment reconstruction operation; generating obstacle prediction information according to the fused information, the obstacle motion information and the road environment reconstruction intermediate information.

[0007] Furthermore, the obstacle motion information, the road environment information and the map information are fused to obtain fused information, including: inputting the obstacle motion information into an obstacle motion encoder for feature extraction to obtain obstacle motion features; inputting the road environment information into a road environment encoder for feature extraction to obtain road environment features; inputting the map information into a map encoder for feature extraction to obtain map features; inputting the obstacle motion features, the road environment features and the map features into a feature fusion device for fusion to obtain fused information.

[0008] Furthermore, road environment reconstruction information is generated based on the fusion information and the map information, including: inputting the fusion information and the map information into a first sub-neural network model to obtain road environment reconstruction information; obstacle prediction information is generated based on the fusion information, the obstacle motion information and the road environment reconstruction intermediate information, including: inputting the fusion information, the obstacle motion information and the road environment reconstruction intermediate information into a second sub-neural network model to obtain obstacle prediction information.

[0009] Furthermore, the fusion information and the map information are input into the first sub-neural network model to obtain road environment reconstruction information, including: inputting the fusion information into the first sub-neural network model to obtain initial road key point information, initial road centerline information and initial road relationship information corresponding to the initial road centerline information, wherein the initial road key point information is used to indicate a preset type of road key point, the initial road centerline information is used to indicate the position of the corresponding road, and the initial road relationship information is used to characterize the positional relationship between all road centerlines indicated by the initial road centerline information; enhancing the initial road key point information according to the map information to obtain target road key point information; enhancing the initial road centerline information according to the map information to obtain target road centerline information; enhancing the initial road relationship information according to the map information to obtain target road relationship information corresponding to the target road centerline information.

[0010] Furthermore, the fusion information, the obstacle motion information and the intermediate information of the road environment reconstruction are input into the second sub-neural network model to obtain obstacle prediction information, including: inputting the fusion information, the obstacle motion information and the intermediate information of the road environment reconstruction into the second sub-neural network model; obtaining an obstacle occupancy map and an obstacle occupancy flow direction map corresponding to future moments output by the second sub-neural network model, wherein the obstacle occupancy map at future moments is used to characterize the obstacle position corresponding to each future moment in the prediction time period, and the obstacle occupancy flow direction map is used to characterize the obstacle flow direction corresponding to each future moment.

[0011] Furthermore, after the obstacle prediction information is generated according to the fusion information, the obstacle motion information and the road environment reconstruction intermediate information, it also includes: determining the pixel points contained in each first obstacle in the current obstacle occupancy map according to the obstacle motion information and the current obstacle occupancy map, and determining the obstacle ID corresponding to each first obstacle, wherein the current obstacle occupancy map is the obstacle occupancy map at the current moment; looping the following loop operations until the obstacle ID of each obstacle in each obstacle occupancy map is determined; determining the distance between each second obstacle in the next obstacle occupancy map and each first obstacle in the target obstacle occupancy map according to the target obstacle occupancy map, the next obstacle occupancy flow map and the next obstacle occupancy map wherein, when the loop operation is executed for the first time, the target obstacle occupancy map is the current obstacle occupancy map, and the next obstacle occupancy flow map is the obstacle occupancy map at the next moment of the target obstacle occupancy map; according to the corresponding relationship and the obstacle ID corresponding to each first obstacle, the obstacle ID corresponding to each second obstacle is determined; the next obstacle occupancy flow map is used as the current obstacle occupancy map, and the obstacle occupancy map at the next moment of the next obstacle occupancy flow map is used as the next obstacle occupancy flow map; according to the obstacle ID of each obstacle in each obstacle occupancy map, obstacles corresponding to the same obstacle ID in different obstacle occupancy maps are connected to obtain predicted trajectory information corresponding to each obstacle.

[0012] Furthermore, before obtaining obstacle motion information, road environment information and map information, the method also includes: obtaining initial obstacle motion information, initial road environment information and initial map information; preprocessing the initial obstacle motion information to obtain the obstacle motion information in a preset format; preprocessing the initial road environment information to obtain the road environment information in the preset format; preprocessing the initial map information to obtain the map information in the preset format; wherein the preset format includes a data format that meets the input requirements of the first sub-neural network model and the second sub-neural network model, the first sub-neural network model is used to generate the road environment reconstruction information based on the fusion information and the map information, and the second sub-neural network model is used to generate the obstacle prediction information based on the fusion information, the obstacle motion information and the road environment reconstruction intermediate information.

[0013] In the second aspect, the present application provides a road environment reconstruction and obstacle prediction device, including: an information acquisition module, used to acquire obstacle motion information, road environment information and map information; an information fusion module, used to fuse the obstacle motion information, the road environment information and the map information to obtain fusion information, wherein the fusion information includes the correlation characteristics between the obstacle motion information, the road environment information and the map information; an environment reconstruction module, used to perform a road environment reconstruction operation according to the fusion information and the map information, obtain road environment reconstruction information, and obtain road environment reconstruction intermediate data generated during the road environment reconstruction operation; an obstacle prediction module, used to generate obstacle prediction information according to the fusion information, the obstacle motion information and the road environment reconstruction intermediate information.

[0014] In a third aspect, the present application provides an electronic device comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the road environment reconstruction and obstacle prediction method as described in the first aspect.

[0015] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the road environment reconstruction and obstacle prediction method as described in the first aspect.

[0016] In a fifth aspect, the present application also provides a mobile tool, including the electronic device shown in the third aspect.

[0017] The above technical solution provided by the embodiment of the present application has the following advantages over the prior art: the embodiment of the present application obtains obstacle motion information, road environment information and map information, and ensures the diversity of reference information by simultaneously considering dynamic information and static information related to the prediction result; the obstacle motion information, road environment information and map information are fused to obtain fused information, wherein the fused information includes the correlation characteristics among the obstacle motion information, road environment information and map information, and by obtaining the correlation characteristics among the obstacle motion information, road environment information and map information, the obstacle motion information and the road environment information are matched, thereby improving the accuracy of the prediction result; a road environment reconstruction operation is performed based on the fused information and the map information to obtain the road The method comprises the following steps: obtaining the road environment reconstruction information, obtaining the intermediate data of the road environment reconstruction generated during the road environment reconstruction operation, enhancing the road environment information through the map information, and improving the accuracy of the prediction result; generating the obstacle prediction information according to the fusion information, the obstacle motion information and the intermediate information of the road environment reconstruction, and generating the obstacle prediction information according to the intermediate data, the fusion information and the obstacle motion information of the road environment reconstruction process, thereby realizing the generation of the obstacle prediction information while obtaining the road environment reconstruction information, and improving the prediction efficiency. The embodiment of the present application solves the technical problem that the prior art can only perform the obstacle prediction after the road environment reconstruction is completed, thereby causing the low prediction efficiency, and achieves the technical effect of accurately and efficiently performing the road environment reconstruction and obstacle prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0020] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0021] Figure 1 A flowchart of a method for road environment reconstruction and obstacle prediction provided in an embodiment of the present application;

[0022] Figure 2 A schematic diagram of an optional lane merging point provided in an embodiment of the present application;

[0023] Figure 3 A schematic diagram of an optional lane bifurcation point provided in an embodiment of the present application;

[0024] Figure 4 A schematic diagram of an optional intersection entry point and intersection exit point provided in an embodiment of the present application;

[0025] Figure 5 A schematic diagram of an optional curvature change point provided in an embodiment of the present application;

[0026] Figure 6 A schematic diagram of an optional lane relationship provided in an embodiment of the present application;

[0027] Figure 7 An optional obstacle occupancy map provided for an embodiment of the present application;

[0028] Figure 8 An optional obstacle occupancy flow diagram provided for an embodiment of the present application;

[0029] Fig. 9 A schematic diagram of the structure of an optional road environment reconstruction and obstacle prediction method provided in an embodiment of the present application;

[0030] Fig.10 A flowchart of an optional obstacle trajectory prediction step provided in an embodiment of the present application;

[0031] Fig.11 A structural block diagram of a road environment reconstruction and obstacle prediction device provided in an embodiment of the present application;

[0032] Fig.12 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0034] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0035] The disclosure below provides many different embodiments or examples to implement different structures of the present invention. In order to simplify the disclosure of the present invention, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present invention. In addition, the present invention can repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0036] In order to solve the technical problem of low efficiency of environmental topology reconstruction and obstacle motion information prediction in the existing autonomous driving scenario, the present application provides a road environment reconstruction and obstacle prediction method, such as Figure 1 As shown: Figure 1 A flowchart of a method for road environment reconstruction and obstacle prediction provided by an embodiment of the present application. A method for road environment reconstruction and obstacle prediction provided by an embodiment of the present application includes:

[0037] S101: Obtain obstacle motion information, road environment information and map information.

[0038] Among them, the obstacle motion information can be current obstacle motion information and historical obstacle motion information, can be obstacle trajectory information, the number of obstacles can be one or more, the obstacle motion information can include information such as the position, speed, direction, size and type of the obstacle movement, the road environment information can be the surrounding environment information detected by the sensor, or it can be labeled environment information obtained after marking according to the surrounding environment information, the label type can include lane line information, stop line, curb, passable area, crosswalk and ground signs, etc., the map information can include navigation-level maps.

[0039] Specifically, the sensor on the mobile device obtains the shape and motion trajectory of the obstacle, and generates the obstacle motion information according to the shape and motion trajectory of the obstacle. The sensor also obtains the road environment data, marks the road environment data to obtain the road environment information. The marking operation can be performed by any one of the sensors, servers or mobile devices. The map information can be obtained through mobile data or WIFI, and the locally pre-stored map information can be obtained according to the positioning information. The map information can be obtained in real time or after downloading.

[0040] S102: Fusing the obstacle motion information, the road environment information and the map information to obtain fused information, wherein the fused information includes correlation features among the obstacle motion information, the road environment information and the map information.

[0041] Among them, the obstacle motion information, road environment information and map information can be fused based on the Transformer algorithm to obtain fused information.

[0042] Specifically, the correlation between the obstacle motion information, the road environment information and the map information is first obtained. The correlation relationship can be the positional relationship of the obstacle motion trajectory mapped on the road environment. The correlation relationship can also be the environmental information corresponding to the surroundings of each obstacle. The obstacle motion information and the road environment information are organically fused according to the correlation relationship to obtain preliminary fusion features, and then the preliminary fusion features are enhanced using the map information to obtain fusion information.

[0043] S103: Perform a road environment reconstruction operation according to the fusion information and the map information to obtain road environment reconstruction information and obtain road environment reconstruction intermediate data generated during the road environment reconstruction operation.

[0044] Among them, road reconstruction operations can be performed based on the deep learning algorithm according to the fusion information and map information to obtain reconstruction information, and the intermediate information of the road environment can be set or changed by relevant personnel before or during training.

[0045] Specifically, based on the deep learning algorithm, the fused information containing all the input information features is enhanced using the map information to obtain the road environment reconstruction information. In the process of generating the road environment reconstruction information, intermediate data will be generated. The obtained intermediate data can be used for other training processes at the same time, thereby improving the training efficiency.

[0046] S104: Generate obstacle prediction information based on the fusion information, obstacle motion information and road environment reconstruction intermediate information.

[0047] Among them, obstacle prediction information can be generated based on a neural network algorithm according to fusion information, obstacle motion information and road environment reconstruction intermediate information.

[0048] It should be noted that the hierarchical data close to the input layer in the neural network algorithm will be closer to the original input data, usually in vectorized or rasterized form. The hierarchical data close to the output layer is usually represented by lines or data. The data in the middle layer can characterize the characteristics of the input data to a greater extent while ensuring the simplicity of the data volume.

[0049] Exemplarily, when the number of hidden layers of the neural network algorithm is 10, layers 3-7 may be selected as intermediate information for road environment reconstruction, and the intermediate information for road environment reconstruction may include intermediate layer data of the neural network algorithm set by relevant personnel.

[0050] The above technical solution provided by the embodiment of the present application has the following advantages over the prior art: the method provided by the embodiment of the present application obtains obstacle motion information, road environment information and map information, and ensures the diversity of reference information by simultaneously considering dynamic information and static information related to the prediction result; the obstacle motion information, road environment information and map information are fused to obtain fused information, wherein the fused information includes the correlation characteristics among the obstacle motion information, road environment information and map information, and by obtaining the correlation characteristics among the obstacle motion information, road environment information and map information, the obstacle motion information and the road environment information are matched, thereby improving the accuracy of the prediction result; the road environment reconstruction operation is performed according to the fused information and the map information, Obtain road environment reconstruction information, and obtain intermediate road environment reconstruction data generated during the road environment reconstruction operation, enhance the road environment information through map information, and improve the accuracy of the prediction result; generate obstacle prediction information according to fusion information, obstacle motion information and intermediate road environment reconstruction information, and generate obstacle prediction information according to intermediate data, fusion information and obstacle motion information in the road environment reconstruction process, thereby achieving the generation of obstacle prediction information while obtaining road environment reconstruction information, and improving prediction efficiency. The embodiment of the present application solves the technical problem that the prior art can only perform obstacle prediction after the road environment reconstruction is completed, thereby resulting in low prediction efficiency, and achieves the technical effect of accurately and efficiently performing road environment reconstruction and obstacle prediction.

[0051] In an optional embodiment, the obstacle motion information, the road environment information and the map information are fused to obtain fused information, including:

[0052] S1021: Input the obstacle motion information into the obstacle motion encoder for feature extraction to obtain obstacle motion features.

[0053] Optionally, the obstacle motion encoder may be a model for feature extraction, for example, feature extraction may be performed on obstacle motion information according to a recurrent neural network algorithm and a convolutional neural network algorithm to obtain obstacle motion features.

[0054] S1022: Input the road environment information into the road environment encoder for feature extraction to obtain road environment features.

[0055] Optionally, the road environment encoder may be a model for feature extraction, for example, feature extraction may be performed on road environment information according to a recurrent neural network algorithm or a convolutional neural network algorithm to obtain road environment features.

[0056] S1023: Input the map information into the map encoder for feature extraction to obtain map features.

[0057] Optionally, the map encoder may be a model for feature extraction, for example, feature extraction of map information may be performed according to a recurrent neural network algorithm or a convolutional neural network algorithm to obtain map features.

[0058] S1024: Input obstacle motion features, road environment features and map features into a feature fusion device for fusion to obtain fusion information.

[0059] In an optional embodiment, generating road environment reconstruction information according to the fusion information and the map information includes:

[0060] S1031: Input the fusion information and map information into the first sub-neural network model to obtain road environment reconstruction information.

[0061] Optionally, the first sub-neural network model can be an artificial neural network, a generative adversarial neural network, a convolutional neural network, a recurrent neural network, a long short-term memory neural network, a graph neural network, etc., and historical data of a preset type of road section within a preset time period is obtained and divided into a sample set and a test set to train the first sub-neural network model.

[0062] Obstacle prediction information is generated based on the fusion information, obstacle motion information and road environment reconstruction intermediate information, including:

[0063] S1032: Input the fusion information, the preset obstacle motion information and the road environment reconstruction intermediate information into the second sub-neural network model to obtain obstacle prediction information.

[0064] Optionally, the second sub-neural network model can be an artificial neural network, a generative adversarial neural network, a convolutional neural network, a recurrent neural network, a long short-term memory neural network, a graph neural network, etc., and historical data of preset types of road sections within a preset time period are obtained and divided into sample sets and test sets to train the second sub-neural network model.

[0065] In an optional embodiment, the fusion information and the map information are input into the first sub-neural network model to obtain the road environment reconstruction information, including:

[0066] S10311: Input the fusion information into the first sub-neural network model to obtain initial road key point information, initial road centerline information and initial road relationship information corresponding to the initial road centerline information, wherein the initial road key point information is used to indicate a preset type of road key point, the initial road centerline information is used to indicate the position of the corresponding road, and the initial road relationship information is used to characterize the positional relationship between all road centerlines indicated by the initial road centerline information.

[0067] Optionally, the types of road key points can be set or changed by relevant personnel.

[0068] S10312: Enhance the initial road key point information according to the map information to obtain the target road key point information.

[0069] It should be noted that since the environmental perception information that can be detected by the sensors installed on the mobile device is limited, the map information can be used to enhance the initial road key point information. Specifically, the map information is used to supplement the environmental information around the mobile device to enhance the initial road key point information.

[0070] S10313: Enhance the initial road centerline information according to the map information to obtain the target road centerline information.

[0071] It should be noted that since the environmental perception information that can be detected by the sensors installed on the mobile device is limited, the map information can be used to enhance the initial road centerline information. Specifically, the map information is used to supplement the environmental information around the mobile device to enhance the initial road centerline information.

[0072] S10314: Enhance the initial road relationship information according to the map information to obtain target road relationship information corresponding to the target road centerline information.

[0073] It should be noted that since the environmental perception information that can be detected by the sensors installed on the mobile device is limited, the map information can be used to enhance the initial road relationship information. Specifically, the map information is used to supplement the environmental information around the mobile device to enhance the initial road relationship information.

[0074] Based on the above embodiment, the present application also provides an optional embodiment, in which the initial road key point information and the target road key point information can include multiple types of road key points, such as: lane merging points, lane bifurcation points, intersection entry points, intersection exit points and curvature change points. For example, Figure 2 As shown: Figure 2A schematic diagram of an optional lane merging point provided in an embodiment of the present application, specifically, the first sub-neural network can output the lane merging point in the form of a vector, and the vector contains the coordinate information of each lane merging point, and the coordinate information can be the x and y coordinates of the lane merging point in the autonomous driving vehicle BEV coordinate system. Figure 3 As shown: Figure 3 A schematic diagram of an optional lane bifurcation point provided in an embodiment of the present application. Specifically, the first sub-neural network can output the lane bifurcation point in the form of a vector, and the vector contains the coordinate information of each lane bifurcation point. The coordinate information can be the x and y coordinates of the lane bifurcation point in the autonomous driving vehicle BEV coordinate system. Figure 4 As shown: Figure 4 A schematic diagram of an optional intersection entry point and intersection exit point provided for an embodiment of the present application, specifically, the first sub-neural network can output the intersection entry point in the form of a vector, the vector contains the coordinate information of each intersection entry point, and the coordinate information can be the x and y coordinates of the intersection exit point in the autonomous driving vehicle BEV coordinate system. The first sub-neural network can output the intersection exit point in the form of a vector, the vector contains the coordinate information of each intersection exit point, and the coordinate information can be the x and y coordinates of the intersection exit point in the autonomous driving vehicle BEV coordinate system. Figure 5 As shown: Figure 5 A schematic diagram of an optional curvature change point provided for an embodiment of the present application, specifically, the first sub-neural network can output the curvature change point in the form of a vector, and the vector contains coordinate information of each curvature change point, and the coordinate information can be the x and y coordinates of the curvature change point in the autonomous driving vehicle BEV coordinate system.

[0075] Based on the above embodiments, the present application also provides an optional embodiment, in which the initial road centerline information and the target road centerline information may include a series of ordered points, which can be represented by x and y coordinates in the autonomous driving vehicle BEV coordinate system. Each road obtained by the first sub-neural network topology corresponds to a set of ordered points, wherein the interval and number of points can be set and changed by relevant personnel according to actual conditions.

[0076] Based on the above embodiment, the present application also provides an optional embodiment, in which both the initial road relationship information and the target road relationship information may include multiple types of road relationships, such as: left neighbor lane, right neighbor lane, leading lane, following lane, reverse adjacent lane and lateral lane, etc. Figure 6 As shown: Figure 6 A schematic diagram of an optional lane relationship provided for an embodiment of the present application.

[0077] In an optional embodiment, the fusion information, the obstacle motion information and the road environment reconstruction intermediate information are input into the second sub-neural network model to obtain the obstacle prediction information, including:

[0078] S10321: Input the fusion information, obstacle motion information and road environment reconstruction intermediate information into the second sub-neural network model.

[0079] S10322: Obtain an obstacle occupancy map and an obstacle occupancy flow direction map corresponding to future moments output by the second sub-neural network model, wherein the obstacle occupancy map at future moments is used to characterize the obstacle position corresponding to each future moment in the prediction time period, and the obstacle occupancy flow direction map is used to characterize the obstacle flow direction corresponding to each future moment.

[0080] Based on the above embodiment, the present application also provides an optional embodiment, the obstacle occupancy map may include an occupancy grid map corresponding to each predicted future moment, each obstacle occupancy map corresponds to a future moment, the time interval between each adjacent future moment can be set and changed by relevant personnel, and each obstacle occupancy map uses each pixel value to indicate whether each pixel is occupied at the moment corresponding to the obstacle occupancy map. For example, Figure 7 As shown: Figure 7 An optional obstacle occupancy map is provided for an embodiment of the present application. The obstacle occupancy flow map may be a dual-channel map. Optionally, the obstacle occupancy flow map may include flow direction information for each pixel point. The flow direction information includes two values ​​contained in the position of each pixel point, which respectively represent the flow direction of the pixel point in the x and y directions pointing to the previous time point in the BEV coordinate system. It should be noted that the obstacle occupancy flow map can be used to match the position occupied by the same obstacle at different times, thereby obtaining the flow direction of the obstacle, and can also be used to obtain the movement speed of the obstacle at that moment. Optionally, the obstacle occupancy flow map may also include corresponding pixel points of the same obstacle in two adjacent obstacle occupancy maps and arrows connecting corresponding pixel points. The direction and size of the arrows can be used to indicate the direction and distance of pixel movement. Figure 8 As shown: Figure 8 An optional obstacle occupancy flow diagram is provided for an embodiment of the present application.

[0081] Based on the above embodiments, the present application also provides an optional embodiment for introducing the target model of the road environment reconstruction and obstacle prediction method of the present application. Fig. 9 A schematic diagram of an optional road environment reconstruction and obstacle prediction method provided in an embodiment of the present application is shown in FIG. Fig. 9As shown in the figure: The overall structure of the model of this embodiment is shown in the figure. There are three input information in total: obstacle motion information, road environment information, and map information. These three inputs first enter an encoder module respectively to extract high-level semantic information. Then the outputs of the three encoders are simultaneously input into a feature fusion device. The feature fusion device mainly fuses the three types of information and extracts the correlation of the related information. The features output by the feature fusion device can comprehensively express all the input information and have been organically fused. The fused features are first used to input the first sub-neural network model to reconstruct the static lane topology. In addition to receiving the fused features, the first sub-neural network module also receives navigation-level map features as enhanced information. The lane topology mainly outputs three pieces of information: target road key point information, target road centerline information, and target road relationship information. Then, the intermediate features of the first sub-neural network model are shared with the second sub-neural network module to enhance the lane and static environment constraints in the second sub-neural network module. The second sub-neural network module also receives the fused features and obstacle motion information features, and comprehensively considers the above information to predict the obstacle information. The main advantage of doing this is that it can enable the second sub-neural network module to be closely integrated with the map information and save operating resources and time. Since the second sub-neural network module must predict the future trajectory of the obstacle based on the map information, according to the existing separate solutions, it is necessary to wait until the environment reconstruction module completes the processing to obtain the environment results before making predictions. In this embodiment, since an overall model is used to jointly reconstruct and predict the environment, the environment reconstruction part can directly provide intermediate features that are more in line with the prediction requirements and save the resource overhead of intermediate processing and transmission.

[0082] Based on the above embodiments, the present application further provides an optional embodiment for introducing a method for generating an obstacle prediction trajectory according to an obstacle occupancy map and an obstacle occupancy flow map output by a second sub-neural network. Fig.10 A flowchart of an optional obstacle trajectory prediction step provided in an embodiment of the present application, such as Fig.10As shown in the figure: Let t0 be the current moment. The obstacle occupancy map at the current moment can be drawn according to the current real perception result, and it is known which obstacle each pixel belongs to. For t1 and later moments t, for any pixel position (xt, yt), if the obstacle occupancy map of the position indicates that the position is occupied by an obstacle, then according to the value of the position in the obstacle flow occupancy map, the pixel position (xt-1, yt-1) at the previous moment can be obtained. In the occupancy map at time t-1, the specific obstacle to which each pixel belongs should have been deduced based on the previous moment. Then, by querying the obstacle ID at the position (xt-1, yt-1), the obstacle corresponding to the position at time t (xt, yt) can be obtained. In this way, the obstacles corresponding to the pixel values ​​in all the occupancy grid maps starting from time t0 can be deduced, and the position of an obstacle at all future moments can be obtained. Connecting them in series forms a trajectory prediction.

[0083] In an optional embodiment, after generating obstacle prediction information according to the fusion information, the obstacle motion information and the road environment reconstruction intermediate information, the method further includes:

[0084] S105: Determine the pixel points included in each first obstacle in the current obstacle occupancy map according to the obstacle motion information and the current obstacle occupancy map, and determine the obstacle ID corresponding to each first obstacle, wherein the current obstacle occupancy map is the obstacle occupancy map at the current moment.

[0085] The obstacle ID is a unique identifier corresponding to each obstacle.

[0086] It should be noted that the pixel points contained in each obstacle in the current obstacle occupancy map can be determined based on the obstacle motion information and the current obstacle occupancy map. By comprehensively considering the obstacle historical motion information in the obstacle motion information and the current obstacle occupancy map, it is possible to accurately distinguish the repeated obstacles when there are repeated parts in the obstacles in the current obstacle occupancy map, and then divide the pixel points.

[0087] S106: Execute the following loop operation repeatedly until the obstacle ID of each obstacle in the obstacle occupation map is determined.

[0088] S107: Determine the correspondence between each second obstacle in the next obstacle occupancy map and each first obstacle in the target obstacle occupancy map according to the target obstacle occupancy map, the next obstacle occupancy flow map and the next obstacle occupancy map, wherein, in the case of executing the loop operation for the first time, the target obstacle occupancy map is the current obstacle occupancy map, and the next obstacle occupancy flow map is the obstacle occupancy map at the next moment of the target obstacle occupancy map.

[0089] Specifically, according to the target obstacle occupancy map, the next obstacle occupancy flow map and the next obstacle occupancy map, it is determined that the pixel point of each second obstacle in the next obstacle occupancy map corresponds to the pixel point of each first obstacle in the target obstacle occupancy map, and then the corresponding relationship between the first obstacle and the second obstacle is determined according to the corresponding relationship between each pixel point of each second obstacle and each pixel point of each first obstacle.

[0090] S108: Determine the obstacle ID corresponding to each second obstacle according to the corresponding relationship and the obstacle ID corresponding to each first obstacle.

[0091] Specifically, according to the corresponding relationship between the first obstacle and the second obstacle, the second obstacle is marked with the obstacle ID of the first obstacle corresponding thereto.

[0092] S109: Taking the next obstacle occupancy flow diagram as the current obstacle occupancy diagram, and taking the obstacle occupancy diagram at the next moment of the next obstacle occupancy flow diagram as the next obstacle occupancy flow diagram.

[0093] Specifically, the current obstacle occupancy map is updated using the next obstacle occupancy flow map, and the next obstacle occupancy flow map is updated using the obstacle occupancy map at the next moment of the next obstacle occupancy flow map, until all pixels in the next obstacle occupancy map do not have pixels corresponding to the current obstacle occupancy map.

[0094] S110: According to the obstacle ID of each obstacle in each obstacle occupancy map, obstacles corresponding to the same obstacle ID in different obstacle occupancy maps are connected to obtain predicted trajectory information corresponding to each obstacle.

[0095] Specifically, each obstacle occupancy map with a pixel correspondence relationship is overlapped as a different layer, and obstacles with the same ID in different layers are connected according to the displacement of each pixel point in the flow information contained in the obstacle occupancy flow map on x and y in the BEV coordinate system, and the flow trajectory of each obstacle is rendered.

[0096] In an optional embodiment, before obtaining obstacle motion information, road environment information and map information, the method further includes:

[0097] S111: Acquire initial obstacle motion information, initial road environment information and initial map information.

[0098] S112: Preprocess the initial obstacle motion information to obtain obstacle motion information in a preset format.

[0099] S113: Preprocess the initial road environment information to obtain road environment information in a preset format.

[0100] S114: Preprocess the initial map information to obtain map information in a preset format, wherein the preset format includes a data format that meets the input requirements of the first sub-neural network model and the second sub-neural network model, the first sub-neural network model is used to generate road environment reconstruction information based on the fusion information and the map information, and the second sub-neural network model is used to generate obstacle prediction information based on the fusion information, obstacle motion information and road environment reconstruction intermediate information.

[0101] Optionally, the preset format may be a data form that can be received by the neural network model, such as a rasterized input form or a vectorized input form.

[0102] Based on the same concept, the present application provides a road environment reconstruction and obstacle prediction device, such as Fig.11 As shown: Fig.11 A structural block diagram of a road environment reconstruction and obstacle prediction device provided in an embodiment of the present application. A road environment reconstruction and obstacle prediction device provided in an embodiment of the present application includes:

[0103] The information acquisition module 201 is used to acquire obstacle motion information, road environment information and map information.

[0104] The information fusion module 202 is used to fuse the obstacle motion information, the road environment information and the map information to obtain fusion information, wherein the fusion information includes correlation characteristics among the obstacle motion information, the road environment information and the map information.

[0105] The environment reconstruction module 203 is used to perform a road environment reconstruction operation according to the fusion information and the map information to obtain road environment reconstruction information and obtain road environment reconstruction intermediate data generated during the road environment reconstruction operation.

[0106] The obstacle prediction module 204 is used to generate obstacle prediction information according to the fusion information, the obstacle motion information and the road environment reconstruction intermediate information.

[0107] The above technical solution provided by the embodiment of the present application has the following advantages over the prior art: the embodiment of the present application obtains obstacle motion information, road environment information and map information, and ensures the diversity of reference information by simultaneously considering dynamic information and static information related to the prediction result; the obstacle motion information, road environment information and map information are fused to obtain fused information, wherein the fused information includes the correlation characteristics among the obstacle motion information, road environment information and map information, and by obtaining the correlation characteristics among the obstacle motion information, road environment information and map information, the obstacle motion information and the road environment information are matched, thereby improving the accuracy of the prediction result; a road environment reconstruction operation is performed based on the fused information and the map information to obtain the road The method comprises the following steps: obtaining the road environment reconstruction information, obtaining the intermediate data of the road environment reconstruction generated during the road environment reconstruction operation, enhancing the road environment information through the map information, and improving the accuracy of the prediction result; generating the obstacle prediction information according to the fusion information, the obstacle motion information and the intermediate information of the road environment reconstruction, and generating the obstacle prediction information according to the intermediate data, the fusion information and the obstacle motion information of the road environment reconstruction process, thereby realizing the generation of the obstacle prediction information while obtaining the road environment reconstruction information, and improving the prediction efficiency. The embodiment of the present application solves the technical problem that the prior art can only perform the obstacle prediction after the road environment reconstruction is completed, thereby causing the low prediction efficiency, and achieves the technical effect of accurately and efficiently performing the road environment reconstruction and obstacle prediction.

[0108] like Fig.12 As shown, an embodiment of the present application provides a device, including a processor 111, a communication interface 112, a memory 113 and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114.

[0109] The memory 113 is used to store computer programs.

[0110] In one embodiment of the present application, the processor 111 is used to execute the program stored in the memory 113 to implement the road environment reconstruction and obstacle prediction method provided by any of the above method embodiments, including:

[0111] Obtain obstacle motion information, road environment information and map information.

[0112] The obstacle motion information, the road environment information and the map information are fused to obtain fused information, wherein the fused information includes correlation features among the obstacle motion information, the road environment information and the map information.

[0113] A road environment reconstruction operation is performed according to the fusion information and the map information to obtain road environment reconstruction information and road environment reconstruction intermediate data generated during the road environment reconstruction operation.

[0114] Obstacle prediction information is generated based on the fusion information, obstacle motion information and road environment reconstruction intermediate information.

[0115] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the road environment reconstruction and obstacle prediction method provided in any of the aforementioned method embodiments are implemented.

[0116] The present application also provides a mobile tool, including the electronic device provided in the aforementioned embodiment.

[0117] Mobile tools can be all equipment with mobility, including vehicles with automatic driving or intelligent driving (including vehicles with passenger functions (such as cars, buses, large buses, minibuses, etc.), vehicles with cargo functions (such as ordinary trucks, vans, trailers, closed trucks, tank trucks, flatbed trucks, container trucks, dump trucks, trucks with special structures), special vehicles (such as logistics distribution vehicles, automatic guided transport vehicles AGV, patrol cars, cranes, cranes, excavators, bulldozers, forklifts, rollers, loaders, off-road engineering vehicles, armored engineering vehicles, sewage treatment vehicles, sanitation vehicles, vacuum trucks, floor washing vehicles, sprinkler trucks, sweeping robots, food delivery robots, shopping guide robots, lawn mowers, golf carts, etc.), vehicles with entertainment functions (such as entertainment vehicles, playground automatic driving devices, balance cars, etc.), rescue vehicles (such as fire trucks, ambulances, power repair vehicles, engineering rescue vehicles, etc.) and robots (such as sweeping robots, food delivery robots, etc.), etc.

[0118] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0119] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0120] It should be understood that the terms used herein are only for the purpose of describing specific example embodiments and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used herein may also be meant to include plural forms. The terms "include", "comprise", "contain", and "have" are inclusive, and therefore specify the existence of stated features, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not interpreted as necessarily requiring them to be performed in the specific order described or illustrated, unless the execution order is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0121] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A road environment reconstruction and obstacle prediction method, characterized in that: include: Obtain obstacle movement information, road environment information and map information; fusing the obstacle motion information, the road environment information and the map information to obtain fused information, wherein the fused information includes correlation features among the obstacle motion information, the road environment information and the map information; Performing a road environment reconstruction operation according to the fusion information and the map information to obtain road environment reconstruction information and obtain road environment reconstruction intermediate data generated during the road environment reconstruction operation; Obstacle prediction information is generated according to the fusion information, the obstacle motion information and the road environment reconstruction intermediate information.

2. The method according to claim 1, characterized in that The obstacle motion information, the road environment information and the map information are integrated to obtain integrated information, including: Inputting the obstacle motion information into an obstacle motion encoder for feature extraction to obtain obstacle motion features; Inputting the road environment information into a road environment encoder for feature extraction to obtain road environment features; Inputting the map information into a map encoder for feature extraction to obtain map features; The obstacle motion features, the road environment features and the map features are input into a feature fusion device for fusion to obtain fusion information.

3. The method according to claim 1, characterized in that Generating road environment reconstruction information according to the fusion information and the map information includes: Inputting the fusion information and the map information into a first sub-neural network model to obtain road environment reconstruction information; Generating obstacle prediction information according to the fusion information, the obstacle motion information and the road environment reconstruction intermediate information, including: The fusion information, the obstacle motion information and the road environment reconstruction intermediate information are input into the second sub-neural network model to obtain obstacle prediction information.

4. The method according to claim 3, characterized in that Inputting the fusion information and the map information into the first sub-neural network model to obtain road environment reconstruction information, including: Inputting the fusion information into the first sub-neural network model to obtain initial road key point information, initial road centerline information, and initial road relationship information corresponding to the initial road centerline information, wherein the initial road key point information is used to indicate a preset type of road key point, the initial road centerline information is used to indicate the position of the corresponding road, and the initial road relationship information is used to characterize the positional relationship between all road centerlines indicated by the initial road centerline information; Enhance the initial road key point information according to the map information to obtain target road key point information; Enhance the initial road centerline information according to the map information to obtain target road centerline information; The initial road relationship information is enhanced according to the map information to obtain target road relationship information corresponding to the target road centerline information.

5. The method according to claim 3, characterized in that: Inputting the fusion information, the obstacle motion information and the road environment reconstruction intermediate information into the second sub-neural network model to obtain obstacle prediction information, including: Inputting the fusion information, the obstacle motion information and the road environment reconstruction intermediate information into the second sub-neural network model; Obtain an obstacle occupancy map and an obstacle occupancy flow direction map corresponding to future moments output by the second sub-neural network model, wherein the obstacle occupancy map at future moments is used to characterize the obstacle positions corresponding to each future moment in the prediction time period, and the obstacle occupancy flow direction map is used to characterize the obstacle flow direction corresponding to each future moment.

6. The method according to claim 5, characterized in that After generating obstacle prediction information according to the fusion information, the obstacle motion information and the road environment reconstruction intermediate information, the method further includes: Determine, according to the obstacle motion information and the current obstacle occupancy map, the pixel points included in each first obstacle in the current obstacle occupancy map, and determine the obstacle ID corresponding to each first obstacle, wherein the current obstacle occupancy map is the obstacle occupancy map at a current moment; The following loop operation is performed repeatedly until the obstacle ID of each obstacle in the obstacle occupation map is determined; Determining a correspondence between each second obstacle in the next obstacle occupancy map and each first obstacle in the target obstacle occupancy map according to the target obstacle occupancy map, the next obstacle occupancy flow map, and the next obstacle occupancy map, wherein, in the case of performing the loop operation for the first time, the target obstacle occupancy map is the current obstacle occupancy map, and the next obstacle occupancy flow map is an obstacle occupancy map at a moment next to the target obstacle occupancy map; Determine an obstacle ID corresponding to each second obstacle according to the corresponding relationship and the obstacle ID corresponding to each first obstacle; Using the next obstacle occupancy flow diagram as the current obstacle occupancy diagram, and using the obstacle occupancy diagram at the next moment of the next obstacle occupancy flow diagram as the next obstacle occupancy flow diagram; According to the obstacle ID of each obstacle in each obstacle occupancy map, obstacles corresponding to the same obstacle ID in different obstacle occupancy maps are connected to obtain predicted trajectory information corresponding to each obstacle.

7. The method according to claim 1, characterized in that Before obtaining obstacle motion information, road environment information and map information, the method further includes: Obtaining initial obstacle motion information, initial road environment information and initial map information; Preprocessing the initial obstacle motion information to obtain the obstacle motion information in a preset format; Preprocessing the initial road environment information to obtain the road environment information in the preset format; Preprocessing the initial map information to obtain the map information in the preset format; Among them, the preset format includes a data format that meets the input requirements of the first sub-neural network model and the second sub-neural network model, the first sub-neural network model is used to generate the road environment reconstruction information based on the fusion information and the map information, and the second sub-neural network model is used to generate the obstacle prediction information based on the fusion information, the obstacle motion information and the road environment reconstruction intermediate information.

8. A road environment reconstruction and obstacle prediction device, characterized in that: include: An information acquisition module is used to obtain obstacle motion information, road environment information and map information; an information fusion module, used to fuse the obstacle motion information, the road environment information and the map information to obtain fusion information, wherein the fusion information includes correlation features among the obstacle motion information, the road environment information and the map information; An environment reconstruction module, used to perform a road environment reconstruction operation according to the fusion information and the map information, obtain road environment reconstruction information, and obtain road environment reconstruction intermediate data generated during the road environment reconstruction operation; The obstacle prediction module is used to generate obstacle prediction information according to the fusion information, the obstacle motion information and the road environment reconstruction intermediate information.

9. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the road environment reconstruction and obstacle prediction method as described in any one of claims 1-7.

10. A mobile tool, characterized in that: The electronic device comprises the electronic device as claimed in claim 9.

Images