Obstacle marking method and device and electronic equipment

Abstract

The invention provides an obstacle marking method and device and electronic equipment, and relates to the technical field of automatic driving. According to the specific implementation scheme, multiple pieces of spatial data of a target obstacle at multiple historical moments are obtained; constructing a target factor graph based on the plurality of spatial data at the plurality of historical moments and one or more constraint factors among the plurality of spatial data at the plurality of historical moments; performing optimization based on the target factor graph to obtain a plurality of optimization data at the plurality of historical moments; and on the basis of the plurality of optimization data at the plurality of historical moments, obtaining annotation information of the target obstacle at each of the plurality of historical moments.

Description

Technical Field

[0001] The present disclosure relates to the field of autonomous driving technology, and in particular to an obstacle marking method, device and electronic device. Background Art

[0002] In the field of autonomous driving, obstacle-related sample data and obstacle-related annotation information are required when training autonomous driving related models. However, in the prior art, obstacle-related annotation information is usually generated or configured by manual annotation, which has low efficiency and accuracy. Therefore, how to efficiently and accurately annotate obstacles has become a technical problem that needs to be solved. Summary of the invention

[0003] The present disclosure provides an obstacle marking method, device, equipment and storage medium.

[0004] According to one aspect of the present disclosure, a method for marking an obstacle is provided, comprising:

[0005] Obtain multiple spatial data of target obstacles at multiple historical moments;

[0006] constructing a target factor graph based on the plurality of spatial data at the plurality of historical moments and one or more constraint factors between the plurality of spatial data at the plurality of historical moments;

[0007] Performing optimization based on the target factor graph to obtain a plurality of optimization data at the plurality of historical moments;

[0008] Based on the multiple optimization data at the multiple historical moments, the labeling information of the target obstacle at each of the multiple historical moments is obtained.

[0009] According to another aspect of the present disclosure, there is provided an obstacle marking device, comprising:

[0010] A spatial data acquisition module is used to acquire multiple spatial data of target obstacles at multiple historical moments;

[0011] A factor graph construction module, used to construct a target factor graph based on the plurality of spatial data at the plurality of historical moments and one or more constraint factors between the plurality of spatial data at the plurality of historical moments;

[0012] A factor graph optimization module, used for performing optimization based on the target factor graph to obtain a plurality of optimization data at the plurality of historical moments;

[0013] The labeling information determination module is used to obtain the labeling information of the target obstacle at each of the multiple historical moments based on the multiple optimization data at the multiple historical moments.

[0014] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0015] at least one processor; and

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.

[0018] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0019] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.

[0020] By adopting the method provided in this embodiment, a target factor graph is constructed through multiple spatial data of the target obstacle at multiple historical moments and one or more constraint factors between the multiple spatial data at the multiple historical moments; optimization is performed based on the target factor graph to obtain the optimized spatial data of the target obstacle at each historical moment, and finally the optimized spatial data at each historical moment is used as the labeling information of the target obstacle at each historical moment. In this way, by optimizing the obstacle spatial data over the entire time series, the labeling information of the target obstacle at multiple historical moments over the entire time series can be efficiently obtained, and when optimizing the obstacle spatial data, the constraints between the multiple spatial data at multiple historical moments are also combined, so that the accuracy of the labeling information of the target obstacle can be guaranteed.

[0021] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0023] Figure 1 is a schematic flow chart of an obstacle marking method according to an embodiment of the present application;

[0024] Figure 2 is a schematic diagram of a scenario of a factor graph according to an embodiment of the present application;

[0025] Figure 3 is a schematic flow chart of an obstacle marking method according to another embodiment of the present application;

[0026] Figure 4 is a schematic block diagram of an obstacle marking device according to an embodiment of the present application;

[0027] Figure 5 is a block diagram of an electronic device used to implement the obstacle labeling method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0029] Figure 1 is a schematic flow chart of the obstacle marking method proposed in the embodiment of the present disclosure, including:

[0030] S110, obtaining multiple spatial data of the target obstacle at multiple historical moments;

[0031] S120, constructing a target factor graph based on the plurality of spatial data at the plurality of historical moments and one or more constraint factors between the plurality of spatial data at the plurality of historical moments;

[0032] S130, performing optimization based on the target factor graph to obtain a plurality of optimization data at the plurality of historical moments;

[0033] S140, based on the multiple optimization data at the multiple historical moments, obtain the labeling information of the target obstacle at each of the multiple historical moments.

[0034] The obstacle annotation of the embodiment of the present disclosure may be performed by an electronic device. The electronic device may be a terminal or a server. Exemplarily, the electronic device may be a terminal device with computing capabilities. Exemplarily, the electronic device may be a server in the cloud; the server may be a single server, or may be at least one server in a server cluster, or may be a server of a distributed system (or may be referred to as a computing node).

[0035] By adopting the method provided in this embodiment, a target factor graph is constructed through multiple spatial data of the target obstacle at multiple historical moments and one or more constraint factors between the multiple spatial data at the multiple historical moments; optimization is performed based on the target factor graph to obtain the optimized spatial data of the target obstacle at each historical moment, and finally the optimized spatial data at each historical moment is used as the labeling information of the target obstacle at each historical moment. In this way, by optimizing the obstacle spatial data over the entire time series, the labeling information of the target obstacle at multiple historical moments over the entire time series can be efficiently obtained, and when optimizing the obstacle spatial data, the constraints between the multiple spatial data at multiple historical moments are also combined, so that the accuracy of the labeling information of the target obstacle can be guaranteed.

[0036] In some possible implementations, the acquiring multiple spatial data of the target obstacle at multiple historical moments includes: acquiring multiple original data of the target obstacle at multiple historical moments; obtaining multiple predicted data of the target obstacle at multiple historical moments based on the multiple original data of the target obstacle at multiple historical moments; and generating multiple spatial data of the target obstacle at multiple historical moments based on the multiple predicted data of the target obstacle at multiple historical moments.

[0037] The acquiring of multiple raw data of the target obstacle at multiple historical moments may include: acquiring one or more raw data of the target obstacle collected by the target vehicle at each of the multiple historical moments. The target vehicle may be an autonomous driving vehicle or a non-autonomous driving vehicle.

[0038] Among them, the one or more original data of the target obstacle collected by the target vehicle at any historical moment among the multiple historical moments include at least one of the following: images of one or more obstacles collected by the target vehicle at any historical moment among the multiple historical moments; point cloud data of one or more obstacles collected by the target vehicle at any historical moment among the multiple historical moments, wherein the one or more obstacles include the target obstacle. The image can be collected by a camera installed on the target vehicle. The point cloud data can be collected by a laser radar installed on the target vehicle. It should be understood that the above-mentioned collection method is only an exemplary description, and the image and / or point cloud data can also be collected by other collection devices installed on the target vehicle, which is not limited or exhaustive in this application.

[0039] The multiple prediction data of the target obstacle at multiple historical moments may be: one or more prediction data of the target obstacle at each of the multiple historical moments. The one or more prediction data may include at least one of the following: one or more bounding boxes, speed, and detected posture. The one or more bounding boxes include: a two-dimensional bounding box and / or a three-dimensional bounding box. The posture includes position and orientation.

[0040] The obtaining, based on multiple original data of the target obstacle at multiple historical moments, multiple predicted data of the target obstacle at multiple historical moments may include: determining a three-dimensional bounding box of the target obstacle at each historical moment based on point cloud data of the target obstacle at each historical moment among the multiple historical moments; determining a two-dimensional bounding box of the target obstacle at each historical moment based on an image of the target obstacle at each historical moment among the multiple historical moments; obtaining a speed and a detected posture of the target obstacle at each historical moment based on the three-dimensional bounding box of the target obstacle at each historical moment; and associating the three-dimensional bounding box of the target obstacle at each historical moment with the two-dimensional bounding box.

[0041] Based on the image of the target obstacle at each historical moment among the multiple historical moments, determining the two-dimensional bounding box of the target obstacle at each historical moment can be: inputting the image of the target obstacle at the i-th historical moment into a two-dimensional bounding box detection model, and obtaining the two-dimensional bounding box of the target obstacle at the i-th historical moment output by the two-dimensional bounding box detection model, wherein i is a positive integer, and the i-th historical moment is one of the multiple historical moments.

[0042] Based on the point cloud data of the target obstacle at each historical moment among the multiple historical moments, determining the three-dimensional bounding box of the target obstacle at each historical moment can be: inputting the point cloud data of the target obstacle at the i-th historical moment into the three-dimensional bounding box detection model, and obtaining the three-dimensional bounding box of the target obstacle at the i-th historical moment output by the three-dimensional bounding box detection model.

[0043] The method for determining the three-dimensional bounding box and the two-dimensional bounding box at each historical moment is the same as that at the i-th historical moment, and will not be described one by one here. The order of determining the three-dimensional bounding box and the two-dimensional bounding box at each historical moment may be to first determine the three-dimensional bounding box and then determine the two-dimensional bounding box, or to first determine the two-dimensional bounding box and then determine the three-dimensional bounding box, which is not limited in this embodiment.

[0044] The two-dimensional bounding box detection model and the three-dimensional bounding box detection model can be configured according to actual conditions, and their training methods are not limited in this embodiment.

[0045] The position and posture of the target obstacle at each historical moment may include: the real position and posture of the target obstacle at each historical moment, and / or the detected position and posture of the target obstacle at each historical moment.

[0046] Optionally, the method of obtaining the detection posture of the target obstacle at each historical moment may include: converting the two-dimensional bounding box of the target obstacle at the i-th historical moment into a specified coordinate system to obtain the two-dimensional bounding box of the target obstacle in the specified coordinate system at the i-th historical moment; using the position and orientation of the two-dimensional bounding box of the target obstacle in the specified coordinate system at the i-th historical moment as the detection position and initial detection orientation of the target obstacle at the i-th historical moment, and using the detection position and initial detection orientation of the target obstacle at the i-th historical moment as the detection posture. Wherein, the orientation of the two-dimensional bounding box may be: the direction of the main axis of the two-dimensional bounding box. For example, in the case where the two-dimensional bounding box is a rectangle, the main axis of the two-dimensional bounding box may be the direction of the axis corresponding to the long side of the two-dimensional bounding box (i.e., any direction of the axis parallel to the long side). It can be understood that this is only an exemplary description, and the specific setting method can be configured according to actual conditions.

[0047] The designated coordinate system may be configured according to actual requirements or circumstances, for example, the designated coordinate system may be a world coordinate system.

[0048] The converting the two-dimensional bounding box of the target obstacle at the i-th historical moment into the designated coordinate system to obtain the two-dimensional bounding box of the target obstacle in the designated coordinate system at the i-th historical moment includes: based on the position of the target vehicle at the i-th historical moment among the multiple historical moments, converting the two-dimensional bounding box of the target obstacle at the i-th historical moment among the multiple historical moments from the image coordinate system to the designated coordinate system to obtain the two-dimensional bounding box of the target obstacle in the designated coordinate system at the i-th historical moment. The position of the target vehicle at each historical moment can be obtained by a position sensor of the target vehicle itself, which is not limited in this embodiment.

[0049] Optionally, obtaining the speed and detection posture of the target obstacle at each historical moment based on the three-dimensional bounding box of the target obstacle at each historical moment includes: obtaining the detection position and initial detection orientation of the target obstacle at each historical moment based on the three-dimensional bounding box of the target obstacle at each historical moment; obtaining the speed of the target obstacle at each historical moment based on the detection position of the target obstacle at each historical moment; adjusting the initial detection orientation of the target obstacle at each historical moment based on the speed of the target obstacle at each historical moment to obtain the target detection orientation of the target obstacle at each historical moment; and using the detection position and target detection orientation of the target obstacle at each historical moment as the detection posture of the target obstacle at each historical moment.

[0050] Based on the three-dimensional bounding box of the target obstacle at each historical moment, obtaining the detection position and initial detection orientation of the target obstacle at each historical moment may include: converting the three-dimensional bounding box of the target obstacle at the i-th historical moment into a specified coordinate system to obtain the three-dimensional bounding box of the target obstacle in the specified coordinate system at the i-th historical moment; converting the three-dimensional bounding box of the target obstacle at the i-th historical moment into a specified coordinate system to obtain the three-dimensional bounding box of the target obstacle in the specified coordinate system at the i-th historical moment; using the position and orientation of the three-dimensional bounding box of the target obstacle in the specified coordinate system at the i-th historical moment as the detection position and initial detection orientation of the target obstacle at the i-th historical moment. Wherein, the orientation of the three-dimensional bounding box may be: the direction of the main axis of the three-dimensional bounding box. For example: in the case where the three-dimensional bounding box is a cuboid, the main axis of the three-dimensional bounding box may be the direction of the axis corresponding to the long side of the three-dimensional bounding box (i.e., any direction of the axis parallel to the long side). It can be understood that this is only an exemplary description, and the specific setting method can be configured according to actual conditions.

[0051] The three-dimensional bounding box of the target obstacle at the i-th historical moment is converted into a designated coordinate system to obtain the three-dimensional bounding box of the target obstacle in the designated coordinate system at the i-th historical moment, including: based on the position of the target vehicle at the i-th historical moment among the multiple historical moments, the three-dimensional bounding box of the target obstacle at the i-th historical moment among the multiple historical moments is converted from the target vehicle coordinate system to the designated coordinate system to obtain the three-dimensional bounding box of the target obstacle in the designated coordinate system at the i-th historical moment. The position of the target vehicle at each historical moment can be obtained by a position sensor device of the target vehicle itself, which is not limited in this embodiment.

[0052] The method of obtaining the real posture of the target obstacle at each historical moment may include: obtaining the real position and real orientation of the target obstacle at each of the multiple historical moments based on the point cloud data of the target obstacle collected by the target vehicle at each of the multiple historical moments, and taking the real position and real orientation of the target obstacle at each of the multiple historical moments as the real posture.

[0053] Adjusting the initial detection direction of the target obstacle at each historical moment based on the speed of the target obstacle at each historical moment to obtain the target detection direction of the target obstacle at each historical moment, including: adjusting the initial detection direction of the target obstacle at each historical moment based on the direction of the speed of the target obstacle at each historical moment and the angle between the initial detection direction of the target obstacle at each historical moment to obtain the target detection direction of the target obstacle at each historical moment.

[0054] For example, taking any historical moment as the i-th historical moment as an example, adjusting the initial detection orientation of the target obstacle at each historical moment based on the direction of the speed of the target obstacle at each historical moment and the angle of the initial detection orientation of the target obstacle at each historical moment to obtain the target detection orientation of the target obstacle at each historical moment may include one of the following: when the angle between the initial detection orientation of the target obstacle at the i-th historical moment and the speed direction of the target obstacle at the i-th historical moment is greater than or equal to 45 degrees and less than 135 degrees, rotating the orientation and / or bounding box of the target obstacle at the i-th historical moment by 90 degrees to obtain the target detection orientation of the target obstacle at the i-th historical moment; when the angle between the initial detection orientation of the target obstacle at the i-th historical moment and the speed direction of the target obstacle at the i-th historical moment is greater than or equal to 1 When the angle between the initial detection orientation of the target obstacle at the i-th historical moment and the speed direction at the i-th historical moment is greater than or equal to 225 degrees and less than 315 degrees, the orientation and / or the bounding box of the target obstacle at the i-th historical moment is rotated 180 degrees to obtain the target detection orientation of the target obstacle at the i-th historical moment; when the angle between the initial detection orientation of the target obstacle at the i-th historical moment and the speed direction at the i-th historical moment is greater than or equal to 225 degrees and less than 315 degrees, the orientation and / or the bounding box of the target obstacle at the i-th historical moment is rotated 270 degrees to obtain the target detection orientation of the target obstacle at the i-th historical moment; when the angle between the initial detection orientation of the target obstacle at the i-th historical moment and the speed direction at the i-th historical moment is less than 45 degrees or greater than 315 degrees, the orientation and / or the bounding box of the target obstacle at the i-th historical moment is not rotated to obtain the target detection orientation of the target obstacle at the i-th historical moment.

[0055] By adopting the method provided in this embodiment, based on the speed of the target obstacle at each historical moment, the initial detection orientation of the target obstacle at each historical moment is adjusted to obtain the target detection orientation of the target obstacle at each historical moment, and then the detection position and target detection orientation of the target obstacle at each historical moment are used as the detection posture of the target obstacle at each historical moment, thereby ensuring the accuracy of the orientation in the adjusted detection posture.

[0056] Further, the aforementioned image of the target obstacle at each of the multiple historical moments may refer to an image containing the target obstacle at each of the multiple historical moments. That is, based on the image of the target obstacle at each of the multiple historical moments, determining the two-dimensional bounding box of the target obstacle at each of the multiple historical moments may specifically be: based on the image containing the target obstacle at each of the multiple historical moments, determining one or more two-dimensional bounding boxes at each of the multiple historical moments, wherein one of the one or more two-dimensional bounding boxes corresponds to the target obstacle. Accordingly, associating the three-dimensional bounding box of the target obstacle at each of the historical moments with the two-dimensional bounding box includes: determining the two-dimensional bounding box corresponding to the target obstacle from the one or more two-dimensional bounding boxes at each of the historical moments, and associating the three-dimensional bounding box of the target obstacle at each of the historical moments with the two-dimensional bounding box.

[0057] Taking any historical moment as the i-th historical moment as an example, determining the two-dimensional bounding box corresponding to the target obstacle from one or more two-dimensional bounding boxes at each historical moment, and associating the three-dimensional bounding box of the target obstacle at each historical moment with the two-dimensional bounding box includes: projecting the three-dimensional bounding box of the target obstacle at the i-th historical moment into the image at the i-th historical moment to obtain the projection position of the three-dimensional bounding box in the image at the i-th historical moment; determining the two-dimensional bounding box corresponding to the target obstacle from one or more two-dimensional bounding boxes in the image at the i-th historical moment based on the projection position of the three-dimensional bounding box in the image at the i-th historical moment, and associating the three-dimensional bounding box of the target obstacle at the i-th historical moment with the two-dimensional bounding box.

[0058] It should be understood that the image of the target vehicle captured at each of the multiple historical moments may contain one or more obstacles, and the manner of associating the two-dimensional and three-dimensional bounding boxes for each obstacle is the same as the above embodiment, which will not be repeated here.

[0059] In this way, the original data of the target obstacle can be analyzed and processed in multiple dimensions to obtain the speed, detection posture, three-dimensional bounding box and its associated two-dimensional bounding box of the target obstacle at each historical moment as the predicted data of the target obstacle at each historical moment, thereby ensuring the accuracy of the predicted data.

[0060] In one example, based on the multiple prediction data of the target obstacle at multiple historical moments, generating multiple spatial data of the target obstacle at multiple historical moments may include: using one or more prediction data of the target obstacle at each of the multiple historical moments as one or more spatial data of the target obstacle at each of the multiple historical moments. That is, the multiple prediction data of the target obstacle at multiple historical moments are the same as the multiple spatial data of the target obstacle at multiple historical moments.

[0061] In one example, generating multiple spatial data of the target obstacle at multiple historical moments based on multiple predicted data of the target obstacle at multiple historical moments may include: using multiple predicted data of the target obstacle at multiple historical moments and multiple original data of the target obstacle at multiple historical moments as multiple spatial data of the target obstacle at multiple historical moments.

[0062] By adopting the method provided by this embodiment, based on the multiple original data of the target obstacle at multiple historical moments, multiple predicted data of the target obstacle at multiple historical moments are obtained, and based on the multiple predicted data of the target obstacle at multiple historical moments, multiple spatial data of the target obstacle at multiple historical moments are generated. In this way, the original data of the target obstacle at multiple historical moments can be accurately processed in multiple dimensions, so that more abundant and accurate multiple spatial data of the target obstacle at each historical moment can be obtained, thereby ensuring the accuracy of the subsequently generated annotation information.

[0063] In some possible implementations, constructing a target factor graph based on the multiple spatial data at the multiple historical moments and one or more constraint factors between the multiple spatial data at the multiple historical moments includes: generating multiple data nodes based on the multiple spatial data at the multiple historical moments, wherein different data nodes among the multiple data nodes correspond to different spatial data; determining one or more constraint factor nodes and multiple data nodes associated with each of the one or more constraint factor nodes based on the one or more constraint factors between the multiple spatial data at the multiple historical moments, wherein different constraint factor nodes among the one or more constraint factor nodes correspond to different constraint factors; connecting each constraint factor node with the multiple data nodes associated with each constraint factor node to generate multiple edges; constructing the target factor graph based on the multiple data nodes, the multiple edges, and the one or more constraint factor nodes.

[0064] Among them, the target factor graph is used to represent the constraints and dependencies between data nodes and constraint factor nodes. Specifically, the target factor graph may include data nodes, constraint factor nodes, and edges; data nodes may be nodes corresponding to spatial data; constraint factor nodes may be nodes corresponding to or containing constraint factors; and edges are used to indicate which data nodes the constraint factor nodes are related to (i.e., constraints and dependencies). In this way, the independence relationship between each data node and constraint factor is clearly demonstrated, which helps to simplify calculations and improve reasoning efficiency.

[0065] The generating of multiple data nodes based on the multiple spatial data at the multiple historical moments may include: generating the kth data node among the multiple data nodes based on the jth spatial data among one or more spatial data at the ith historical moment among the multiple historical moments, where i, j, and k are all positive integers. Among them, the kth data node may include: the display information of the kth data node displayed in the target factor graph, and the jth spatial data at the ith historical moment corresponding to the kth data node. The jth spatial data at the ith historical moment corresponding to the kth data node may be displayed or not displayed in the target factor graph. The method for determining each data node is the same as that for the kth data node, and will not be repeated here.

[0066] Here, it should be pointed out that in the process of constructing the target factor graph, a corresponding relationship is established between each spatial data and each data node. At this time, the spatial data corresponding to each data node can be called the initial spatial data corresponding to the data node.

[0067] The constraint factor may be a calculation function for representing the constraint relationship between multiple spatial data. Here, any constraint factor may correspond to at least two spatial data among all the spatial data, and the spatial data corresponding to different constraint factors are at least partially different.

[0068] The determining of the multiple data nodes associated with each constraint factor node in the one or more constraint factor nodes may include: based on the mth constraint factor corresponding to the mth constraint factor node, determining at least two spatial data corresponding to the mth constraint factor, and determining the data nodes corresponding to the at least two spatial data, wherein m is a positive integer, and the mth constraint factor node is any one of the one or more constraint factor nodes. Since the determination method of the multiple data nodes associated with each constraint factor node is the same as that of the mth constraint factor node, they are not described one by one.

[0069] In this way, by using multiple spatial data and their constraint factors at multiple historical moments, data nodes and edges can be generated, so that a target factor graph can be constructed based on each data node and edge. Through the expression of the target factor graph, the spatiotemporal constraint relationship between multiple spatial data of the target obstacle at multiple historical moments can be more clearly expressed.

[0070] Furthermore, a data node may include a variable node and / or an observation node.

[0071] Optionally, the generating of multiple data nodes based on the multiple spatial data at the multiple historical moments includes: when there are one or more original data and one or more predicted data in the multiple spatial data at the multiple historical moments, generating one or more observation nodes based on the one or more original data, generating one or more variable nodes based on the one or more predicted data, and using the one or more observation nodes and the one or more variable nodes as the multiple data nodes, wherein different observation nodes among the one or more observation nodes correspond to different original data, and different variable nodes among the one or more variable nodes correspond to different predicted data.

[0072] The generating one or more observation nodes based on the one or more original data may include: generating the ath observation node in the one or more observation nodes based on the ath original data in the one or more original data, where a is a positive integer. The method of generating each observation node of the one or more observation nodes is the same as the method of generating the ath observation node, which will not be repeated here.

[0073] The generating one or more variable nodes based on the one or more prediction data may include: generating the bth variable node in the one or more variable nodes based on the bth prediction data in the one or more prediction data, where b is a positive integer. The method of generating each variable node in the one or more variable nodes is the same as the method of generating the bth variable node, which will not be described in detail here.

[0074] In this way, one or more observation nodes are generated based on the one or more original data, and one or more variable nodes are generated based on the one or more predicted data, and the one or more observation nodes and the one or more variable nodes are used as the multiple data nodes. In this way, the observation nodes corresponding to the original data can be added to the target factor graph, that is, the real and accurate original information is added, and then it can be ensured that when the factor graph is optimized later, the real and accurate original information is used to constrain other predicted information, which can improve the accuracy of optimization.

[0075] Optionally, the generating of multiple data nodes based on the multiple spatial data at the multiple historical moments includes: when there is no variable data in the multiple spatial data at the multiple historical moments and only multiple prediction data, generating multiple variable nodes based on the multiple prediction data, and using the multiple variable nodes as the multiple data nodes. The processing method of generating multiple variable nodes based on the multiple prediction data is similar to the above-mentioned implementation method, and will not be described in detail here.

[0076] It should be pointed out that in the above examples, data nodes can be divided into two types: variable nodes and observation nodes according to the corresponding spatial data. Among them, the original data corresponding to each observation node is unchanged in the process of constructing and optimizing the target factor graph, so the original data corresponding to each observation node is called original data at any stage; the predicted data corresponding to each variable node can be adjusted or optimized in the process of optimizing the target factor graph, so the predicted data corresponding to the variable node obtained when constructing the target factor graph is called initial predicted data.

[0077] The one or more constraint factors between the multiple spatial data at the multiple historical moments include at least one of the following: one or more constraint factors between the multiple spatial data at every two adjacent historical moments among the multiple historical moments; one or more constraint factors between the multiple spatial data at each historical moment among the multiple historical moments.

[0078] Among multiple historical moments, one or more constraint factors between multiple spatial data at any two adjacent historical moments may be: constraint factors between one or more spatial data at the earlier moment and one or more spatial data at the later moment of the two adjacent historical moments.

[0079] In a plurality of historical moments, any constraint factor between a plurality of spatial data at any historical moment may be a constraint factor between at least two spatial data at the historical moment. For example, in the i-th historical moment, the plurality of spatial data includes four spatial data, and the one or more constraint factors between the four spatial data at the i-th historical moment may include at least one of a constraint factor of three spatial data, a constraint factor of two spatial data, a constraint factor of four spatial data, and the like.

[0080] In this way, the spatial data at each two adjacent historical moments can be constrained, and the multiple spatial data at each historical moment can also be constrained. In this way, richer constraint relationships can be obtained, and the spatiotemporal constraint relationship between multiple spatial data of target obstacles at multiple historical moments can be expressed more clearly.

[0081] Specifically, the one or more constraint factors between multiple spatial data at every two adjacent historical moments in the multiple historical moments include at least one of the following: a constraint factor between the speeds of every two adjacent historical moments in the multiple historical moments; a constraint factor between the speed and the actual posture at every two adjacent historical moments; a constraint factor between the actual posture and the original data at every two adjacent historical moments.

[0082] The constraint factor between the speeds of any two adjacent historical moments can be: the difference between the speed of the earlier moment and the speed of the later moment in the two adjacent historical moments. For example, assuming that the two adjacent historical moments are time t and time t+1, the constraint factor between the speeds of the two adjacent historical moments can be expressed as follows: v =v i,t+1 -v i,t ; where v i,t+1 represents the speed vi, v at the later time (t+1 time) i,t represents the velocity vi at the previous time (time t), e v It represents the difference between the speed at the earlier moment and the speed at the later moment in two adjacent historical moments.

[0083] The constraint factor between the speed and the true posture in any two adjacent historical moments can be: the true posture in the later moment of the two adjacent historical moments is subtracted from the true posture in the earlier moment to obtain a first value, and the difference between the first value and the product of the speed in the earlier moment and the time interval is used. For example, assuming that the two adjacent historical moments are time t and time t+1, the constraint factor between the speed and the true posture in the two adjacent historical moments is as follows: e M =x i,t+1 -x i,t -v i,t Δt; where x i,t+1 is the true position at the later time t+1, x i,t is the true position at the previous time t, v i,t is the velocity at the previous moment, Δt is the time interval, e M Indicates the final calculated difference.

[0084] The constraint factor between the true posture and the original data in any two adjacent historical moments can be: the difference between the original data at the earlier moment and the second value between the two adjacent historical moments, and the second value is equal to the product of the true posture at the earlier moment, the inverse of the true posture at the later moment, and the original data at the later moment.

[0085] For example, when the original data is point cloud data, two adjacent historical moments are time t and time t+1, and the constraint factor between the true pose and the original data in the two adjacent historical moments is as follows: e P=P i,t -x i,t (x i,t+1 ) -1 P i,t+1 ; Among them, P i,t is the point cloud data at the previous moment, x i,t is the true position at the previous moment, (x i,t+1 ) -1 is the inverse of the true pose at the next moment, P i,t+1 is the point cloud data at the later time, e P Indicates the difference between the original data at the previous time and the second value.

[0086] It should be pointed out that the above raw data is only illustrative of point cloud data. In actual processing, the raw data may also be images or other actually collected or observed data, which is not limited or exhaustive in this embodiment.

[0087] In this way, the speed, true posture, and original data at every two adjacent historical moments can be constrained, thereby more clearly expressing the spatiotemporal constraint relationship between the speed, true posture, and original data of the target obstacle at every two adjacent historical moments in multiple historical moments.

[0088] The one or more constraint factors between multiple spatial data at each of the multiple historical moments include at least one of the following: a constraint factor between the real posture and the detected posture at each of the multiple historical moments; a constraint factor between the real posture, the bounding box, and the original data at each historical moment, wherein the bounding box includes a two-dimensional bounding box and / or a three-dimensional bounding box.

[0089] Preferably, the enclosing box may refer to a three-dimensional enclosing box. Optionally, the enclosing box may also be a two-dimensional enclosing box, which is not limited in this example.

[0090] The constraint factor between the real pose and the detected pose at any historical moment can be: the product of the real pose and the inverse of the detected pose at that historical moment. Taking time t as an example, the constraint factor between the real pose and the detected pose at the historical moment can be expressed as follows: in, is the inverse of the detected pose at historical time t, x i,t is the real position at historical moment t, e D Represents the final product result.

[0091] The constraint factor between the true pose, bounding box, and original data at any historical moment can be: the sum of the absolute value of the product of the inverse of the true pose at the historical moment and the original data and the size difference of the bounding box. Taking the historical moment t and the bounding box as a three-dimensional bounding box as an example, when the original data is point cloud data, the constraint factor between the true pose, three-dimensional bounding box, and original data at the historical moment t can be expressed as: e B =∑{|(x i,t ) -1 P i,t |-0.5S i}; where (x i,t ) -1 represents the inverse of the true position at historical time t, P i,t represents the point cloud data at historical time t, S i represents the size of the three-dimensional bounding box at the historical time t (the size of the three-dimensional bounding box is used to represent the length, width and height of the target obstacle), e B Represents the final summation result.

[0092] In this way, the bounding box, true posture, and original data at each historical moment can be constrained, thereby more clearly expressing the spatial constraint relationship between the bounding box, true posture, and original data of the target obstacle at each historical moment.

[0093] Combination Figure 2 The above target factor graph is exemplified as follows: Figure 2 Constraint factor 1 is the constraint factor between the speeds of every two adjacent historical moments in multiple historical moments; constraint factor 2 is the constraint factor between the speed and the real posture in every two adjacent historical moments; constraint factor 3 is the constraint factor between the real posture and the original data (i.e., point cloud data) in every two adjacent historical moments; constraint factor 4 is the constraint factor between the real posture and the detected posture at each historical moment in the multiple historical moments; constraint factor 5 is the constraint factor between the real posture, the size of the bounding box, and the original data at each historical moment; speed t1 and speed t2, speed t2 and speed t3 are the speeds of two adjacent historical moments respectively; real posture t1 and real posture t2, real posture t2 and real posture t3 are the real postures of two adjacent historical moments respectively; detected posture t1 and detected posture t2, detected posture t2 and detected posture t3 are the detected postures of two adjacent historical moments respectively; Si, Si1, Si2 are the bounding boxes of the target obstacles at each moment (i.e., the size of the bounding box of the target obstacle, i.e., length, width, and height); each connecting line is an edge.

[0094] In some possible implementations, the optimization based on the target factor graph to obtain multiple optimization data at the multiple historical moments includes: based on the constraint factor corresponding to each constraint factor node in the target factor graph, calculating multiple spatial data corresponding to multiple data nodes connected by multiple edges of each constraint factor node to obtain one or more constraint values; based on the one or more constraint values, optimizing the multiple spatial data corresponding to the multiple data nodes to obtain an optimized target factor graph, wherein each of the multiple data nodes in the optimized target factor graph corresponds to optimized spatial data; based on the optimized target factor graph, obtaining multiple optimization data at the multiple historical moments.

[0095] In this embodiment, based on the constraint factors corresponding to each constraint factor node in the target factor graph, the multiple spatial data corresponding to the multiple data nodes connected by the multiple edges of each constraint factor node are calculated to obtain one or more constraint values, and based on the one or more constraint values, the multiple spatial data corresponding to the multiple data nodes are optimized, which can be executed in a loop or iterative manner. That is, in the current iterative processing, based on the constraint factors corresponding to each constraint factor node in the target factor graph, the multiple spatial data corresponding to the multiple data nodes connected by the multiple edges of each constraint factor node are calculated to obtain one or more constraint values; based on the one or more constraint values, it is determined whether the preset conditions are met; if the preset conditions are not met, the multiple spatial data corresponding to the multiple data nodes are optimized and the next iterative processing is performed; if the preset conditions are met, the optimized target factor graph is obtained, and each of the multiple data nodes in the optimized target factor graph corresponds to the optimized spatial data.

[0096] The above multiple iterations are described below as an example:

[0097] In any iterative processing, based on the constraint factors corresponding to each constraint factor node in the target factor graph, multiple spatial data corresponding to multiple data nodes connected by multiple edges of each constraint factor node are calculated to obtain one or more constraint values, which may include: in the current iterative processing, based on the qth constraint factor corresponding to the qth constraint factor node in the target factor graph, multiple spatial data corresponding to multiple data nodes connected by multiple edges of the qth constraint factor node are calculated to obtain the qth constraint value among the one or more constraint values ​​of the current iterative processing, where q is a positive integer. The calculation method of each constraint value is the same as the calculation method of the qth constraint value, and one or more constraint values ​​of the current iterative processing can be finally obtained, which will not be repeated here.

[0098] In any iterative process, after obtaining one or more constraint values ​​of the current iterative process, the spatial data corresponding to each data node is optimized based on the one or more constraint values ​​to obtain an optimized target factor graph, including: judging whether the preset conditions are met based on the one or more constraint values ​​of the current iterative process; if the preset conditions are met, the multiple spatial data corresponding to the multiple data nodes are used as the optimized target factor graph; if the preset conditions are not met, the multiple spatial data corresponding to the multiple data nodes are adjusted based on the nonlinear optimization algorithm to obtain the multiple adjusted spatial data corresponding to the multiple data nodes. After obtaining the multiple adjusted spatial data corresponding to the multiple data nodes, the next iterative process is performed, that is, the process of returning to the constraint factor corresponding to each constraint factor node in the target factor graph, calculating the multiple adjusted spatial data corresponding to the multiple data nodes connected by the multiple edges of each constraint factor node, and obtaining one or more adjusted constraint value processing, and so on, until the one or more adjusted constraint values ​​obtained after a certain iterative process meet the preset conditions, and the multiple spatial data corresponding to the multiple data nodes are used as the optimized target factor graph.

[0099] It should be pointed out that when the current iteration process is the first iteration process, the multiple spatial data corresponding to the multiple data nodes may refer to the initial spatial data corresponding to each of the multiple data nodes. Accordingly, if one or more constraint values ​​of the first iteration process satisfy the preset conditions, the initial spatial data corresponding to each data node may be directly used as the optimized spatial data corresponding to each data node, and the optimized target factor graph may be directly obtained.

[0100] When the current iteration process is not the first iteration process, the multiple spatial data corresponding to the multiple data nodes may refer to the multiple spatial data corresponding to the multiple data nodes after the last adjustment. Accordingly, if one or more constraint values ​​of the non-first iteration process meet the preset conditions, the spatial data corresponding to each data node after the last adjustment may be used as the optimized spatial data corresponding to each data node to obtain the optimized target factor graph.

[0101] Wherein, the adjustment of the multiple spatial data corresponding to the multiple data nodes based on the nonlinear optimization algorithm to obtain the multiple adjusted spatial data corresponding to the multiple data nodes may be: based on the nonlinear optimization algorithm and the multiple constraint factors corresponding to the multiple constraint factor nodes in the factor graph, the optimization direction and step size of the multiple spatial data corresponding to the multiple data nodes are determined; based on the optimization direction and step size of the multiple spatial data corresponding to the multiple data nodes, the currently optimized spatial data corresponding to each data node in the multiple data nodes is obtained. Wherein, the nonlinear optimization algorithm includes one of the following: gradient descent algorithm, Newton algorithm, quasi-Newton algorithm, etc. Wherein, the optimization direction may also be referred to as the search direction. When the optimization algorithm is a gradient descent algorithm, the optimization direction or search direction may be a negative gradient direction or a gradient direction, and the step size may be the amplitude of the adjustment. It can be understood that this is only an exemplary description, and the specific optimization direction and step size can be set according to the actual situation, and this application does not limit it.

[0102] Whether the one or more constraint values ​​meet the preset condition can be determined based on one or more threshold values ​​corresponding to the one or more constraint values. Specifically, it can be: when any constraint value among the one or more constraint values ​​is greater than the corresponding threshold value, it is determined that the constraint value does not meet the preset condition; when any constraint value among the one or more constraint values ​​is less than or equal to the corresponding threshold value, it is determined that the constraint value meets the preset condition, wherein the one or more threshold values ​​corresponding to the one or more constraint values ​​can be set according to actual conditions, and the threshold values ​​corresponding to different constraint values ​​among the one or more threshold values ​​corresponding to the one or more constraint values ​​can be the same or different, which is not limited here. For example, any one of the one or more threshold values ​​can be 0, 0.1, 0.2, 1, etc.

[0103] The obtaining of the multiple optimization data at the multiple historical moments based on the optimized target factor graph may include: using the optimized spatial data corresponding to each data node in the multiple data nodes in the optimized target factor graph as the multiple optimization data at the multiple historical moments.

[0104] By adopting the method provided in this embodiment, the constraint value obtained by calculating the multiple spatial data at multiple historical moments through the constraint factor, and then optimizing the multiple spatial data at multiple historical moments through the constraint value, the multiple spatial data at multiple historical moments after optimization can be made to better comply with the time and space constraints, thereby ensuring the accuracy of the multiple optimized data at multiple historical moments finally obtained.

[0105] Furthermore, data nodes can be divided into two types: variable nodes and observation nodes according to the corresponding spatial data.

[0106] In one example, multiple data nodes are all variable nodes. The above embodiments may specifically be: based on the constraint factors corresponding to each constraint factor node in the target factor graph, multiple prediction data corresponding to multiple variable nodes connected by multiple edges of each constraint factor node are calculated to obtain one or more constraint values; based on the one or more constraint values, multiple prediction data corresponding to the multiple variable nodes are optimized to obtain an optimized target factor graph, wherein each variable node in the multiple variable nodes in the optimized target factor graph corresponds to optimized prediction data; based on the optimized target factor graph, multiple optimization data at the multiple historical moments are obtained.

[0107] In this example, based on the constraint factors corresponding to each constraint factor node in the target factor graph, multiple predicted data corresponding to multiple variable nodes connected by multiple edges of each constraint factor node are calculated to obtain one or more constraint values, and based on the one or more constraint values, multiple predicted data corresponding to the multiple variable nodes are optimized, which can also be executed in a loop or iterative manner. That is, in the current iteration, based on the constraint factors corresponding to each constraint factor node in the target factor graph, multiple predicted data corresponding to multiple variable nodes connected by multiple edges of each constraint factor node are calculated to obtain one or more constraint values; based on the one or more constraint values, it is determined whether the preset conditions are met; if the preset conditions are not met, the multiple predicted data corresponding to the multiple variable nodes are optimized and the next iteration is performed; if the preset conditions are met, the optimized target factor graph is obtained, and each variable node in the multiple data nodes in the optimized target factor graph corresponds to the optimized predicted data.

[0108] It should be noted that when the current iteration process is the first iteration process, the multiple prediction data corresponding to the multiple variable nodes may refer to the initial prediction data corresponding to each variable node in the multiple variable nodes. Accordingly, if one or more constraint values ​​of the first iteration process meet the preset conditions, the initial prediction data corresponding to each variable node may be directly used as the optimized prediction data corresponding to each variable node, and the optimized target factor graph may be directly obtained.

[0109] When the current iteration process is not the first iteration process, the multiple prediction data corresponding to the multiple variable nodes may refer to the multiple prediction data corresponding to the multiple variable nodes after the previous adjustment. Accordingly, if one or more constraint values ​​of the non-first iteration process meet the preset conditions, the prediction data corresponding to each variable node after the previous adjustment may be used as the optimized prediction data corresponding to each variable node to obtain the optimized target factor graph.

[0110] Since the specific processing method in this example is similar to that in the aforementioned embodiment, it will not be described repeatedly.

[0111] In one example, based on the one or more constraint values, the multiple spatial data corresponding to the multiple data nodes are optimized to obtain an optimized target factor graph, including: when the multiple data nodes include one or more observation nodes and one or more variable nodes, based on the one or more constraint values, the one or more predicted data corresponding to the one or more variable nodes are optimized to obtain the optimized target factor graph, wherein each of the one or more variable nodes in the optimized target factor graph corresponds to optimized predicted data, and each of the one or more observation nodes corresponds to original data.

[0112] In this example, based on the constraint factors corresponding to each constraint factor node in the target factor graph, the multiple predicted data corresponding to the multiple variable nodes connected by the multiple edges of each constraint factor node are calculated to obtain one or more constraint values, and based on the one or more constraint values, the multiple spatial data corresponding to the multiple data nodes are optimized. This can also be performed in a loop or iterative manner. That is, in the current iteration, based on the constraint factors corresponding to each constraint factor node in the target factor graph, the one or more predicted data corresponding to the one or more variable nodes connected by the multiple edges of each constraint factor node and / or the one or more original data corresponding to the one or more observation nodes are calculated to obtain one or more constraint values; based on the one or more constraint values, it is determined whether the preset conditions are met; if the preset conditions are not met, the multiple predicted data corresponding to the multiple variable nodes are optimized and the next iteration is performed; if the preset conditions are met, the optimized target factor graph is obtained, at which time each variable node in the one or more variable nodes in the optimized target factor graph corresponds to the optimized predicted data, and each observation node in the one or more observation nodes corresponds to the original data.

[0113] In any iterative processing, based on the constraint factors corresponding to each constraint factor node in the target factor graph, multiple spatial data corresponding to multiple data nodes connected by multiple edges of each constraint factor node are calculated to obtain one or more constraint values, which may include: in the current iterative processing, based on the qth constraint factor corresponding to the qth constraint factor node in the target factor graph, one or more predicted data corresponding to one or more variable nodes connected by multiple edges of the qth constraint factor node, and / or one or more original data corresponding to one or more observation nodes are calculated to obtain the qth constraint value among the one or more constraint values ​​of the current iterative processing, where q is a positive integer. The calculation method of each constraint value is the same as the calculation method of the qth constraint value, and finally one or more constraint values ​​of the current iterative processing can be obtained, which will not be repeated here.

[0114] In any iterative process, after obtaining one or more constraint values ​​of the current iterative process, optimizing one or more spatial data corresponding to the one or more variable nodes based on the one or more constraint values ​​to obtain one or more optimized spatial data corresponding to the one or more variable nodes included in the optimized target factor graph includes:

[0115] Determining whether a preset condition is met based on one or more constraint values ​​of the current iterative processing;

[0116] In the case where the one or more constraint values ​​satisfy a preset condition, the one or more predicted data corresponding to the one or more variable nodes are used as the one or more optimized predicted data corresponding to the one or more variable nodes included in the optimized target factor graph, and the one or more original data corresponding to the one or more observation nodes are directly used as the one or more original data corresponding to the one or more observation nodes included in the optimized target factor graph;

[0117] When the one or more constraint values ​​do not satisfy a preset condition, one or more prediction data corresponding to the one or more variable nodes are adjusted based on a nonlinear optimization algorithm to obtain one or more adjusted prediction data corresponding to the one or more variable nodes.

[0118] After obtaining the one or more adjusted prediction data corresponding to the one or more variable nodes, the next iterative processing is performed, that is, returning to execute the constraint factor corresponding to each constraint factor node in the target factor graph, calculating the multiple spatial data corresponding to the multiple data nodes connected by the multiple edges of each constraint factor node, and obtaining one or more constraint value processing, and so on, until the one or more adjusted constraint values ​​obtained after a certain iterative processing meet the preset conditions, and the multiple spatial data corresponding to the multiple data nodes are used as the optimized target factor graph, and finally the one or more prediction data corresponding to the one or more variable nodes are used as the one or more optimized prediction data corresponding to the one or more variable nodes contained in the optimized target factor graph, and the one or more original data corresponding to the one or more observation nodes are directly used as the one or more original data corresponding to the one or more observation nodes contained in the optimized target factor graph.

[0119] It should be noted that when the current iteration process is the first iteration process, the multiple prediction data corresponding to the multiple variable nodes may refer to the initial prediction data corresponding to each variable node in the multiple variable nodes. Accordingly, if one or more constraint values ​​of the first iteration process satisfy the preset conditions, the initial prediction data corresponding to each variable node may be directly used as the optimized prediction data corresponding to the variable node included in the optimized target factor graph, and the one or more original data corresponding to the one or more observation nodes may be directly used as the one or more original data corresponding to the one or more observation nodes included in the optimized target factor graph.

[0120] When the current iteration process is not the first iteration process, the multiple prediction data corresponding to the multiple variable nodes may refer to the multiple prediction data corresponding to the multiple variable nodes after the last adjustment. Correspondingly, if one or more constraint values ​​of the non-first iteration process meet the preset conditions, the one or more prediction data corresponding to the one or more variable nodes may be used as the one or more optimized prediction data corresponding to the one or more variable nodes included in the optimized target factor graph, and the one or more original data corresponding to the one or more observation nodes may be directly used as the one or more original data corresponding to the one or more observation nodes included in the optimized target factor graph.

[0121] Since the specific processing method in this example is similar to that in the aforementioned embodiment, it will not be described repeatedly.

[0122] By adopting the above processing, combined with the types of data nodes contained in the target factor graph, different types of data nodes are adjusted or processed differently, so that when the data node contains an observation node, that is, a node corresponding to the original data, only the predicted data corresponding to the variable node can be optimized, thereby ensuring that the real data is not tampered with and ensuring the accuracy of the optimization.

[0123] In some possible implementations, the method further includes one of the following:

[0124] In the case where there is no missing moment in the multiple historical moments, using the labeling information of the target obstacle at each historical moment in the multiple historical moments as the labeling information of the continuous historical moments of the target obstacle;

[0125] In the case that there is one or more missing moments among the multiple historical moments, the insertion annotation information of the target obstacle at each of the one or more missing moments is determined, and the insertion annotation information of the target obstacle at each of the one or more missing moments and the annotation information of the target obstacle at each of the multiple historical moments are used as the annotation information of the continuous historical moments of the target obstacle.

[0126] Wherein, determining the inserted annotation information of the target obstacle at each of the one or more missing moments includes: determining one or more pairs of candidate connected historical moments adjacent to the one or more missing moments among the multiple historical moments; and determining the inserted annotation information of the target obstacle at each of the one or more missing moments based on the annotation information of the one or more pairs of candidate connected historical moments.

[0127] The method of determining whether there are one or more missing moments in the multiple historical moments includes: determining whether there are one or more missing moments in the multiple historical moments based on the collection time interval of the target vehicle. The collection time interval of the target vehicle can be configured according to actual conditions, and this embodiment does not limit it.

[0128] Specifically, based on the collection time interval of the target vehicle, determining whether there are one or more missing moments in the multiple historical moments includes: determining that there are one or more missing moments in the multiple historical moments when the duration between one or more adjacent historical moments in the multiple historical moments is greater than the collection time interval of the target vehicle; and determining that there are not one or more missing moments in the multiple historical moments when the duration between one or more adjacent historical moments in the multiple historical moments is not greater than the collection time interval of the target vehicle.

[0129] The step of determining one or more pairs of candidate connected historical moments adjacent to the one or more missing moments from among the multiple historical moments comprises: taking adjacent historical moments among the multiple historical moments whose duration between each two adjacent historical moments is greater than the collection time interval as the one or more pairs of candidate connected historical moments.

[0130] The determining, based on the annotation information of the one or more pairs of candidate connected historical moments, the inserted annotation information of the target obstacle at each of the one or more missing moments may include: based on the annotation information of each pair of candidate adjacent historical moments in the one or more pairs of candidate connected historical moments, inserting annotation information at the historical moment corresponding to the collection time interval of each pair of candidate connected histories in the one or more pairs of candidate connected historical moments by proportional interpolation, to obtain the inserted annotation information of the target obstacle at each of the one or more missing moments.

[0131] By adopting the method provided in this embodiment, it is possible to determine whether there are missing moments in the multiple historical moments, and finally generate identification information of continuous historical moments of the target obstacle based on whether there are missing moments in the multiple historical moments, thereby ensuring the integrity of the labeling information of the target obstacle in time and space.

[0132] Furthermore, when there are one or more missing moments among multiple historical moments, the inserted annotation information of the missing moments is determined through the annotation information of the existing historical moments, so that the accuracy and rationality of the inserted annotation information of the target obstacle can be guaranteed.

[0133] Combination Figure 3 The above implementation is exemplarily described, and the specific steps include:

[0134] Step 301, obtaining point cloud data of one or more obstacles collected by the target vehicle at each of multiple historical moments, images of one or more obstacles collected by the target vehicle at each of the multiple historical moments, and the position of the target vehicle at each of the multiple historical moments.

[0135] In steps 302 to 308, any one of the one or more obstacles is used as a target obstacle as an example. It should be understood that the processing method of each obstacle is the same as the target obstacle, so steps 302 to 308 are also applicable to processing each obstacle. They are not described one by one for the sake of brevity.

[0136] Step 302, 3D model detection: input the point cloud data of the target obstacle at each historical moment into the three-dimensional bounding box detection model, and obtain the three-dimensional bounding box of the target obstacle at each historical moment output by the three-dimensional bounding box detection model.

[0137] Step 303, 2D model detection: input the image of the target obstacle at each historical moment into the two-dimensional bounding box detection model, and obtain the two-dimensional bounding box of the target obstacle at each historical moment output by the two-dimensional bounding box detection model.

[0138] Step 304, speed initialization: convert the three-dimensional bounding box of the target obstacle at each historical moment into a specified coordinate system to obtain the three-dimensional bounding box of the target obstacle in the specified coordinate system at each historical moment; use the position and orientation of the three-dimensional bounding box of the target obstacle in the specified coordinate system at each historical moment as the detection position and initial detection orientation of the target obstacle at each historical moment; and obtain the speed of the target obstacle at each historical moment based on the detection position of the target obstacle at each historical moment.

[0139] The converting the three-dimensional bounding box of the target obstacle at each historical moment into the designated coordinate system to obtain the three-dimensional bounding box of the target obstacle in the designated coordinate system at each historical moment includes: based on the position of the target vehicle at each historical moment in the multiple historical moments, converting the three-dimensional bounding box of the target obstacle at each historical moment in the multiple historical moments from the target vehicle coordinate system to the designated coordinate system to obtain the three-dimensional bounding box of the target obstacle in the designated coordinate system at each historical moment.

[0140] Step 305, consistency repair: based on the speed of the target obstacle at each historical moment, adjust the initial detection direction of the target obstacle at each historical moment to obtain the target detection direction of the target obstacle at each historical moment; use the detection position and target detection direction of the target obstacle at each historical moment as the detection posture of the target obstacle at each historical moment.

[0141] Step 306, associating 2D and 3D bounding boxes: associating the 3D bounding box of the target obstacle at each historical moment with the 2D bounding box.

[0142] The above steps obtain the speed of the target obstacle at each historical moment, the detected posture, and the detailed processing of the association between the three-dimensional bounding box and the two-dimensional bounding box at each historical moment, which is the same as the previous embodiment and will not be repeated.

[0143] Step 307, constructing a target factor graph: constructing a target factor graph based on multiple spatial data of the target obstacle at the multiple historical moments (i.e., at least one of point cloud data, image, two-dimensional bounding box, three-dimensional bounding box, detection posture, and speed) and one or more constraint factors between the multiple spatial data at the multiple historical moments.

[0144] Step 308, labeling information: performing optimization based on the target factor graph to obtain multiple optimization data at the multiple historical moments; obtaining labeling information of the target obstacle at each of the multiple historical moments based on the multiple optimization data at the multiple historical moments.

[0145] Furthermore, it may also include one of the following: when there is no missing moment between the multiple historical moments, using the labeling information of the target obstacle at each of the multiple historical moments as the labeling information of the continuous historical moments of the target obstacle; when there is one or more missing moments among the multiple historical moments, determining the inserted labeling information of the target obstacle at each of the one or more missing moments, and using the inserted labeling information of the target obstacle at each of the one or more missing moments and the labeling information of the target obstacle at each of the multiple historical moments as the labeling information of the continuous historical moments of the target obstacle.

[0146] The present disclosure also provides an obstacle marking device, such as Figure 4 As shown, including:

[0147] The spatial data acquisition module 401 is used to acquire multiple spatial data of the target obstacle at multiple historical moments;

[0148] A factor graph construction module 402, configured to construct a target factor graph based on the plurality of spatial data at the plurality of historical moments and one or more constraint factors between the plurality of spatial data at the plurality of historical moments;

[0149] A factor graph optimization module 403 is used to perform optimization based on the target factor graph to obtain multiple optimization data at the multiple historical moments;

[0150] The labeling information determination module 404 is used to obtain the labeling information of the target obstacle at each of the multiple historical moments based on the multiple optimization data at the multiple historical moments.

[0151] The factor graph construction module is used to generate multiple data nodes based on the multiple spatial data at the multiple historical moments, wherein different data nodes among the multiple data nodes correspond to different spatial data; determine one or more constraint factor nodes and multiple data nodes associated with each of the one or more constraint factor nodes based on one or more constraint factors between the multiple spatial data at the multiple historical moments, wherein different constraint factor nodes among the one or more constraint factor nodes correspond to different constraint factors; connect each constraint factor node with the multiple data nodes associated with each constraint factor node to generate multiple edges; and construct the target factor graph based on the multiple data nodes, the multiple edges, and the one or more constraint factor nodes.

[0152] The factor graph optimization module is used to calculate multiple spatial data corresponding to multiple data nodes connected by multiple edges of each constraint factor node based on the constraint factors corresponding to each constraint factor node in the target factor graph to obtain one or more constraint values; based on the one or more constraint values, optimize the multiple spatial data corresponding to the multiple data nodes to obtain an optimized target factor graph, wherein each of the multiple data nodes in the optimized target factor graph corresponds to optimized spatial data; based on the optimized target factor graph, obtain multiple optimization data at the multiple historical moments.

[0153] The factor graph construction module is used to generate one or more observation nodes based on the one or more original data and one or more variable nodes based on the one or more predicted data when there are one or more original data and one or more predicted data in the multiple spatial data at the multiple historical moments, and use the one or more observation nodes and the one or more variable nodes as the multiple data nodes, wherein different observation nodes in the one or more observation nodes correspond to different original data, and different variable nodes in the one or more variable nodes correspond to different predicted data.

[0154] The factor graph optimization module is used to optimize one or more prediction data corresponding to the one or more variable nodes based on the one or more constraint values ​​to obtain the optimized target factor graph when the multiple data nodes include one or more observation nodes and one or more variable nodes, wherein each of the one or more variable nodes in the optimized target factor graph corresponds to the optimized prediction data, and each of the one or more observation nodes corresponds to the original data.

[0155] The one or more constraint factors between the multiple spatial data at the multiple historical moments include at least one of the following: one or more constraint factors between the multiple spatial data at every two adjacent historical moments among the multiple historical moments; one or more constraint factors between the multiple spatial data at each historical moment among the multiple historical moments.

[0156] The one or more constraint factors between multiple spatial data at every two adjacent historical moments in the multiple historical moments include at least one of the following: a constraint factor between the speeds of every two adjacent historical moments in the multiple historical moments; a constraint factor between the speed and each posture in every two adjacent historical moments; a constraint factor between each posture in every two adjacent historical moments and the original data.

[0157] The one or more constraint factors between multiple spatial data at each of the multiple historical moments include at least one of the following: constraint factors between each posture and detected posture at each of the multiple historical moments; constraint factors between the real posture, bounding box, and original data at each historical moment, wherein the bounding box includes a two-dimensional bounding box and / or a three-dimensional bounding box.

[0158] The annotation information determination module is used to perform one of the following:

[0159] In the case where there is no missing moment between the multiple historical moments, the labeling information of the target obstacle at each historical moment in the multiple historical moments is used as the labeling information of the continuous historical moments of the target obstacle;

[0160] In the case that there is one or more missing moments among the multiple historical moments, the insertion annotation information of the target obstacle at each of the one or more missing moments is determined, and the insertion annotation information of the target obstacle at each of the one or more missing moments and the annotation information of the target obstacle at each of the multiple historical moments are used as the annotation information of the continuous historical moments of the target obstacle.

[0161] The annotation information determination module is used to determine one or more pairs of candidate connected historical moments adjacent to the one or more missing moments among the multiple historical moments; based on the annotation information of the one or more pairs of candidate connected historical moments, determine the insertion annotation information of the target obstacle at each missing moment among the one or more missing moments.

[0162] The spatial data acquisition module is used to acquire multiple original data of the target obstacle at multiple historical moments; obtain multiple predicted data of the target obstacle at multiple historical moments based on the multiple original data of the target obstacle at multiple historical moments; and generate multiple spatial data of the target obstacle at multiple historical moments based on the multiple predicted data of the target obstacle at multiple historical moments.

[0163] The spatial data acquisition module is used to determine a three-dimensional bounding box of the target obstacle at each historical moment based on the point cloud data of the target obstacle at each historical moment; determine a two-dimensional bounding box of the target obstacle at each historical moment based on the image of the target obstacle at each historical moment; obtain a speed and a detected posture of the target obstacle at each historical moment based on the three-dimensional bounding box of the target obstacle at each historical moment; and associate the three-dimensional bounding box of the target obstacle at each historical moment with the two-dimensional bounding box.

[0164] The spatial data acquisition module is used to obtain the detection position and initial detection orientation of the target obstacle at each historical moment based on the three-dimensional bounding box of the target obstacle at each historical moment; obtain the speed of the target obstacle at each historical moment based on the detection position of the target obstacle at each historical moment; adjust the initial detection orientation of the target obstacle at each historical moment based on the speed of the target obstacle at each historical moment to obtain the target detection orientation of the target obstacle at each historical moment; and use the detection position and target detection orientation of the target obstacle at each historical moment as the detection posture of the target obstacle at each historical moment.

[0165] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0166] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0167] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0168] Figure 5A schematic block diagram of an example electronic device 500 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0169] like Figure 5 As shown, the device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0170] A number of components in the device 500 are connected to the I / O interface 505, including: an input unit 506, such as a keyboard, a mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0171] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 501 performs the various methods and processes described above. For example, in some embodiments, the above method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of the method described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform the above method in any other appropriate manner (e.g., by means of firmware).

[0172] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0173] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0174] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0175] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0176] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0177] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0178] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0179] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An obstacle marking method, include: Obtain multiple spatial data of target obstacles at multiple historical moments; constructing a target factor graph based on the plurality of spatial data at the plurality of historical moments and one or more constraint factors between the plurality of spatial data at the plurality of historical moments; Performing optimization based on the target factor graph to obtain a plurality of optimization data at the plurality of historical moments; Based on the multiple optimization data at the multiple historical moments, the labeling information of the target obstacle at each of the multiple historical moments is obtained.

2. The method according to claim 1, in, The constructing a target factor graph based on the plurality of spatial data at the plurality of historical moments and one or more constraint factors between the plurality of spatial data at the plurality of historical moments comprises: Based on the plurality of spatial data at the plurality of historical moments, generating a plurality of data nodes, wherein different data nodes among the plurality of data nodes correspond to different spatial data; Based on one or more constraint factors between the plurality of spatial data at the plurality of historical moments, determine one or more constraint factor nodes and a plurality of data nodes associated with each of the one or more constraint factor nodes, wherein different constraint factor nodes in the one or more constraint factor nodes correspond to different constraint factors; Connecting each constraint factor node with a plurality of data nodes associated with each constraint factor node to generate a plurality of edges; The target factor graph is constructed based on the multiple data nodes, the multiple edges, and one or more constraint factor nodes.

3. The method according to claim 2, in, The optimization is performed based on the target factor graph to obtain multiple optimization data at the multiple historical moments, including: Based on the constraint factors corresponding to each constraint factor node in the target factor graph, multiple spatial data corresponding to multiple data nodes connected by multiple edges of each constraint factor node are calculated to obtain one or more constraint values; Based on the one or more constraint values, optimizing the multiple spatial data corresponding to the multiple data nodes to obtain an optimized target factor graph, wherein each of the multiple data nodes in the optimized target factor graph corresponds to the optimized spatial data; Based on the optimized target factor graph, a plurality of optimization data at the plurality of historical moments are obtained.

4. The method according to claim 3, in, The generating of multiple data nodes based on the multiple spatial data at the multiple historical moments includes: In the case where there are one or more original data and one or more predicted data in the multiple spatial data at the multiple historical moments, one or more observation nodes are generated based on the one or more original data, and one or more variable nodes are generated based on the one or more predicted data, and the one or more observation nodes and the one or more variable nodes are used as the multiple data nodes, wherein different observation nodes among the one or more observation nodes correspond to different original data, and different variable nodes among the one or more variable nodes correspond to different predicted data.

5. The method according to claim 4, in, The step of optimizing the plurality of spatial data corresponding to the plurality of data nodes based on the one or more constraint values ​​to obtain an optimized target factor graph includes: In a case where the multiple data nodes include one or more observation nodes and one or more variable nodes, one or more prediction data corresponding to the one or more variable nodes are optimized based on the one or more constraint values ​​to obtain the optimized target factor graph, wherein each of the one or more variable nodes in the optimized target factor graph corresponds to optimized prediction data, and each of the one or more observation nodes corresponds to original data.

6. The method according to any one of claims 1 to 5, in, The one or more constraint factors between the plurality of spatial data at the plurality of historical moments include at least one of the following: One or more constraint factors between the plurality of spatial data at every two adjacent historical moments in the plurality of historical moments; One or more constraint factors between the plurality of spatial data at each of the plurality of historical moments.

7. The method according to claim 6, in, The one or more constraint factors between the plurality of spatial data at each two adjacent historical moments in the plurality of historical moments include at least one of the following: A constraint factor between the speeds of every two adjacent historical moments in the plurality of historical moments; The constraint factor between the velocity and the true posture in each two adjacent historical moments; The constraint factor between the true pose and the original data in each two adjacent historical moments.

8. The method according to claim 6, wherein the one or more constraint factors between the plurality of spatial data at each of the plurality of historical moments include at least one of the following: A constraint factor between the real posture and the detected posture at each historical moment in the multiple historical moments; The constraint factors between the real pose, bounding box, and original data at each historical moment, in, The bounding box includes a two-dimensional bounding box and / or a three-dimensional bounding box.

9. The method according to claim 1, further comprising one of the following: In the case where there is no missing moment between the multiple historical moments, the labeling information of the target obstacle at each historical moment in the multiple historical moments is used as the labeling information of the continuous historical moments of the target obstacle; In the case that there is one or more missing moments among the multiple historical moments, the insertion annotation information of the target obstacle at each of the one or more missing moments is determined, and the insertion annotation information of the target obstacle at each of the one or more missing moments and the annotation information of the target obstacle at each of the multiple historical moments are used as the annotation information of the continuous historical moments of the target obstacle.

10. The method according to claim 9, in, The determining the insertion annotation information of the target obstacle at each missing moment in the one or more missing moments includes: Determining, from the plurality of historical moments, one or more pairs of candidate connected historical moments adjacent to the one or more missing moments; Based on the annotation information of the one or more pairs of candidate connected historical moments, the insertion annotation information of the target obstacle at each missing moment in the one or more missing moments is determined.

11. The method according to claim 7 or 8, in, The obtaining of multiple spatial data of the target obstacle at multiple historical moments includes: Obtain multiple raw data of the target obstacle at multiple historical moments; Based on a plurality of original data of the target obstacle at a plurality of historical moments, a plurality of predicted data of the target obstacle at a plurality of historical moments are obtained; Based on the multiple prediction data of the target obstacle at multiple historical moments, multiple spatial data of the target obstacle at multiple historical moments are generated.

12. The method according to claim 11, in, The obtaining, based on a plurality of original data of the target obstacle at a plurality of historical moments, a plurality of predicted data of the target obstacle at a plurality of historical moments comprises: Determine, based on the point cloud data of the target obstacle at each of the multiple historical moments, a three-dimensional bounding box of the target obstacle at each of the historical moments; Determine, based on the image of the target obstacle at each of the multiple historical moments, a two-dimensional bounding box of the target obstacle at each historical moment; Based on the three-dimensional bounding box of the target obstacle at each historical moment, obtaining the speed and detection posture of the target obstacle at each historical moment; The three-dimensional bounding box of the target obstacle at each historical moment is associated with the two-dimensional bounding box.

13. The method according to claim 12, in, The obtaining the speed and the detected posture of the target obstacle at each historical moment based on the three-dimensional bounding box of the target obstacle at each historical moment includes: Based on the three-dimensional bounding box of the target obstacle at each historical moment, obtaining the detection position and initial detection orientation of the target obstacle at each historical moment; Based on the detected position of the target obstacle at each historical moment, obtaining the speed of the target obstacle at each historical moment; Based on the speed of the target obstacle at each historical moment, adjusting the initial detection direction of the target obstacle at each historical moment to obtain the target detection direction of the target obstacle at each historical moment; The detected position and target detection orientation of the target obstacle at each historical moment are used as the detected posture of the target obstacle at each historical moment.

14. An obstacle marking device, include: A spatial data acquisition module is used to acquire multiple spatial data of target obstacles at multiple historical moments; A factor graph construction module, used to construct a target factor graph based on the plurality of spatial data at the plurality of historical moments and one or more constraint factors between the plurality of spatial data at the plurality of historical moments; A factor graph optimization module, used for performing optimization based on the target factor graph to obtain a plurality of optimization data at the plurality of historical moments; The labeling information determination module is used to obtain the labeling information of the target obstacle at each of the multiple historical moments based on the multiple optimization data at the multiple historical moments.

15. The device according to claim 14, in, The factor graph construction module is used to generate multiple data nodes based on the multiple spatial data at the multiple historical moments, wherein different data nodes among the multiple data nodes correspond to different spatial data; determine one or more constraint factor nodes and multiple data nodes associated with each of the one or more constraint factor nodes based on one or more constraint factors between the multiple spatial data at the multiple historical moments, wherein different constraint factor nodes among the one or more constraint factor nodes correspond to different constraint factors; connect each constraint factor node with the multiple data nodes associated with each constraint factor node to generate multiple edges; and construct the target factor graph based on the multiple data nodes, the multiple edges, and the one or more constraint factor nodes.

16. The device according to claim 15, in, The factor graph optimization module is used to calculate multiple spatial data corresponding to multiple data nodes connected by multiple edges of each constraint factor node based on the constraint factor corresponding to each constraint factor node in the target factor graph to obtain one or more constraint values; Based on the one or more constraint values, optimizing the multiple spatial data corresponding to the multiple data nodes to obtain an optimized target factor graph, wherein each of the multiple data nodes in the optimized target factor graph corresponds to the optimized spatial data; Based on the optimized target factor graph, a plurality of optimization data at the plurality of historical moments are obtained.

17. The device according to claim 16, in, The factor graph construction module is used to generate one or more observation nodes based on the one or more original data and one or more variable nodes based on the one or more predicted data when there are one or more original data and one or more predicted data in the multiple spatial data at the multiple historical moments, and use the one or more observation nodes and the one or more variable nodes as the multiple data nodes, wherein different observation nodes in the one or more observation nodes correspond to different original data, and different variable nodes in the one or more variable nodes correspond to different predicted data.

18. The device according to claim 17, in, The factor graph optimization module is used to optimize one or more prediction data corresponding to the one or more variable nodes based on the one or more constraint values ​​to obtain the optimized target factor graph when the multiple data nodes include one or more observation nodes and one or more variable nodes, wherein each of the one or more variable nodes in the optimized target factor graph corresponds to the optimized prediction data, and each of the one or more observation nodes corresponds to the original data.

19. The device according to any one of claims 14 to 18, in, The one or more constraint factors between the plurality of spatial data at the plurality of historical moments include at least one of the following: One or more constraint factors between the plurality of spatial data at every two adjacent historical moments in the plurality of historical moments; One or more constraint factors between the plurality of spatial data at each of the plurality of historical moments.

20. The device according to claim 19, in, The one or more constraint factors between the plurality of spatial data at each two adjacent historical moments in the plurality of historical moments include at least one of the following: A constraint factor between the speeds of every two adjacent historical moments in the plurality of historical moments; The constraint factor between the velocity and the true posture in each two adjacent historical moments; The constraint factor between the true pose and the original data in each two adjacent historical moments.

21. The apparatus according to claim 19, wherein the one or more constraint factors between the plurality of spatial data at each of the plurality of historical moments include at least one of the following: A constraint factor between the real posture and the detected posture at each historical moment in the multiple historical moments; The constraint factors between the real pose, bounding box, and original data at each historical moment, in, The bounding box includes a two-dimensional bounding box and / or a three-dimensional bounding box.

22. The apparatus according to claim 14, wherein the annotation information determination module is configured to perform one of the following: In the case where there is no missing moment between the multiple historical moments, the labeling information of the target obstacle at each historical moment in the multiple historical moments is used as the labeling information of the continuous historical moments of the target obstacle; In the case that there is one or more missing moments among the multiple historical moments, the insertion annotation information of the target obstacle at each of the one or more missing moments is determined, and the insertion annotation information of the target obstacle at each of the one or more missing moments and the annotation information of the target obstacle at each of the multiple historical moments are used as the annotation information of the continuous historical moments of the target obstacle.

23. The device according to claim 22, in, The annotation information determination module is used to determine one or more pairs of candidate connected historical moments adjacent to the one or more missing moments among the multiple historical moments; based on the annotation information of the one or more pairs of candidate connected historical moments, determine the insertion annotation information of the target obstacle at each missing moment among the one or more missing moments.

24. The device according to claim 20 or 21, in, The spatial data acquisition module is used to acquire a plurality of original data of the target obstacle at a plurality of historical moments; based on the plurality of original data of the target obstacle at a plurality of historical moments, obtain a plurality of predicted data of the target obstacle at a plurality of historical moments; Based on the multiple prediction data of the target obstacle at multiple historical moments, multiple spatial data of the target obstacle at multiple historical moments are generated.

25. The device according to claim 24, in, The spatial data acquisition module is used to determine a three-dimensional bounding box of the target obstacle at each historical moment based on the point cloud data of the target obstacle at each historical moment among the multiple historical moments; Based on the image of the target obstacle at each of the multiple historical moments, a two-dimensional bounding box of the target obstacle at each historical moment is determined; based on the three-dimensional bounding box of the target obstacle at each historical moment, a speed and a detected posture of the target obstacle at each historical moment are obtained; and the three-dimensional bounding box of the target obstacle at each historical moment is associated with the two-dimensional bounding box.

26. The device according to claim 25, in, The spatial data acquisition module is used to obtain the detection position and initial detection orientation of the target obstacle at each historical moment based on the three-dimensional bounding box of the target obstacle at each historical moment; Based on the detected position of the target obstacle at each historical moment, obtaining the speed of the target obstacle at each historical moment; Based on the speed of the target obstacle at each historical moment, adjusting the initial detection direction of the target obstacle at each historical moment to obtain the target detection direction of the target obstacle at each historical moment; The detected position and target detection orientation of the target obstacle at each historical moment are used as the detected posture of the target obstacle at each historical moment.

27. An electronic device, include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 13.

28. A non-transitory computer-readable storage medium storing computer instructions, in, The computer instructions are used to cause the computer to execute the method according to any one of claims 1-13.

29. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 13.

Images