Road surface element reconstruction method and device, electronic equipment and storage medium

Abstract

A road surface element reconstruction method and device, electronic equipment and storage medium are disclosed. The method comprises: obtaining an environment image set, the environment image set comprising: a plurality of image acquisition devices arranged at different positions of a movable device collecting a plurality of frames of images at different angles for the environment around the movable device at the current time, each frame of image in the environment image set containing at least one road surface target; based on the environment image set, generating a first feature map containing semantic segmentation information in a bird's eye view space via a neural network; based on the semantic segmentation information, determining the respective pixel regions of each road surface target in the first feature map; mapping the respective pixel regions of each road surface target in the first feature map to a corresponding preset coordinate system of the movable device to obtain a second feature map; performing multi-target tracking on the second feature map to obtain a tracking result; and based on the tracking result, generating a road surface element reconstruction result. The disclosure can ensure the integrity of the road surface element reconstruction result.

Description

Technical Field

[0001] This disclosure relates to driving technology, and in particular to a method, apparatus, electronic device, and storage medium for reconstructing road features. Background Technology

[0002] Environmental perception is a key research area in fields such as autonomous driving and scene understanding. In some cases, there is a need to reconstruct road features when performing environmental perception. Current road feature reconstruction methods only reconstruct and output road targets in front of the forward-looking camera, and cannot reconstruct road targets outside the field of view of the forward-looking camera. Summary of the Invention

[0003] To address the problem that current road feature reconstruction methods cannot reconstruct road targets outside the field of view of a forward-looking camera, thus affecting the integrity of the road feature reconstruction results, this disclosure is proposed. Embodiments of this disclosure provide a road feature reconstruction method, apparatus, electronic device, and storage medium.

[0004] According to one aspect of the present disclosure, a method for reconstructing road surface features is provided, comprising: Acquire an environmental image set, which includes: multiple frames of images from different perspectives of the environment around the mobile device acquired at the current moment by multiple image acquisition devices set at different locations of the mobile device, and each frame of the environmental image set contains at least one road surface target. Based on the environmental image set, a first feature map containing semantic segmentation information in a bird's-eye view space is generated via a neural network; Based on the semantic segmentation information, the pixel regions of each road surface target in the first feature map are determined; The pixel regions of each road surface target in the first feature map are mapped to the preset coordinate system corresponding to the mobile device to obtain the second feature map; Multi-target tracking is performed on the second feature map to obtain the tracking results; Based on the tracking results, road surface feature reconstruction results are generated.

[0005] According to another aspect of the present disclosure, a road surface feature reconstruction apparatus is provided, comprising: The first acquisition module is used to acquire an environmental image set, which includes: multiple frames of images from different perspectives of the environment around the mobile device acquired at the current moment by multiple image acquisition devices set at different locations of the mobile device, and each frame of the environmental image set contains at least one road surface target. The first generation module is used to generate a first feature map containing semantic segmentation information in a bird's-eye view space based on the environmental image set obtained by the first acquisition module via a neural network. The determining module is used to determine the pixel region of each of the road surface targets in the first feature map based on the semantic segmentation information contained in the first feature map generated by the first generating module. The second acquisition module is used to map the pixel regions of each road surface target in the first feature map determined by the determining module to the preset coordinate system corresponding to the mobile device, so as to obtain the second feature map. A multi-target tracking module is used to perform multi-target tracking on the second feature map obtained by the second acquisition module to obtain tracking results; The second generation module is used to generate road surface feature reconstruction results based on the tracking results obtained by the multi-target tracking module.

[0006] According to another aspect of the present disclosure, a computer-readable storage medium is provided, the storage medium storing a computer program for performing the above-described road surface element reconstruction method.

[0007] According to another aspect of the present disclosure, an electronic device is provided, comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the above-described road surface feature reconstruction method.

[0008] Based on the road surface feature reconstruction method, apparatus, electronic device, and storage medium provided in the above embodiments of this disclosure, a first feature map containing semantic segmentation information in a bird's-eye view can be generated via a neural network from an environmental image set including multiple frames of images from different perspectives acquired by multiple image acquisition devices located at different positions on a mobile device. A second feature map can be generated based on the first feature map, and a road surface feature reconstruction result can be generated based on the tracking results obtained from multi-target tracking of the second feature map. Thus, the input for road surface feature reconstruction is not a single forward-looking camera observation, but rather multi-view observations based on multiple image acquisition devices. The road surface features are ultimately reconstructed in a bird's-eye view, thus the field of view corresponding to the road surface feature reconstruction result can be considered the entire field of view. The field of view corresponding to the road surface feature reconstruction result is broader, and the integrity of the road surface feature reconstruction result can be better guaranteed.

[0009] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0010] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to offer a further understanding of the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0011] Figure 1 This is a schematic diagram illustrating the scenario to which this disclosure applies.

[0012] Figure 2 This is a schematic flowchart of a road surface element reconstruction method provided in an exemplary embodiment of this disclosure.

[0013] Figure 3 This is a schematic diagram of the road surface element reconstruction result in an exemplary embodiment of this disclosure.

[0014] Figure 4 This is a schematic flowchart of a road surface feature reconstruction method provided in another exemplary embodiment of this disclosure.

[0015] Figure 5 This is a schematic flowchart of a road surface element reconstruction method provided in another exemplary embodiment of this disclosure.

[0016] Figure 6 This is a schematic flowchart of a road surface element reconstruction method provided in yet another exemplary embodiment of this disclosure.

[0017] Figure 7 This is a schematic flowchart of a road surface element reconstruction method provided in yet another exemplary embodiment of this disclosure.

[0018] Figure 8 This is a schematic diagram of the structure of a road surface element reconstruction device provided in an exemplary embodiment of this disclosure.

[0019] Figure 9 This is a schematic diagram of the structure of a road surface element reconstruction device provided in another exemplary embodiment of this disclosure.

[0020] Figure 10 This is a schematic diagram of the structure of a road surface element reconstruction device provided in another exemplary embodiment of this disclosure.

[0021] Figure 11 This is a schematic diagram of the structure of a road surface element reconstruction device provided in yet another exemplary embodiment of this disclosure.

[0022] Figure 12 This is a structural diagram of an electronic device provided in an exemplary embodiment of this disclosure. Detailed Implementation

[0023] Hereinafter, exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present disclosure, and not all embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein.

[0024] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0025] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0026] It should also be understood that in the embodiments disclosed herein, "multiple" can refer to two or more, and "at least one" can refer to one, two or more.

[0027] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0028] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship.

[0029] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0030] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0031] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0032] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0033] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0034] The embodiments disclosed herein can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0035] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0036] Application Overview In some cases, vehicles equipped with Advanced Driving Assistance Systems (ADAS) require road feature reconstruction.

[0037] In the process of realizing this disclosure, the inventors discovered that the current road surface feature reconstruction method is based on traditional camera sensors, which can only reconstruct and output road surface targets in front of the forward-looking camera installed on the vehicle, and cannot reconstruct road surface targets outside the field of view of the forward-looking camera. As a result, the field of view corresponding to the road surface feature reconstruction result is only the field of view of the forward-looking camera, the field of view corresponding to the road surface feature reconstruction result is narrow, and the integrity of the road surface feature reconstruction result cannot be guaranteed.

[0038] Exemplary System like Figure 1 As shown, the schematic diagram of the scenario to which this disclosure applies may include: a vehicle 12, an electronic device 14, and multiple cameras 16; wherein, the electronic device 14 may be installed on the vehicle 12, or the electronic device 14 may not be installed on the vehicle 12, but may communicate remotely with the vehicle 12; the multiple cameras 16 may be installed at different locations on the vehicle 12.

[0039] In practice, multiple cameras 16 can periodically collect environmental images. The images collected by each of the multiple cameras 16 can be provided to the electronic device 14. The electronic device 14 can execute the road surface element reconstruction method in the embodiments of this disclosure based on the images provided by the multiple cameras 16 to obtain road surface element reconstruction results with a wider field of view, thereby ensuring the integrity of the road surface element reconstruction results.

[0040] Exemplary methods Figure 2 This is a schematic flowchart of a road surface element reconstruction method provided in an exemplary embodiment of this disclosure. Figure 2 The method shown includes steps 210, 220, 230, 240, 250 and 260, which are explained below.

[0041] Step 210: Obtain an environmental image set, which includes multiple frames of images from different perspectives of the environment around the mobile device, collected at the current moment by multiple image acquisition devices located at different positions of the mobile device. Each frame of the environmental image set contains at least one road surface target.

[0042] Alternatively, the mobile device can be a vehicle, for example... Figure 1 Vehicle 12; the image acquisition device can be a camera, for example... Figure 1 The camera 16 in the mobile device; the number of multiple image acquisition devices set in different positions of the mobile device can be four. The four image acquisition devices can be set in front of the mobile device, the left front, the right front, and the rear of the mobile device respectively. Of course, the number of multiple image acquisition devices set in different positions of the mobile device can also be three, five or more.

[0043] It should be noted that the road surface targets involved in the embodiments of this disclosure are all static road surface elements, including but not limited to zebra crossings, stop lines, road direction arrows, etc.

[0044] Step 220: Based on the environmental image set, generate a first feature map containing semantic segmentation information in the Bird's Eye View (BEV) space via a neural network.

[0045] In step 220, the environmental image set obtained in step 210 can be provided as input to the neural network, which can then perform calculations to generate a first feature map containing semantic segmentation information in the bird's-eye view space. The semantic segmentation information can include the category of each pixel in the first feature map, and the categories of the pixels include, but are not limited to, zebra crossing category, stop line category, road direction arrow category, etc. The first feature map can also be called the BEV space segmentation result or BEV observation.

[0046] Step 230: Based on semantic segmentation information, determine the pixel regions of each road surface target in the first feature map.

[0047] Since the semantic segmentation information includes the category of each pixel in the first feature map, in step 230, the pixel region of each road surface target in the first feature map can be extracted efficiently and quickly by referring to the semantic segmentation information; wherein, the category of each pixel in the pixel region corresponding to any road surface target is the category corresponding to that road surface target.

[0048] Step 240: Map the pixel regions of each road surface target in the first feature map to the preset coordinate system corresponding to the mobile device to obtain the second feature map.

[0049] For ease of understanding, the embodiments of this disclosure are all described using the case where the preset coordinate system corresponding to the mobile device is the vehicle coordinate system.

[0050] It should be noted that a transformation relationship between the bird's-eye view coordinate system and the vehicle coordinate system can be established in advance. In step 240, the pixel regions of each road surface target in the first feature map can be mapped to a blank feature map of a predetermined size in the vehicle coordinate system using the pre-established transformation relationship, thereby forming a second feature map. The second feature map may include the rectangular boxes of each road surface target.

[0051] Step 250: Perform multiple object tracking (MOT) on the second feature map to obtain the tracking results.

[0052] It should be noted that in the embodiments of this disclosure, environmental image sets can be acquired periodically. After acquiring each environmental image set, a corresponding first feature map and a second feature map can be generated for that environmental image set. This allows for the periodic acquisition of the second feature map, thus forming a sequence including multiple frames of second feature maps. In step 250, a multi-target tracking algorithm can be used to perform multi-target tracking on the sequence including multiple frames of second feature maps. Specifically, the relationship between road surface targets in consecutive frames is clarified, the same road surface target in consecutive frames is associated, and a unique tracking ID is assigned, thereby obtaining the tracking result.

[0053] Step 260: Based on the tracking results, generate road surface feature reconstruction results.

[0054] Optionally, the rendering of the road surface feature reconstruction results can be found in [reference needed]. Figure 3The road surface feature reconstruction results can be used to present information such as the location and size of each road surface target in the environment surrounding the mobile device.

[0055] In the embodiments of this disclosure, an environmental image set comprising multiple frames of images from different perspectives acquired by multiple image acquisition devices positioned at different locations on a mobile device can be used. A first feature map containing semantic segmentation information in a bird's-eye view can be generated via a neural network. A second feature map can be generated based on the first feature map. Based on the tracking results obtained from multi-target tracking of the second feature map, a road surface element reconstruction result can be generated. Thus, the input for road surface element reconstruction is not a single forward-looking camera observation, but rather multi-view observations from multiple image acquisition devices. The road surface elements are ultimately reconstructed in a bird's-eye view, resulting in a field of view corresponding to the road surface element reconstruction result that can be considered the entire field of view. This broader field of view ensures better completeness of the road surface element reconstruction result.

[0056] exist Figure 2 Based on the illustrated embodiments, as Figure 4 As shown, before step 260, the method further includes step 252.

[0057] Step 252: Obtain the first target feature map sequence corresponding to multiple historical environmental image sets. The multiple historical environmental image sets represent multiple environmental image sets collected at multiple historical moments by multiple image acquisition devices set at different locations of the mobile device. The first target feature map sequence includes: second feature maps of multiple historical frames corresponding to the multiple historical environmental image sets.

[0058] As described above, in the embodiments of this disclosure, environmental image sets can be acquired periodically. At the current moment, a corresponding environmental image set can be acquired, which can be called the current environmental image set, and the second feature map corresponding to the current environmental image set can be called the current second feature map. Similarly, at each of multiple historical moments (i.e., moments other than the current moment), a corresponding environmental image set can be acquired, which can be called a historical environmental image set, and the second feature map corresponding to the historical environmental image set can be called a historical second feature map. Furthermore, the second feature maps of multiple historical frames, each corresponding one-to-one with multiple historical environmental image sets, can be sequentially arranged to form a first target feature map sequence.

[0059] Step 260 may include steps 2601, 2603, 2605, 2607 and 2609.

[0060] Step 2601: Based on the tracking results corresponding to the first target feature map sequence, determine the first tracked road surface target.

[0061] It should be noted that by performing multi-target tracking on the first target feature map sequence, the tracking results corresponding to the first target feature map sequence can be obtained. Based on the obtained tracking results, it can be determined which road targets have been successfully tracked. Each of these successfully tracked road targets can be considered as a first road target. Since the subsequent processing methods for each first road target are similar, the following mainly introduces the processing method for a single first road target.

[0062] Step 2603: Take the first state variable of the second feature map corresponding to the last frame of the first target feature map sequence as the reference state variable, and take the first uncertainty corresponding to the first state variable as the reference uncertainty.

[0063] Optionally, the state variables of any road surface target may include: the three-dimensional coordinates of the center point of the rectangular frame of the road surface target, as well as the length, width, and yaw angle of the rectangular frame of the road surface target.

[0064] Optionally, the uncertainty corresponding to any state variable can be used to characterize the degree of reliability of that state variable, and any uncertainty can be characterized using the covariance matrix.

[0065] Step 2605: Based on the reference state variable, reference uncertainty, first observation information of the first road surface target in the current second feature map, and observation noise, determine the second state variable of the first road surface target corresponding to the current second feature map, and the second uncertainty of the second state variable.

[0066] Optionally, the first observation information of the first road surface target in the current second feature map may include: the three-dimensional coordinates of the four vertices of the detection box of the first road surface target in the current second feature map; the observation noise may refer to the noise present when observing the current second feature map.

[0067] In one specific embodiment, step 2605 includes: The first observation information is transformed into the world coordinate system to obtain the second observation information; Determine the target Jacobian matrix of the second observation information relative to the state variables; The Kalman gain is determined based on the reference uncertainty, the target Jacobian matrix, and the observation noise. The second state variable is determined based on the reference state variable, Kalman gain, second observation information, and target Jacobian matrix; The second uncertainty is determined based on the identity matrix, Kalman gain, target Jacobian matrix, and reference uncertainty.

[0068] It should be noted that the transformation relationship between the vehicle coordinate system and the world coordinate system can be predetermined, specifically in the form of a transformation matrix. The first observation information can be considered as observation information in the vehicle coordinate system. After obtaining the first observation information, this transformation matrix can be used to transform the first observation information from the vehicle coordinate system to the world coordinate system to obtain the second observation information. In a specific example, the first observation information includes four three-dimensional coordinates. By multiplying each of the four three-dimensional coordinates by the transformation matrix, four additional three-dimensional coordinates corresponding to the four original three-dimensional coordinates can be obtained. These additional four three-dimensional coordinates can then form the second observation information.

[0069] After obtaining the second observation information, the target Jacobian moment of the second observation information relative to the state variables can be determined. The target Jacobian matrix can be used to characterize the gradient information of the second observation information relative to the state variables.

[0070] Optionally, determining the target Jacobian matrix of the second observation information relative to the state variables includes: Based on the second observation information, the observation three-dimensional coordinates of the top left, top right, bottom left, and bottom right vertices of the detection box of the first road surface target are determined to obtain four observation three-dimensional coordinates; Based on the four observed three-dimensional coordinates, the length, width, and yaw angle of the detection frame for the first road surface target are determined; Based on length, width, yaw angle, and rotation matrix between preset coordinate system and world coordinate system, determine the first Jacobian matrix corresponding to the top left vertex, the second Jacobian matrix corresponding to the top right vertex, the third Jacobian matrix corresponding to the bottom left vertex, and the fourth Jacobian matrix corresponding to the bottom right vertex. The target Jacobian matrix is ​​determined based on the first Jacobian matrix, the second Jacobian matrix, the third Jacobian matrix, and the fourth Jacobian matrix.

[0071] It should be noted that the detection boxes involved in the embodiments of this disclosure can all be rectangular boxes.

[0072] Assume the three-dimensional coordinates of the top-left vertex in the second observation information are represented as follows: The three-dimensional coordinates of the upper right vertex are represented as follows: The three-dimensional coordinates of the lower left vertex are represented as follows: The three-dimensional coordinates of the lower right vertex are represented as follows: Then, four observed three-dimensional coordinates can be obtained. The observed three-dimensional coordinates of the top-left vertex, top-right vertex, bottom-left vertex, and bottom-right vertex can be obtained as follows: , , , .

[0073] Next, based on the four observed three-dimensional coordinates, the length, width, and yaw angle of the detection box for the first road surface target can be determined. The formulas used to determine the length, width, and yaw angle are as follows:

[0074]

[0075]

[0076] in, l , w , θ The length, width, and yaw angle of the detection frame for the first road surface target are represented in sequence.

[0077] Optionally, based on the four observed three-dimensional coordinates, the three-dimensional coordinates of the center point of the detection frame of the first road surface target can also be determined. The formula used to determine the three-dimensional coordinates of the center point of the detection frame of the first road surface target is as follows:

[0078]

[0079]

[0080] in, x , y , z The three-dimensional coordinates of the center point of the detection frame of the first road surface target are represented sequentially. x coordinate, y coordinate, z coordinate.

[0081] Next, based on the determined length, width, yaw angle, and rotation matrix between the vehicle coordinate system and the world coordinate system, we can determine the first Jacobian matrix corresponding to the top left vertex, the second Jacobian matrix corresponding to the top right vertex, the third Jacobian matrix corresponding to the bottom left vertex, and the fourth Jacobian matrix corresponding to the bottom right vertex.

[0082] Optionally, the transformation matrix between the vehicle coordinate system and the world coordinate system can be predetermined, and the transformation matrix between the vehicle coordinate system and the world coordinate system can be expressed in the following form:

[0083] Among them, T wv T represents the transformation matrix between the vehicle coordinate system and the world coordinate system. wv The nine elements in the upper left corner form the rotation matrix between the vehicle coordinate system and the world coordinate system. t 1 represents the translation vector in the x-direction. t2 represents the translation vector in the y-direction. t 3 represents the translation vector in the z-direction. 。

[0084] Optionally, the first Jacobian matrix is ​​represented as ,but:

[0085] =﹣sin( θ )·( r 11 · l / 2+ r 12 · w / 2)+ cos( θ )·( r 12 · l / 2- r 11 · w / 2) =[( r 11 ·cos( θ )+ r 12 sin( θ )] / 2 =[( r 12 ·cos( θ ) - r 11 sin( θ )] / 2 =﹣sin( θ )·( r 21 · l / 2+ r 22 · w / 2)+ cos( θ )·( r 22 · l / 2- r 21 · w / 2) =[( r 21 ·cos( θ )+ r 22 sin( θ)] / 2 =[( r 22 ·cos( θ ) - r 21 sin( θ )] / 2 =﹣sin( θ )·( r 31 · l / 2+ r 32 · w / 2)+ cos( θ )·( r 32 · l / 2- r 31 · w / 2) =[( r 31 ·cos( θ )+ r 32 sin( θ )] / 2 =[( r 32 ·cos( θ ) - r 31 sin( θ )] / 2 Optionally, the second Jacobian matrix is ​​represented as ,but:

[0086] =﹣sin( θ )·( r 11 · l / 2- r 12 · w / 2)+ cos( θ )·( r 12 · l / 2+ r 11 · w / 2) =[( r 11 ·cos( θ )+ r 12 ·without( θ )] / 2 =[(﹣ r 12 ·cos( θ )+ r 11 ·without( θ )] / 2 =﹣sin( θ )·( r 21 · l / 2- r 22 · w / 2)+ cos( θ )·( r 22 · l / 2+ r 21 · w / 2) =[( r 21 ·cos( θ )+ r 22 ·without( θ )] / 2 =[(﹣ r 22 ·cos( θ ) + r 21 ·without( θ )] / 2 =﹣sin( θ )·( r 31 · l / 2- r 32 · w / 2)+ cos( θ )·( r 32 · l / 2+ r 31 · w / 2) =[( r 31 ·cos( θ )+r 32 sin( θ )] / 2 =[(﹣ r 32 ·cos( θ ) + r 31 sin( θ )] / 2 Optionally, the third Jacobian matrix is ​​represented as ,but:

[0087] =﹣sin( θ )·(﹣ r 11 · l / 2+ r 12 · w / 2)+ cos( θ )·(﹣ r 12 · l / 2- r 11 · w / 2) =﹣( r 11 ·cos( θ )+ r 12 sin( θ )] / 2 =[( r 12 ·cos( θ ) - r 11 sin( θ )] / 2 =﹣sin( θ )·(﹣ r 21 · l / 2+ r 22 · w / 2)+ cos( θ )·(﹣ r 22 · l / 2- r 21 · w / 2) =﹣( r 21 ·cos( θ )+ r 22 sin( θ )] / 2 =[( r 22 ·cos( θ ) - r 21 sin( θ )] / 2 =﹣sin( θ )·(﹣ r 31 · l / 2+ r 32 · w / 2)+ cos( θ )·(﹣ r 32 · l / 2- r 31 · w / 2) =﹣( r 31 ·cos( θ )+ r 32 sin( θ )] / 2 =[( r 32 ·cos( θ ) - r 31 sin( θ )] / 2 Optionally, the fourth Jacobian matrix is ​​represented as ,but:

[0088] =﹣sin( θ )·(﹣ r 11 · l / 2- r 12 · w / 2)+ cos( θ )·(﹣ r 12 · l / 2+ r 11 · w / 2) =﹣( r 11 ·cos( θ )+ r 12 ·without( θ )] / 2 =[(﹣ r 12 ·cos( θ )+ r 11 ·without( θ )] / 2 =﹣sin( θ )·(﹣ r 21 · l / 2- r 22 · w / 2)+ cos( θ )·(﹣ r 22 · l / 2+ r 21 · w / 2) =﹣( r 21 ·cos( θ )+ r 22 ·without( θ )] / 2 =[(﹣ r 22 ·cos( θ ) + r 21 ·without( θ )] / 2 =﹣sin( θ )·(﹣ r 31 · l / 2- r 32 · w / 2)+ cos( θ )·(﹣ r 32 · l / 2+ r31 · w / 2) =﹣( r 31 ·cos( θ )+ r 32 sin( θ )] / 2 =[(﹣ r 32 ·cos( θ ) + r 31 sin( θ )] / 2 Among the formulas involved, l Indicates length, w Indicates width, θ Indicates the yaw angle. , , , , , , , For calculation Intermediate variables at time, For calculation Intermediate variables at time, , , , , , , , For calculation Intermediate variables at time, For calculation Intermediate variables at that time.

[0089] In obtaining The target Jacobian matrix can then be determined using the following formula. :

[0090] In this implementation, based on the second observation information, four observation three-dimensional coordinates can be obtained efficiently and reliably. Based on the four observation three-dimensional coordinates, the length, width, and yaw angle of the first road surface target can be determined efficiently and reliably. By substituting the determined length, width, and yaw angle into the corresponding formulas for calculation, the first to fourth Jacobian matrices can be determined efficiently and reliably. Thus, the target Jacobian matrix can be obtained efficiently and reliably, so that the target Jacobian matrix can be used in subsequent steps.

[0091] Specifically, the Kalman gain can be determined based on the reference uncertainty, the target Jacobian matrix, and the observation noise; the second state variable can be determined based on the reference state variable, the Kalman gain, the second observation information, and the target Jacobian matrix; and the second uncertainty can be determined based on the identity matrix, the Kalman gain, the target Jacobian matrix, and the reference uncertainty.

[0092] Alternatively, any uncertainty can be characterized using the covariance matrix, and the formula for determining the Kalman gain can be:

[0093] in, K Indicates Kalman gain, P k-1 Indicates reference uncertainty. H Describe the target Jacobian matrix. R k This indicates observation noise.

[0094] Alternatively, the formula used to determine the second state variable can be:

[0095] in, x k Represents the second state variable. x k-1 Indicates the reference state variable. K Indicates Kalman gain, y k This indicates the second observation information. H Let represent the target Jacobian matrix.

[0096] Optionally, any uncertainty is characterized using the covariance matrix, and the formula used to determine the second uncertainty is: P k = ( I-KH ) P k-1 in, P k This indicates the second uncertainty. I Represents the identity matrix. K Indicates Kalman gain, H Describe the target Jacobian matrix. P k-1 This indicates the reference uncertainty.

[0097] It is evident that, given that the reference uncertainty, target Jacobian matrix, observation noise, reference state variable, and second observation information are all known, the Kalman gain, second state variable, and second uncertainty can be determined efficiently and reliably simply by substituting the relevant information into the corresponding formulas for calculation.

[0098] Step 2607: In response to the second uncertainty satisfying the preset convergence condition, the road surface element reconstruction result is generated based on the second state variable.

[0099] Step 2609: In response to the second uncertainty not meeting the preset convergence condition, the reference state variable is updated to the second state variable, and the reference uncertainty is updated to the second uncertainty.

[0100] After determining the second state variable and the second uncertainty, the second uncertainty can be compared with the preset uncertainty.

[0101] If the second uncertainty does not exceed the preset uncertainty, it can be determined that the second uncertainty meets the preset convergence condition, and then the road surface element reconstruction result can be generated based on the second state variable. Optionally, the location, size, and other information of the first road surface target presented in the road surface element reconstruction result can be determined based on the second state variable.

[0102] If the second uncertainty exceeds the preset uncertainty, it can be determined that the second uncertainty does not meet the preset convergence condition. In this case, the reference state variable is updated to the second state variable, and the reference uncertainty is updated to the second uncertainty. Next, the second feature map at the current time will become the second feature map of a historical frame. The second feature map of that historical frame will be added to the first target feature map sequence. Simultaneously, a new current second feature map will be obtained. From this, a new second state variable and a new second uncertainty can be obtained again, and based on this, it can be determined whether the new second uncertainty meets the preset convergence condition. Subsequent processes follow the same logic.

[0103] In one example, there are five historical moments: T1, T2, T3, T4, and T5. The current moment is T6. The environmental image sets corresponding to T1, T2, T3, T4, T5, and T6 are Q1, Q2, Q3, Q4, Q5, and Q6, respectively. The second feature maps corresponding to T1, T2, T3, T4, T5, and T6 are Z1, Z2, Z3, Z4, Z5, and Z6, respectively.

[0104] After obtaining Z6, it can be determined whether the state variable corresponding to Z5 of the first road surface target and the covariance matrix corresponding to the state variable of the first road surface target corresponding to Z5 have been determined.

[0105] If the state variables corresponding to the first road surface target in Z5, and the covariance matrix corresponding to the state variables of the first road surface target in Z5, have not been determined beforehand, then the state variables and covariance matrix can be initialized. Optionally, this can be based on the observation information of the first road surface target in Z6, and referring to the calculation method described above. x , y , z , l , w , θ The formula determines the corresponding state variables for the first road surface target. These state variables can be used as the state variables of the first road surface target corresponding to Z6. In addition, the noise level of the initial state variables can be calculated based on multiple experiments to initialize the covariance matrix. This covariance matrix can then be used as the covariance matrix of the state variables of the first road surface target corresponding to Z6.

[0106] If the state variables of the first road surface target corresponding to Z5 and the covariance matrix corresponding to the state variables of the first road surface target corresponding to Z5 have been determined beforehand, then the state variables of the first road surface target corresponding to Z6 and the covariance matrix corresponding to the state variables of the first road surface target corresponding to Z5 can be determined based on the state variables of the first road surface target corresponding to Z5, the covariance matrix corresponding to the state variables of the first road surface target corresponding to Z5, the observation information and observation noise of the first road surface target in Z6.

[0107] If the covariance matrix corresponding to the state variable of the first road surface target Z6 does not exceed the threshold (equivalent to the second uncertainty in the above text satisfying the preset convergence condition), then the road surface element reconstruction result can be generated based on the state variable of the first road surface target Z6.

[0108] If the covariance matrix corresponding to the state variable of the first road surface target in Z6 exceeds the threshold (equivalent to the second uncertainty in the above text not meeting the preset convergence condition), then the environmental image set Q7 corresponding to the next time T7 of T6 can be obtained, and the second feature map Z7 corresponding to Q7 can be obtained. Based on the state variable of the first road surface target in Z6, the covariance matrix corresponding to the state variable of the first road surface target in Z6, the observation information and observation noise of the first road surface target in Z7, the state variable of the first road surface target in Z7 and the covariance matrix corresponding to the state variable of the first road surface target in Z7 are determined.

[0109] If the covariance matrix corresponding to the state variable of the first road surface target Z7 does not exceed the threshold, the road surface feature reconstruction result can be generated based on the state variable of the first road surface target Z7.

[0110] If the covariance matrix corresponding to the state variable of the first road surface target Z7 exceeds the threshold, the environmental image set Q8 corresponding to the next time step T8 of T7 can be obtained. The subsequent process is similar to the processing after obtaining Q7, and will not be described in detail here.

[0111] In the embodiments of this disclosure, by introducing the Extended Kalman Filter (EKF) algorithm, the state variables of the tracked road surface targets can be updated and optimized in real time until convergence, thereby obtaining high-precision road surface feature reconstruction results. Furthermore, a rectangular bounding box model (i.e., each detection box is a rectangle) can be selected during filtering, thereby achieving rapid convergence of the state variables.

[0112] exist Figure 2 Based on the illustrated embodiments, as Figure 5 As shown, before step 250, the method further includes step 242.

[0113] Step 242: Obtain the first target feature map sequence corresponding to multiple historical environmental image sets. The multiple historical environmental image sets represent multiple environmental image sets collected at multiple historical moments by multiple image acquisition devices set at different locations of the mobile device. The first target feature map sequence includes: second feature maps of multiple historical frames corresponding to the multiple historical environmental image sets.

[0114] It should be noted that the specific implementation process of step 242 can be referred to the description of step 252, and will not be repeated here.

[0115] Step 250 may include steps 2501, 2503, 2505 and 2507.

[0116] Step 2501: Based on the tracking results corresponding to the first target feature map sequence, determine the tracked second road surface target.

[0117] It should be noted that the specific implementation process of step 2501 can be referred to the description of step 2601, and will not be repeated here.

[0118] Step 2503: Perform target detection on the current second feature map to determine the third road surface target in the current second feature map.

[0119] In step 2503, a target detection algorithm can be used to detect targets in the current second feature map to determine each road surface target in the current second feature map. Each of these road surface targets can be used as a third road surface target. Since the subsequent processing methods for each third road surface target are similar, the following mainly introduces the processing method for a single third road surface target.

[0120] Step 2505: In response to the fact that the third road surface target and the second road surface target are of the same category, the calculation method corresponding to the category of the third road surface target is used to calculate the cost between the third road surface target and the second road surface target.

[0121] In one specific implementation, a calculation method corresponding to the category of the third road surface target is used to calculate the cost value between the third road surface target and the second road surface target, including: In response to the fact that the category of the third road surface target is the road surface arrow category, the first predicted detection box of the second road surface target in the current second feature map is determined based on the tracking result corresponding to the first target feature map sequence; Calculate the intersection over union ratio (IoU) of the first predicted detection box and the first actual detection box of the third road surface target in the current second feature map, as well as the Euclidean distance of the first detection box; Calculate the first ratio between the cross-union ratio of the detection frames and the preset cross-union ratio; Calculate the second ratio between the Euclidean distance of the first detection frame and the preset Euclidean distance; Based on the weighted result of the first ratio and the second ratio, the cost between the third pavement target and the second pavement target is determined.

[0122] When the category of the third road surface target is the road surface arrow type, the actual detection box of the second road surface target in the second feature map of each frame in the history of the first target feature map sequence can be determined based on the tracking results corresponding to the first target feature map sequence, and the detection box of the second road surface target in the current second feature map can be predicted accordingly to determine the first predicted detection box.

[0123] Next, the intersection-union ratio (IU) and Euclidean distance of the first actual detection box and the first predicted detection box obtained by target detection of the third road surface target in the current second feature map can be calculated. The calculated IU can represent the ratio of the intersection and union of the first actual detection box and the first predicted detection box. The calculated Euclidean distance of the first detection box can be the Euclidean distance between the center point of the first actual detection box and the center point of the first predicted detection box.

[0124] Then, the intersection-union ratio of the detection boxes can be divided by the preset intersection-union ratio to obtain the first ratio, and the Euclidean distance of the first detection box can be divided by the preset Euclidean distance to obtain the second ratio.

[0125] Next, the first ratio and the second ratio can be weighted, and the cost between the third road surface target and the second road surface target can be determined based on the weighted result. Optionally, when weighting the first ratio and the second ratio, the weights corresponding to the first ratio and the second ratio can be set according to the actual situation, as long as the sum of the weights corresponding to the first ratio and the second ratio is 1. After obtaining the weighted result, the weighted result can be directly used as the cost, or the weighted result can be mapped to a specified numerical range (e.g., 0 to 1, 0 to 5, etc.), and the mapped value can be used as the cost.

[0126] It should be noted that there may be multiple third-road targets and multiple second-road targets. Therefore, when calculating the cost value for the i-th third-road target and the j-th second-road target, the following formula can be used: I ij = k 1· d ij / d max +(1- k 1) ·rect_ iou ij / rect_ iou min in, I ij This represents the cost between the i-th third pavement target and the j-th second pavement target. k 1 indicates a preset weight. d ij This represents the intersection-union ratio (IoU) of the detection boxes calculated for the i-th third pavement target and the j-th second pavement target. d max Represents the preset intersection-union ratio, rect_ iou ij Represents the Euclidean distance between the first detection bounding boxes calculated for the i-th third road surface target and the j-th second road surface target, rect_ iou min This indicates the preset Euclidean distance.

[0127] In this implementation, for road targets of the road arrow category, the corresponding cost can be calculated efficiently and reliably by calculating the cross-union ratio and Euclidean distance of the detection boxes and further combining them with weighted processing.

[0128] In another specific implementation, a calculation method corresponding to the category of the third road surface target is used to calculate the cost value between the third road surface target and the second road surface target, including: In response to the fact that the category of the third road surface target is zebra crossing or stop line, the second predicted detection box of the second road surface target in the current second feature map is determined based on the tracking result corresponding to the first target feature map sequence. Project the second actual detection box of the third road surface target in the current second feature map onto the long side of the second predicted detection box to obtain the projection line segment; Calculate the intersection-to-union ratio of the projected line segment and the line segment along the longer side; Calculate the Euclidean distance between the second predicted detection box and the second actual detection box; Calculate the third ratio of the line segment intersection-union ratio to the preset intersection-union ratio; Calculate the fourth ratio of the Euclidean distance of the second detection frame to the preset Euclidean distance; Based on the weighted result of the third ratio and the fourth ratio, the cost between the third pavement target and the second pavement target is determined.

[0129] When the category of the third road surface target is zebra crossing or stop line, the actual detection box of the second road surface target in the second feature map of each frame in the history of the first target feature map sequence can be determined based on the tracking results corresponding to the first target feature map sequence. Based on this, the detection box of the second road surface target in the current second feature map can be predicted to determine the second predicted detection box.

[0130] Next, the target detection of the third road surface target in the current second feature map is performed, and the second actual detection box is projected onto the long side of the second predicted detection box to determine the projection line segment. The intersection-union ratio of the projection line segment and the long side is calculated, and the Euclidean distance between the second predicted detection box and the second actual detection box is calculated. The calculated line segment intersection-union ratio represents the ratio of the intersection and union of the projection line segment and the long side. The calculated Euclidean distance between the second detection box can be the Euclidean distance between the center point of the second actual detection box and the center point of the second predicted detection box.

[0131] Then, the line segment intersection-union ratio can be divided by the preset intersection-union ratio to obtain the third ratio, and the Euclidean distance of the second detection box can be divided by the preset Euclidean distance to obtain the fourth ratio.

[0132] Next, the third and fourth ratios can be weighted, and the cost between the third and second road surface targets can be determined based on the weighted result. Optionally, when weighting the third and fourth ratios, the weights corresponding to each ratio can be set according to the actual situation, as long as the sum of the weights corresponding to each ratio is 1. After obtaining the weighted result, it can be directly used as the cost, or the weighted result can be mapped to a specified numerical range (e.g., 0 to 1, 0 to 5, etc.), and the mapped value can be used as the cost.

[0133] It should be noted that there may be multiple third-road targets and multiple second-road targets. Therefore, when calculating the cost value for the i-th third-road target and the j-th second-road target, the following formula can be used: I ij = k 1· d ij / d max +(1- k 1) line_ iou ij / line_ iou min in, I ij This represents the cost between the i-th third pavement target and the j-th second pavement target. k 1 indicates a preset weight. d ij This represents the intersection-union ratio of the line segments calculated for the i-th third pavement target and the j-th second pavement target. d max Indicates the preset intersection-union ratio, line_ iou ij Represents the Euclidean distance between the second detection boxes calculated for the i-th third road surface target and the j-th second road surface target, rect_ iou min This indicates the preset Euclidean distance.

[0134] In this implementation, for road surface targets of zebra crossing or stop line categories, the corresponding cost can be calculated efficiently and reliably by calculating the intersection-union ratio of line segments and the Euclidean distance of the detection box, and further combining it with weighted processing.

[0135] Step 2507: Based on the tracking results corresponding to the first target feature map sequence and the cost value, determine the tracking results corresponding to the third target feature map sequence. The third target feature map sequence includes: the second feature map of multiple frames in the first target feature map sequence, and the current second feature map.

[0136] In step 2507, given the tracking results corresponding to the first target feature map sequence and the known cost between the third road surface target and the second road surface target, the Hungarian matching algorithm can be used to perform many-to-many matching of road surface targets.

[0137] Suppose that based on the tracking results corresponding to the first target feature map sequence, five second road targets are identified, namely road target 1, road target 2, road target 3, road target 4, and road target 5. Among them, road target 1, road target 2, and road target 3 are of the road arrow category. By performing target detection on the current second feature map, two third road targets are identified, namely road target 6 and road target 7. Road target 6 is of the road arrow category. Then, the cost value between road target 6 and road target 1, road target 2, and road target 3 can be calculated respectively.

[0138] Assuming the cost between road target 6 and road target 1 is value 1, the cost between road target 6 and road target 2 is value 2, and the cost between road target 6 and road target 3 is value 3, then we can first filter out the cost values ​​that exceed the preset cost values ​​from value 1, value 2, and value 3.

[0139] If, after filtering, both cost value 1 and cost value 2 are retained, and cost value 1 is less than cost value 2, then road surface target 6 and road surface target 1 can be considered to belong to the same road surface target. If cost value 1, cost value 2, and cost value 3 are all filtered out, then road surface target 6 and road surface target 1 can be considered to not belong to the same road surface target. Furthermore, road surface target 6 is a road surface target that is different from road surface target 1, road surface target 2, and road surface target 3, and road surface target 6 is a newly appearing road surface target.

[0140] The relationship between road surface target 6 and road surface targets 1, 2, 3, 4, and 5 was clarified using the above method. Similarly, the relationship between road surface target 7 and road surface targets 1, 2, 3, 4, and 5 can also be clarified, thereby obtaining the tracking results corresponding to the third target feature map sequence.

[0141] In the embodiments of this disclosure, cost values ​​can be calculated using an adaptive method for different types of road surface elements, thereby ensuring the accuracy and reliability of the calculation results. This ensures the accuracy and reliability of the tracking results obtained from multi-target tracking, which is beneficial for obtaining high-precision road surface element reconstruction results.

[0142] exist Figure 2 Based on the illustrated embodiments, as Figure 6 As shown, the method further includes steps 270 and 280.

[0143] Step 270: Compress the road surface feature reconstruction results to obtain the compressed results; wherein, the road surface feature reconstruction results include: the optimized three-dimensional coordinates of the four vertices of the detection box of each road surface target.

[0144] It should be noted that by using the extended Kalman filter algorithm, the state variables of each road surface target can be continuously optimized and updated. Based on the state variables when the road surface target converges, the three-dimensional coordinates of the four vertices of the detection box of the road surface target can be inferred. These three-dimensional coordinates can be used as the optimized three-dimensional target, thereby obtaining the road surface element reconstruction result.

[0145] Then, the road feature reconstruction results can be compressed according to the Protobuf protocol to obtain compressed results. Protobuf is a language-independent, platform-independent, and scalable method for serializing structured data, which can be used for communication and data storage.

[0146] Step 280: Transmit the compression result to the host computer. The compression result is used by the host computer to update the environment model.

[0147] Optionally, the environment model can be a three-dimensional model.

[0148] In step 280, the compression result can be transmitted to the host computer in accordance with the Protobuf protocol. The host computer can update the environment model of the environment where the mobile device is located based on the obtained compression result. For example, the host computer can add various road targets at appropriate locations in the environment model.

[0149] In the embodiments of this disclosure, the road surface feature reconstruction results can be compressed and transmitted. In this way, the road surface feature reconstruction results require less space, and using the road surface feature reconstruction results to update the environmental model helps to ensure the accuracy and reliability of the environmental model.

[0150] In an optional example, such as Figure 7 As shown, the road surface element reconstruction process can include the following steps: BEV neural network output, perception preprocessing, multi-target tracking, filtering optimization, and vectorization output.

[0151] The BEV neural network output step can be implemented by the BEV neural network output module. The BEV neural network output module can predict multiple images (equivalent to the environmental image set mentioned above) through the neural network and output the BEV space segmentation result (equivalent to the first feature map mentioned above).

[0152] The perception preprocessing step can be implemented by the perception preprocessing module. The perception preprocessing module can extract the pixel regions of various road targets (including zebra crossings, stop lines, road arrows, etc.) in BEV observation. By transformation, the region of the corresponding road target under the vehicle coordinate system can be found, thereby obtaining the outermost rectangle of the corresponding road target (equivalent to mapping the pixel region of each road target in the first feature map above to the preset coordinate system corresponding to the mobile device to obtain the second feature map).

[0153] The multi-target tracking step can be implemented by the multi-target tracking module. The multi-target tracking module can calculate the cost between the new observation (equivalent to the third road surface target in the above text) and the existing road surface target (equivalent to the second road surface target in the above text). Through the Hungarian matching algorithm, the new observation and the existing road surface target can be associated and assigned a unique tracking ID.

[0154] The filtering optimization step can be implemented by the filtering optimization module. For successfully tracked road targets, the filtering optimization module can use new observations to optimize the existing reconstruction results. Considering real-time performance, the extended Kalman filter algorithm is used for road target optimization, and the state variable for each road target is chosen as X. = (x, y, z, l, w, θ) T ,in, (x, y, z) T The three-dimensional coordinates of the center point of the rectangle. l, w, θ These represent the length, width, and yaw angle of the rectangle, respectively. Each new observation uses the coordinates of the four vertices of the rectangle. y 1. y 2. y 3. y 4. Update the state variables and the corresponding covariance matrix. When the covariance matrix is ​​less than a certain threshold, it is considered that the road surface target has converged and the update stops.

[0155] The vectorization output step can be implemented by the vectorization output module. For road surface features, the vectorization output module can output only the coordinates of the four vertices of the rectangle and supports compression transmission using the protobuf protocol (equivalent to compressing the road surface feature reconstruction results according to the Protobuf protocol mentioned above, obtaining the compressed results, and transmitting the compressed results to the host computer). Compared with point cloud representation, this method has the advantage of occupying less space.

[0156] In summary, the road surface feature reconstruction method in the embodiments of this disclosure differs from traditional single forward-looking camera observation. Based on BEV observation, it can achieve more stable tracking and reconstruction of static road surface features and is not limited to the field of view of the forward-looking camera. For different road surface features, the cost is calculated in a matching manner during tracking association, which can obtain stable and accurate tracking results. The selection of the rectangular box model as the filter model can achieve rapid convergence of state variables.

[0157] Any of the road surface feature reconstruction methods provided in this disclosure can be executed by any suitable device with data processing capabilities, including but not limited to: terminal devices and servers. Alternatively, any of the road surface feature reconstruction methods provided in this disclosure can be executed by a processor, such as by a processor executing any of the road surface feature reconstruction methods mentioned in this disclosure by calling corresponding instructions stored in memory. Further details will not be elaborated below.

[0158] Exemplary device Figure 8 This is a schematic diagram of the structure of a road surface element reconstruction device provided in an exemplary embodiment of this disclosure. Figure 8 The device shown includes a first acquisition module 810, a first generation module 820, a determination module 830, a second acquisition module 840, a multi-target tracking module 850, and a second generation module 860.

[0159] The first acquisition module 810 is used to acquire an environmental image set, which includes: multiple frames of images from different perspectives of the environment around the mobile device acquired at the current moment by multiple image acquisition devices set at different locations of the mobile device. Each frame of the environmental image set contains at least one road surface target. The first generation module 820 is used to generate a first feature map containing semantic segmentation information in a bird's-eye view space based on the environmental image set acquired by the first acquisition module 810 via a neural network. The determining module 830 is used to determine the pixel region of each road surface target in the first feature map based on the semantic segmentation information contained in the first feature map generated by the first generating module 820. The second acquisition module 840 is used to map the pixel regions of each road surface target in the first feature map determined by the determination module 830 to a preset coordinate system corresponding to the mobile device, so as to obtain the second feature map. The multi-target tracking module 850 is used to perform multi-target tracking on the second feature map obtained by the second acquisition module 840 to obtain the tracking result; The second generation module 860 is used to generate road surface element reconstruction results based on the tracking results obtained by the multi-target tracking module 850.

[0160] In an optional example, such as Figure 9 As shown, the device also includes: The third acquisition module 852 is used to acquire a first target feature map sequence corresponding to multiple historical environmental image sets before generating road surface element reconstruction results based on the tracking results obtained by the multi-target tracking module 850. The multiple historical environmental image sets represent multiple environmental image sets acquired by multiple image acquisition devices set at different locations of the mobile device at multiple historical moments. The first target feature map sequence includes: second feature maps of multiple historical frames corresponding to the multiple historical environmental image sets. The second generation module 860 includes: The first determining submodule 8601 is used to determine the first tracked road surface target based on the tracking result corresponding to the first target feature map sequence obtained by the multi-target tracking module 850. The second determining submodule 8603 is used to take the first state variable of the second feature map corresponding to the last frame of the first target feature map sequence, which is determined by the first determining submodule 8601, as a reference state variable, and take the first uncertainty corresponding to the first state variable as a reference uncertainty. The third determining submodule 8605 is used to determine the second state variable of the first road surface target corresponding to the current second feature map, and the second uncertainty of the second state variable, based on the reference state variable determined by the second determining submodule 8603, the reference uncertainty determined by the second determining submodule 8603, the first observation information of the first road surface target in the current second feature map, and the observation noise. The generation submodule 8607 is used to generate road surface element reconstruction results based on the second state variable determined by the third determination submodule 8605 in response to the second uncertainty determined by the third determination submodule 8605 satisfying the preset convergence condition. The update submodule 8609 is used to update the reference state variable to the second state variable and the reference uncertainty to the second uncertainty in response to the second uncertainty determined by the third determination submodule 8605 not satisfying the preset convergence condition.

[0161] In an optional example, the third determining submodule 8605 includes: The transformation unit is used to transform the first observation information to the world coordinate system to obtain the second observation information; The first determining unit is used to determine the target Jacobian matrix of the second observation information obtained by the transformation unit relative to the state variables; The second determining unit is used to determine the Kalman gain based on the reference uncertainty determined by the second determining submodule 8603, the target Jacobian matrix determined by the first determining unit, and the observation noise. The third determining unit is used to determine the second state variable based on the reference state variable determined by the second determining submodule 8603, the Kalman gain determined by the second determining unit, the second observation information obtained by the conversion unit, and the target Jacobian matrix determined by the first determining unit. The fourth determining unit is used to determine the second uncertainty based on the identity matrix, the Kalman gain determined by the second determining unit, the target Jacobian matrix determined by the first determining unit, and the reference uncertainty determined by the second determining submodule 8603.

[0162] In one optional example, the first determined unit includes: The first determining subunit is used to determine the observation three-dimensional coordinates of the top left, top right, bottom left, and bottom right vertices of the detection box of the first road surface target based on the second observation information obtained by the transformation unit, so as to obtain four observation three-dimensional coordinates. The second determining subunit is used to determine the length, width, and yaw angle of the detection frame of the first road surface target based on the four observed three-dimensional coordinates obtained from the first determining subunit. The third determining sub-unit is used to determine the first Jacobian matrix corresponding to the top left vertex, the second Jacobian matrix corresponding to the top right vertex, the third Jacobian matrix corresponding to the bottom left vertex, and the fourth Jacobian matrix corresponding to the bottom right vertex based on the length, width, yaw angle determined by the second determining sub-unit and the rotation matrix between the preset coordinate system and the world coordinate system. The fourth determining subunit is used to determine the target Jacobian matrix based on the first Jacobian matrix, the second Jacobian matrix, the third Jacobian matrix, and the fourth Jacobian matrix determined by the third determining subunit.

[0163] In one optional example, any uncertainty is characterized using the covariance matrix, and the formula for determining the Kalman gain is:

[0164] in, K Indicates Kalman gain, P k-1 Indicates reference uncertainty. H Describe the target Jacobian matrix. R k This indicates observation noise.

[0165] In one optional example, the formula used to determine the second state variable is:

[0166] in, x k Represents the second state variable. x k-1 Indicates the reference state variable. K Indicates Kalman gain, y k This indicates the second observation information. H Let represent the target Jacobian matrix.

[0167] In one optional example, any uncertainty is characterized using the covariance matrix, and the formula used to determine the second uncertainty is: P k = ( I-KH ) P k-1 in, P k This indicates the second uncertainty. I Represents the identity matrix. K Indicates Kalman gain,H Describe the target Jacobian matrix. P k-1 This indicates the reference uncertainty.

[0168] In an optional example, such as Figure 10 As shown, the device also includes: The fourth acquisition module 842 is used to acquire a first target feature map sequence corresponding to multiple historical environmental image sets before performing multi-target tracking on the second feature map obtained by the second acquisition module 840 and obtaining the tracking result. The multiple historical environmental image sets represent multiple environmental image sets acquired by multiple image acquisition devices set at different locations of the mobile device at multiple historical moments. The first target feature map sequence includes: the second feature map of multiple frames corresponding to the multiple historical environmental image sets. Multi-target tracking module 850 includes: The fourth determining submodule 8501 is used to determine the tracked second road surface target based on the tracking result corresponding to the first target feature map sequence obtained by the fourth acquisition module 842; The fifth determination submodule 8503 is used to perform target detection on the current second feature map and determine the third road surface target in the current second feature map; The calculation submodule 8505 is used to calculate the cost between the third road surface target and the second road surface target in response to the fact that the third road surface target determined by the fifth determination submodule 8503 is of the same category as the second road surface target determined by the fourth determination submodule 8501, using a calculation method corresponding to the category of the third road surface target. The sixth determining submodule 8507 is used to determine the tracking result corresponding to the third target feature map sequence based on the tracking result corresponding to the first target feature map sequence obtained by the fourth acquisition module 842 and the cost value calculated by the calculation submodule 8505. The third target feature map sequence includes: the second feature map of multiple frames in the first target feature map sequence, and the current second feature map.

[0169] In an optional example, computation submodule 8505 includes: The fifth determining unit is used to determine the first predicted detection box of the second road surface target in the current second feature map based on the tracking result corresponding to the first target feature map sequence obtained by the fourth obtaining module 842, in response to the fifth determining submodule 8503 determining the category of the third road surface target as the road surface arrow category. The first calculation unit is used to calculate the cross-union ratio and Euclidean distance of the first predicted detection box determined by the fifth determination unit, and the first actual detection box of the third road surface target in the current second feature map. The second calculation unit is used to calculate the first ratio between the cross-union ratio of the detection frames calculated by the first calculation unit and the preset cross-union ratio. The third calculation unit is used to calculate the second ratio between the Euclidean distance of the first detection frame calculated by the first calculation unit and the preset Euclidean distance; The sixth determining unit is used to determine the cost between the third pavement target and the second pavement target based on the weighted result of the first ratio calculated by the second calculation unit and the second ratio calculated by the third calculation unit.

[0170] In an optional example, computation submodule 8505 includes: The seventh determining unit is used to determine the second predicted detection box of the second road surface target in the current second feature map based on the tracking result corresponding to the first target feature map sequence obtained by the fourth acquisition module 842, in response to the fifth determining submodule 8503 determining the category of the third road surface target as zebra crossing category or stop line category. The projection unit is used to project the second actual detection box of the third road surface target in the current second feature map determined by the seventh determining unit onto the long side of the second predicted detection box to obtain the projection line segment. The third calculation unit is used to calculate the intersection-union ratio of the projected line segment obtained by the projection unit with the line segment of the long side; The fourth calculation unit is used to calculate the Euclidean distance between the second predicted detection box and the second actual detection box determined by the seventh determination unit; The fifth calculation unit is used to calculate the third ratio between the line segment intersection-union ratio calculated by the third calculation unit and the preset intersection-union ratio; The sixth calculation unit is used to calculate the fourth ratio between the Euclidean distance of the second detection frame calculated by the fourth calculation unit and the preset Euclidean distance; The eighth determining unit is used to determine the cost between the third pavement target and the second pavement target based on the weighted result of the third ratio calculated by the fifth calculation unit and the fourth ratio calculated by the sixth calculation unit.

[0171] In an optional example, such as Figure 11 As shown, the device also includes: Compression module 870 is used to compress the road surface feature reconstruction results to obtain compressed results; wherein, the road surface feature reconstruction results include: the optimized three-dimensional coordinates of the four vertices of the detection box of each road surface target; The transmission module 880 is used to transmit the compression result obtained by the compression module 870 to the host computer. The compression result is used by the host computer to update the environment model.

[0172] Exemplary electronic devices Below, for reference Figure 12This describes an electronic device according to embodiments of the present disclosure. The electronic device may be either or both of a first device and a second device, or a standalone device independent of them, which may communicate with the first device and the second device to receive acquired input signals from them.

[0173] Figure 12 A block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0174] like Figure 12 As shown, the electronic device 1200 includes one or more processors 1210 and memory 1220.

[0175] The processor 1210 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 1200 to perform desired functions.

[0176] The memory 1220 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1210 may execute the program instructions to implement the road feature reconstruction methods of the various embodiments of this disclosure described above and / or other desired functions.

[0177] In one example, the electronic device 1200 may also include an input device 1230 and an output device 1240, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0178] For example, when the electronic device is a first device or a second device, the input device 1230 may be a microphone or a microphone array. When the electronic device is a standalone device, the input device 1230 may be a communication network connector for receiving acquired input signals from the first device and the second device.

[0179] In addition, the input device 1230 may also include, for example, a keyboard, a mouse, etc. The output device 1240 can output various information to the outside. The output device 1240 may include, for example, a monitor, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0180] Of course, for the sake of simplicity, Figure 12Only some of the components of the electronic device 1200 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 1200 may include any other suitable components depending on the specific application.

[0181] Exemplary computer program products and computer-readable storage media In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps in the road feature reconstruction methods according to various embodiments of this disclosure as described in the "Exemplary Methods" section of this specification.

[0182] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0183] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the steps in the road feature reconstruction methods according to various embodiments of this disclosure as described in the "Exemplary Methods" section of this specification.

[0184] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0185] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0186] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0187] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0188] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0189] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0190] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0191] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A method for reconstructing road surface elements, comprising: Acquire an environmental image set, which includes: multiple frames of images from different perspectives of the environment around the mobile device acquired at the current moment by multiple image acquisition devices set at different locations of the mobile device, and each frame of the environmental image set contains at least one road surface target. Based on the environmental image set, a first feature map containing semantic segmentation information in a bird's-eye view space is generated via a neural network; Based on the semantic segmentation information, the pixel regions of each road surface target in the first feature map are determined; The pixel regions of each road surface target in the first feature map are mapped to the preset coordinate system corresponding to the mobile device to obtain the second feature map; A first target feature map sequence corresponding to multiple historical environmental image sets is obtained. The multiple historical environmental image sets represent multiple environmental image sets collected by multiple image acquisition devices set at different locations of the mobile device at multiple historical moments. The first target feature map sequence includes: the second feature map of multiple historical frames corresponding to the multiple historical environmental image sets. Multi-target tracking is performed on the second feature map to obtain the tracking results; Based on the tracking results, road surface feature reconstruction results are generated; The process of generating road surface feature reconstruction results based on the tracking results includes: Based on the tracking results corresponding to the first target feature map sequence, the first tracked road surface target is determined; The first state variable of the second feature map corresponding to the last frame of the first target feature map sequence is used as the reference state variable, and the first uncertainty corresponding to the first state variable is used as the reference uncertainty. Based on the reference state variable, the reference uncertainty, the first observation information and observation noise of the first road surface target in the current second feature map, the second state variable of the first road surface target corresponding to the current second feature map and the second uncertainty of the second state variable are determined. In response to the second uncertainty satisfying the preset convergence condition, a road surface element reconstruction result is generated based on the second state variable.

2. The method according to claim 1, wherein, The method further includes: In response to the second uncertainty not satisfying the preset convergence condition, the reference state variable is updated to the second state variable, and the reference uncertainty is updated to the second uncertainty.

3. The method according to claim 2, wherein, The step of determining the second state variable of the first road surface target corresponding to the current second feature map, and the second uncertainty of the second state variable, based on the reference state variable, the reference uncertainty, the first observation information of the first road surface target in the current second feature map, and the observation noise, includes: The first observation information is transformed to the world coordinate system to obtain the second observation information; Determine the target Jacobian matrix of the second observation information relative to the state variables; The Kalman gain is determined based on the reference uncertainty, the target Jacobian matrix, and the observation noise. The second state variable is determined based on the reference state variable, the Kalman gain, the second observation information, and the target Jacobian matrix; The second uncertainty is determined based on the identity matrix, the Kalman gain, the target Jacobian matrix, and the reference uncertainty.

4. The method according to claim 3, wherein, Determining the target Jacobian matrix of the second observation information relative to the state variables includes: Based on the second observation information, the observation three-dimensional coordinates of the top left, top right, bottom left, and bottom right vertices of the detection box of the first road surface target are determined to obtain four observation three-dimensional coordinates; Based on the four observed three-dimensional coordinates, the length, width, and yaw angle of the detection frame for the first road surface target are determined; Based on the length, width, yaw angle, and rotation matrix between the preset coordinate system and the world coordinate system, determine the first Jacobian matrix corresponding to the top left vertex, the second Jacobian matrix corresponding to the top right vertex, the third Jacobian matrix corresponding to the bottom left vertex, and the fourth Jacobian matrix corresponding to the bottom right vertex. The target Jacobian matrix is ​​determined based on the first Jacobian matrix, the second Jacobian matrix, the third Jacobian matrix, and the fourth Jacobian matrix.

5. The method according to claim 3, wherein, Any uncertainty is characterized using the covariance matrix, and the formula for determining the Kalman gain is as follows: in, K Indicates the Kalman gain, P k-1 This indicates the reference uncertainty. H Let the target Jacobian matrix be represented. R k This represents the observed noise.

6. The method according to claim 3, wherein, The formula used to determine the second state variable is: in, x k This represents the second state variable. x k-1 This represents the reference state variable. K Indicates the Kalman gain, y k This indicates the second observation information. H Let represent the target Jacobian matrix.

7. The method according to claim 3, wherein, Each uncertainty is characterized using the covariance matrix, and the formula used to determine the second uncertainty is: P k =（ I-KH ） P k-1 in, P k This represents the second uncertainty. I Represents the identity matrix, K Indicates the Kalman gain, H Let the target Jacobian matrix be represented. P k-1 This represents the reference uncertainty.

8. The method according to claim 1, wherein, The multi-target tracking of the second feature map to obtain the tracking result includes: Based on the tracking results corresponding to the first target feature map sequence, the tracked second road surface target is determined; Target detection is performed on the current second feature map to determine the third road surface target in the current second feature map; In response to the fact that the third road surface target and the second road surface target are of the same category, a calculation method corresponding to the category of the third road surface target is used to calculate the cost between the third road surface target and the second road surface target; Based on the tracking results corresponding to the first target feature map sequence and the cost value, the tracking results corresponding to the third target feature map sequence are determined. The third target feature map sequence includes: the second feature map of multiple historical frames in the first target feature map sequence, and the current second feature map.

9. The method according to claim 8, wherein, The calculation method corresponding to the category of the third road surface target, calculating the cost value between the third road surface target and the second road surface target, includes: In response to the fact that the category of the third road surface target is the road surface arrow category, based on the tracking result corresponding to the first target feature map sequence, the first predicted detection box of the second road surface target in the current second feature map is determined; Calculate the cross-union ratio and Euclidean distance of the first predicted detection box and the first actual detection box of the third road surface target in the current second feature map; Calculate the first ratio of the detection frame intersection-union ratio to the preset intersection-union ratio; Calculate the second ratio of the Euclidean distance of the first detection frame to the preset Euclidean distance; Based on the weighted result of the first ratio and the second ratio, the cost value between the third road surface target and the second road surface target is determined.

10. The method according to claim 8, wherein, The calculation method corresponding to the category of the third road surface target, calculating the cost value between the third road surface target and the second road surface target, includes: In response to the fact that the category of the third road surface target is zebra crossing or stop line, based on the tracking result corresponding to the first target feature map sequence, a second predicted detection box of the second road surface target in the current second feature map is determined; Project the second actual detection box of the third road surface target in the current second feature map onto the long side of the second predicted detection box to obtain the projection line segment; Calculate the intersection-union ratio of the projected line segment and the line segment of the longer side; Calculate the Euclidean distance between the second predicted detection box and the second actual detection box; Calculate the third ratio of the line segment intersection-union ratio to the preset intersection-union ratio; Calculate the fourth ratio of the Euclidean distance of the second detection frame to the preset Euclidean distance; Based on the weighted result of the third ratio and the fourth ratio, the cost value between the third road surface target and the second road surface target is determined.

11. The method according to any one of claims 1-10, further comprising: The road surface feature reconstruction results are compressed to obtain a compressed result; wherein, the road surface feature reconstruction results include: the optimized three-dimensional coordinates of the four vertices of the detection box of each road surface target; The compression result is transmitted to the host computer, and the compression result is used by the host computer to update the environment model.

12. A road surface feature reconstruction device, comprising: The first acquisition module is used to acquire an environmental image set, which includes: multiple frames of images from different perspectives of the environment around the mobile device acquired at the current moment by multiple image acquisition devices set at different locations of the mobile device, and each frame of the environmental image set contains at least one road surface target. The first generation module is used to generate a first feature map containing semantic segmentation information in a bird's-eye view space based on the environmental image set obtained by the first acquisition module via a neural network. The determining module is used to determine the pixel region of each of the road surface targets in the first feature map based on the semantic segmentation information contained in the first feature map generated by the first generating module. The second acquisition module is used to map the pixel regions of each road surface target in the first feature map determined by the determining module to the preset coordinate system corresponding to the mobile device, so as to obtain the second feature map. The third acquisition module is used to acquire a first target feature map sequence corresponding to multiple historical environmental image sets. The multiple historical environmental image sets represent multiple environmental image sets acquired by multiple image acquisition devices located at different positions of the mobile device at multiple historical moments. The first target feature map sequence includes: the second feature map of multiple historical frames corresponding to the multiple historical environmental image sets. A multi-target tracking module is used to perform multi-target tracking on the second feature map obtained by the second acquisition module to obtain tracking results; The second generation module is used to generate road surface element reconstruction results based on the tracking results obtained by the multi-target tracking module. The second generation module includes: a first determination submodule, used to determine the tracked first road surface target based on the tracking results corresponding to the first target feature map sequence; a second determination submodule, used to take the first state variable of the second feature map corresponding to the first road surface target in the last historical frame of the first target feature map sequence as a reference state variable, and take the first uncertainty corresponding to the first state variable as a reference uncertainty; a third determination submodule, used to determine the second state variable of the first road surface target corresponding to the current second feature map and the second uncertainty of the second state variable based on the reference state variable, the reference uncertainty, the first observation information and observation noise of the first road surface target in the current second feature map; and a generation submodule, used to generate road surface element reconstruction results based on the second state variable in response to the second uncertainty satisfying a preset convergence condition.

13. A computer-readable storage medium storing a computer program for performing the road feature reconstruction method according to any one of claims 1-11.

14. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the road feature reconstruction method according to any one of claims 1-11.

Images