Method, device and electronic device for reconstructing high-precision map

Abstract

The present application relates to a method, device and electronic device for reconstructing a high-precision map. The method for reconstructing a high-precision map includes: obtaining image information and current position information, the image information is obtained by a sensor mounted on a mobile platform and includes an image of an observed object; matching the local area of ​​the current position information and the image information in the high-precision map to obtain a target object corresponding to the observed object in the high-precision map; using the model of the observed object as the model of the target object in the high-precision map to reconstruct the high-precision map. The present application can improve the accuracy of objects in high-precision maps.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device and electronic device for reconstructing a high-precision map. Background Art

[0002] With the development of autonomous driving technology, the demand for high-precision maps is increasing.

[0003] However, the applicant found that the accuracy of the high-precision maps constructed in the relevant technology needs to be further improved. Summary of the invention

[0004] In order to at least partially solve the problems existing in the related art, the present application provides a method, device and electronic device for reconstructing a high-precision map, which can effectively improve the accuracy of the reconstructed high-precision map.

[0005] The first aspect of the present application provides a method for reconstructing a high-precision map, comprising: obtaining image information and current position information, the image information being obtained by a sensor mounted on a mobile platform and including an image of an observed object; matching the current position information and a local area of ​​the image information in the high-precision map to obtain a target object in the high-precision map corresponding to the observed object; and using a model of the observed object as a model of the target object in the high-precision map to reconstruct the high-precision map.

[0006] According to certain embodiments of the present application, the current position information and the local area of ​​the image information are matched in the high-precision map to obtain the target object corresponding to the observed object in the high-precision map, including: determining the first position information of each image of at least one observed object in the image information; determining the second position information in the high-precision map corresponding to the first position information and the current position information based on the first mapping relationship; and determining the target object in the high-precision map based on the second position information.

[0007] According to certain embodiments of the present application, determining the first position information of each image of at least one observed object in the image information includes: for each frame in the acquired multiple frames of image information, performing image recognition on the image information of the current frame to obtain image area information of each image of at least one observed object in the image information; and determining the depth information of each at least one observed object based at least on the difference between the image area information of at least one observed object in a specified number of adjacent frames of the current frame.

[0008] According to certain embodiments of the present application, using a model of an observed object as a model of a target object in a high-precision map includes generating a target model corresponding to a model of at least one observed object in the high-precision map based on the second position information.

[0009] According to certain embodiments of the present application, the above method may also include: constructing a model of at least one observation object based on image area information of each image of at least one observation object in the image information, and depth information of at least one observation object. Accordingly, a target model corresponding to the model of at least one observation object is generated in the high-precision map based on the second position information, including: for a two-dimensional target object, based on the image area information of each image of at least one observation object in the image information, performing texturing at a position corresponding to the second position information to generate a target model; for a three-dimensional target object, performing three-dimensional model rendering at a position corresponding to the second position information by using the constructed model of at least one observation object to generate a target model.

[0010] According to certain embodiments of the present application, image recognition is performed on image information of a current frame to obtain image region information of each image of at least one observed object in the image information, including: using an image recognition model to process the image information of the current frame to obtain image region information of each image of at least one observed object in the image information, wherein the image recognition model includes at least two channels, and each of the at least two channels performs image recognition at different scales.

[0011] According to certain embodiments of the present application, the depth information of at least one observed object is determined based on at least one difference between the image area information of at least one observed object in a specified number of adjacent frames of the current frame, including: processing the image area information of at least one observed object in the current frame and a specified number of adjacent frames based on a synchronous positioning and mapping method to obtain the difference between the image area information of at least one observed object; and determining the depth information of at least one observed object based on the difference between the image area information of at least one observed object.

[0012] The second aspect of the present application provides a device for reconstructing a high-precision map, comprising: an information acquisition module, a target object acquisition module and a high-precision map reconstruction module. The information acquisition module is used to acquire image information and current position information, the image information is acquired by a sensor mounted on a mobile platform, and includes an image of an observed object; the target object acquisition module is used to match the local area of ​​the current position information and the image information in the high-precision map to obtain the target object corresponding to the observed object in the high-precision map; the high-precision map reconstruction module is used to use the model of the observed object as the model of the target object in the high-precision map to reconstruct the high-precision map.

[0013] The third aspect of the present application provides an electronic device, comprising: a processor; and a memory on which executable codes are stored, and when the executable codes are executed by the processor, the processor executes the above method.

[0014] The fourth aspect of the present application also provides a computer-readable storage medium on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor executes the above method.

[0015] The fifth aspect of the present application also provides a computer program product, including executable code, which implements the above method when executed by a processor.

[0016] The method, device and electronic device for reconstructing high-precision maps provided in the present application match the image information collected by a mobile platform and the image collection position in the high-precision map to obtain the target object in the high-precision map corresponding to each observed object in the image information. This facilitates the reconstruction of the target object based on the image information of the observed object in the image information, thereby realizing the reconstruction of the high-precision map and improving the accuracy of the high-precision map.

[0017] In addition, the observed objects obtained through image recognition have richer structure, size information, etc., which helps to build a more refined model of template objects in high-precision maps.

[0018] In addition, the structural information of the observed object is determined based on image recognition and simultaneous positioning and mapping, which effectively reduces the complexity of data fusion for determining the observed object through vector data.

[0019] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The present application and other objects, features and advantages will become more apparent by describing in more detail exemplary embodiments of the present application in conjunction with the accompanying drawings, wherein like reference numerals generally represent like components in the exemplary embodiments of the present application.

[0021] Figure 1 An exemplary system architecture of a method, device, and electronic device for reconstructing a high-precision map according to an embodiment of the present application is schematically shown;

[0022] Figure 2 A flowchart of a method for reconstructing a high-precision map according to an embodiment of the present application is schematically shown;

[0023] Figure 3 A schematic diagram of the structure of a device for reconstructing a high-precision map according to an embodiment of the present application is shown;

[0024] Figure 4 A schematic diagram of a high-precision map according to an embodiment of the present application is schematically shown;

[0025] Figure 5A schematic diagram of a partially reconstructed high-precision map according to an embodiment of the present application is schematically shown;

[0026] Figure 6 A schematic diagram schematically shows the second location information in a high-precision map according to an embodiment of the present application;

[0027] Figure 7 The schematic diagram of the structure of the image recognition model according to the embodiment of the present application is shown;

[0028] Figure 8 A schematic diagram of determining depth information according to an embodiment of the present application is schematically shown;

[0029] Fig. 9 A block diagram of a device for reconstructing a high-precision map according to an embodiment of the present application is schematically shown;

[0030] Fig.10 A block diagram of an electronic device according to an embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0031] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to make the present application more thorough and complete, and to fully convey the scope of the present application to those skilled in the art.

[0032] The terms used in this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The terms "comprise", "include", etc. used herein indicate the existence of features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0033] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.

[0034] It should be understood that although the terms "first", "second", "third", etc. may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, "multiple" means two or more, unless otherwise clearly and specifically defined.

[0035] The core of autonomous driving, such as high-speed autonomous navigation driving (Navigation Guided Pilot, NGP for short), is high-precision maps and high-precision positioning.

[0036] High-precision maps are a type of electronic maps, specifically belonging to the category of third-generation electronic navigation.

[0037] The first generation of electronic navigation mainly presents 2D flat maps, which require downloading map data of various cities to the local computer, and the route planning is calculated by the local chip to get the result. It can complete the journey from point A to point B, but the calculation logic can only consider the distance and road attributes (such as national highways, provincial highways, ring roads, toll highways, etc.).

[0038] The second generation of electronic navigation is the widely used electronic map navigation. In the second generation of electronic navigation technology, navigation equipment began to be connected to the Internet, and big data and cloud computing technologies were gradually introduced. When going from point A to point B, there are more and more issues to consider in navigation, such as whether the road is congested, and whether to choose the shortest distance or the shortest time route. Moreover, as the map accuracy improves, the navigation can further pop up road images and guide routes when passing through forks.

[0039] The third generation of electronic navigation is based on high-precision maps. High-precision maps will be used more in the vehicle's autonomous driving function. In related technologies, mainstream autonomous driving functions are mostly L2 level. For example, the vehicle uses the front camera and millimeter wave radar to confirm whether there is a vehicle in the passable area directly in front of the vehicle. If there is a vehicle, follow the vehicle within the safe braking distance. If there is no vehicle, accelerate to the set cruising speed. In related technologies, this function cannot be completed by relying on non-high-precision maps, and the coordinate accuracy of non-high-precision maps is about 5-10 meters. The absolute accuracy of high-precision maps can be at the sub-meter level, such as about 10 centimeters, and the relative lateral accuracy is often even higher. High-precision maps not only have high-precision coordinates, but also accurate road shapes, and the slope, curvature, heading, elevation, and roll data of each lane are also included.

[0040] In addition, the location of each lane line, whether it is a dotted line, a solid line, or a double yellow line, the color of the line, the road divider, and even the arrows and text on the road will be described. In addition, in order to facilitate the realization of autonomous driving, the speed limit and recommended speed of each lane can also be provided. The crosswalks, billboards along the road, dividers, speed limit signs, traffic lights, roadside telephone stops, etc. in the high-precision map can be collectively referred to as the absolute geographic coordinates of traffic participants. Their physical dimensions and their characteristics will also appear in the data of the high-precision map.

[0041] However, the description of these material properties is still relatively general, such as which coordinate range contains traffic lights, street lights, etc., but the specific shapes of traffic lights, street lights, etc. are not described accurately enough. Figure 6 shown.

[0042] The embodiment of the present application matches the image information collected by the mobile platform and the image collection location in the high-precision map, and determines the object image information corresponding to the target object in the high-precision map. This facilitates the reconstruction of the target object based at least on the object image information, thereby realizing the reconstruction of the high-precision map and improving the accuracy of the high-precision map.

[0043] The following will be passed Figures 1 to 10 A method, device and electronic device for reconstructing a high-precision map in an embodiment of the present application are described in detail.

[0044] Figure 1 The following schematically illustrates an exemplary system architecture of a method, device, and electronic device for reconstructing a high-precision map according to an embodiment of the present application. It should be noted that: Figure 1 The architecture shown is merely an example of a system architecture to which the embodiments of the present application can be applied, to help those skilled in the art understand the technical content of the present application, but it does not mean that the embodiments of the present application cannot be used in other devices, systems, environments or scenarios.

[0045] See also Figure 1 According to this embodiment, the system architecture 100 may include mobile devices 101, 102, 103, a network 104, and a cloud 105. The network 104 is used to provide a medium for communication links between the mobile devices 101, 102, 103 and the cloud 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0046] Users can use mobile devices 101, 102, 103 to interact with other mobile devices and cloud 105 through network 104 to receive or send information, such as sending image recognition requests, depth information requests and receiving model structures and model parameters, image recognition results, depth information, etc. Mobile devices 101, 102, 103 can be installed with various communication client applications, such as car applications, navigation applications, automatic / assisted driving applications, web browser applications, database applications, search applications, instant messaging tools, email clients, social platform software, etc. It should be noted that operations such as image recognition and depth calculation can be performed locally or in the cloud.

[0047] Mobile devices 101 , 102 , 103 include but are not limited to automobiles, robots, tablet computers, laptop computers, and other electronic devices that can support functions such as surfing the Internet and human-computer interaction.

[0048] The cloud 105 can receive image recognition requests, depth information requests, etc., and can also send image recognition results, depth information, object models, intelligent navigation, road condition information, etc. to mobile devices 101, 102, 103. For example, the cloud 105 can be a background management server, a server cluster, a car network, etc.

[0049] It should be noted that the number of mobile devices, networks and cloud servers is only illustrative, and any number of mobile devices, networks and cloud servers may be provided as required.

[0050] Figure 2 A flowchart of a method for reconstructing a high-precision map according to an embodiment of the present application is schematically shown.

[0051] like Figure 2 As shown, this embodiment provides a method for reconstructing a high-precision map, which includes operations S210 to S230, as follows.

[0052] In operation S210 , image information and current position information are acquired, where the image information is acquired by a sensor mounted on a mobile platform and includes an image of an observed object.

[0053] In this embodiment, image information can be collected through a mobile platform, and the time information of collecting each frame of image can be recorded during the image information collection process. The mobile platform includes but is not limited to: a vehicle, robot, ship, aircraft, etc. equipped with a shooting device.

[0054] Sensors include but are not limited to image sensors, infrared sensors, position sensors, radars, etc.

[0055] The current position information may be determined based on real-time kinematic (RTK) technology. Specifically, the positioning information may be obtained through GPS positioning and / or Beidou positioning, and high-precision positioning information may be obtained using RTK technology.

[0056] Figure 3 A schematic diagram showing the structure of a device for reconstructing a high-precision map according to an embodiment of the present application

[0057] See also Figure 3 The vehicle 10 for collecting high-precision map data may be equipped with a shooting device 20 and a positioning device 30. It should be noted that in addition to the wide-angle camera, the shooting device 20 on the top of the vehicle 10 may also be equipped with a gyroscope and other devices to detect road slope, curvature, heading, elevation, roll and other data.

[0058] In related technologies, the collected information can be used to draw a vector map, which can then be processed through a series of precision overlays to obtain a high-precision map.

[0059] It should be noted that, in addition to Figure 3 In addition to the vehicle information collection method shown, high-precision maps can also be created by collecting information through drones.

[0060] In operation S220, the current position information and the local area of ​​the image information are matched in the high-precision map to obtain the target object corresponding to the observed object in the high-precision map.

[0061] In this embodiment, the high-precision map can be pre-built. The high-precision map can be divided into three layers: the map layer, which saves the structured information of the road, which defines the lane lines, traffic signs, traffic rules information, etc. The positioning layer saves the original point cloud map and extracts some features such as telephone poles, buildings, traffic signs, etc., which are used for point cloud matching and positioning. The dynamic layer dynamically updates some information in real time, such as: current road conditions, real-time traffic rules, traffic accidents ahead, etc. This application can reconstruct the map layer and the positioning layer.

[0062] The construction process of high-precision maps can include four steps: map collection, point cloud map production, map annotation, and map storage. For example, the map collection process can be completed by a data collection vehicle equipped with sensors such as radar, camera, positioning equipment, and inertial measurement unit (IMU). The collection vehicle can refer to Figure 3As shown. The point cloud map production process can generate a 3D model by point cloud stitching and other methods. The map annotation process is based on the point cloud map, annotating the lane line information, traffic sign information, traffic light information, etc., to obtain the structured information of the road. The map storage process is mainly to save the annotated information in a fixed format.

[0063] The local area in this embodiment can be the area where the image corresponding to the recognition result with a higher confidence level is located. For example, the captured image includes multiple objects, and the distances of different objects from the shooting device may vary greatly. The image area where the object is too far away from the shooting device or the image area where the image size is too small cannot obtain a recognition result with a higher confidence level. In order to ensure the accuracy of the reconstructed high-precision map, only the local area corresponding to the high confidence level can be reconstructed. The confidence threshold can be set according to expert experience or reconstruction effect. In addition, in order to reduce the consumption of computing resources for reconstructing high-precision maps, the map reconstruction process can be performed on images of preset areas. For example, the probability that the observed objects appearing in the upper, lower, left and right areas of the captured image are close to the shooting device is relatively high, and the upper, lower, left and right areas can be used as local areas.

[0064] In some embodiments, the local area includes, but is not limited to: an area around the captured image, an image area of ​​an object that occupies a larger proportion of the image, an image area of ​​an object that is closer to the camera, etc.

[0065] For example, for ground elements (such as various lines on the ground), the disappearance (appearance) of a certain type of element can be identified at the bottom of the video, which is the end (beginning) of the reconstruction of the map of this type of element, and the current position information (such as position coordinates) at that time point can be recorded at the same time.

[0066] For example, for elements on both sides of the road, such as buildings and green belts, the disappearance (appearance) of a certain type of element can be identified at the left and right ends of the video, which is the end (beginning) of the reconstruction of the map of this type of element, and the current position information at that time point can be recorded at the same time.

[0067] For example, for elements above the road surface such as traffic lights and road signs, the disappearance (appearance) of a certain type of element can be identified at the top of the video, which is the end (beginning) of the reconstruction of the map of this type of element, and the current position information at that time point can be recorded at the same time.

[0068] In operation S230, the model of the observed object is used as the model of the target object in the high-precision map to reconstruct the high-precision map.

[0069] In this embodiment, after an object is identified from an image, an image of the object can be obtained, which includes more detailed information, so that the object in the high-precision map can be reconstructed.

[0070] Since the point cloud data collected by radar is relatively sparse, the detailed structure of objects in high-precision maps cannot be well expressed.

[0071] Figure 4 A schematic diagram of a high-precision map according to an embodiment of the present application is schematically shown.

[0072] See also Figure 4 , Figure 4 The traffic lights enclosed by the dotted frame in the high-precision map lack structural details, such as the size information of the horizontal and vertical poles of the traffic lights, the size information of the lamp holders, the number of lights, the colors of the lights, etc. In this embodiment, the traffic light model in the high-precision map can be reconstructed with the help of the collected images of the traffic lights.

[0073] Figure 5 A schematic diagram of a partially reconstructed high-precision map according to an embodiment of the present application is schematically shown.

[0074] See also Figure 5 Based on the image of the traffic light in the collected image information, the size of the light pole and the size of the light housing of the traffic light can be calculated, and the number of lights, color and other information can be obtained from the image information. Therefore, a traffic light model can be constructed based on this information, and the detail expression capability of the constructed model is much higher than that of the model constructed based on the point cloud data collected by the radar.

[0075] It should be noted that Figure 5 In the above example, only the traffic lights are reconstructed, which is only an example and is not limited here. For example, buildings and trees on the roadside can also be reconstructed.

[0076] also, Figure 5 The model in can be a model generated by adjusting a pre-built model. For example, a lamp pole can be constructed by one or more cylindrical models. The lamp body of a traffic light can be constructed by a lamp housing model and at least one lamp model. For example, the lamp housing can be a cube, a cylinder, etc. The lamp model can be a spherical model or a bulb model, etc., so that it is easy to quickly build a traffic light model with a similar structure and size to the traffic light observed by the sensor.

[0077] The process of model construction may be a process of model size adjustment and model assembly, and may specifically adopt a variety of related technologies, which are not limited here.

[0078] The matching process is exemplified below.

[0079] In some embodiments, matching the current location information and the local area of ​​the image information in the high-precision map to obtain the target object corresponding to the observed object in the high-precision map may include the following operations.

[0080] First, first position information of each image of at least one observed object in the image information is determined. The first position information may be coordinate information in the image, such as (x1, y1), etc. The image of the observed object may occupy an area, which may be represented by multiple coordinates, such as a quadrilateral area of ​​an image such as a billboard, which may be represented by four vertices.

[0081] Then, based on the first mapping relationship, the second position information corresponding to the first position information and the current position information in the high-precision map is determined. Specifically, a section perpendicular to the direction of travel of the car in the high-precision map can be determined based on the current position information, and then the corresponding position, such as a certain area in the section, is determined from the section based on the first position. Among them, the first mapping relationship can be determined by calibration, coordinate conversion, and the like. For example, after the shooting device is fixed at a specific position of the car, the first position information of the target object in the shooting position and the shot image and the second position information of the target object in the high-precision map can be calibrated.

[0082] Next, the target object in the high-precision map is determined based on the second position information. In this way, the target object corresponding to a certain area can be determined.

[0083] Figure 6 A schematic diagram of the second location information in a high-precision map according to an embodiment of the present application is schematically shown.

[0084] See also Figure 6 In the high-precision map before reconstruction, the street lamp on the side of the road can be represented by a vertical line, and the position of the vertical line can be represented by at least two coordinates, such as coordinate 1 (x1, y1, z1) and coordinate 2 (x2, y2, z2). The first position information is (x3, y3). If the coordinates are successfully matched with coordinate 1 after the coordinate conversion based on the first mapping relationship (such as the difference between the two coordinates is less than the preset range), it can be determined that the street lamp is the target object.

[0085] In some embodiments, the shooting device may shoot video information, and for each frame of multiple frames of image information in the acquired video information, determining the first position information of each image of at least one observed object in the image information may include the following operations.

[0086] First, image recognition is performed on the image information of the current frame to obtain image region information of each image of at least one observed object in the image information. A variety of related technologies can be used for image recognition. For example, a variety of classification models can be used for image recognition. For example, AlexNet, VGG19, ResNet_152, InceptionV4, DenseNet, etc.

[0087] Then, the depth information of at least one observed object is determined based on the difference between the image area information of at least one observed object in the specified number of adjacent frames of the current frame. When the distance between the shooting device and the observed object changes, the image of the observed object captured by the shooting device changes, and the change rule between the two can be determined by calibration and other methods. In this way, the depth information of the captured object can be determined based on the variable of the moving distance of the shooting device and the image of the observed object, such as the calibration ratio.

[0088] In some embodiments, using the model of the observed object as the model of the target object in the high-precision map may include the following operations: generating a target model corresponding to each model of at least one observed object in the high-precision map based on the second position information. Figure 5 As shown, after identifying that the observed objects include street lamps, a three-dimensional (3D) model of the street lamp can be constructed based on the recognition results of the street lamps (such as the size information in the image of the street lamp pole (which can be converted into the size information in the high-precision map), the number of lamps, the color information of the lamps, the size information of the lamps, etc.), thereby effectively improving the ability to express the detailed information of the target object in the high-precision map.

[0089] In some embodiments, the above method may further include the following operations: Based on the image area information of each image of at least one observed object in the image information and the depth information of at least one observed object, construct a model of at least one observed object. The model of the observed object constructed in this way is more compatible with the high-precision map.

[0090] Accordingly, generating a target model corresponding to a respective model of at least one observed object in a high-precision map based on the second position information may include the following operations.

[0091] For example, for a two-dimensional target object, based on the image area information of each image of at least one observed object in the image information, texture processing is performed at the position corresponding to the second position information to generate a target model. For example, a two-dimensional object can be textured. The texture can be obtained by matching from an image database, such as a corresponding texture image for a billboard. The texture can be generated based on a preset rule, such as a monochrome image.

[0092] For example, for a three-dimensional target object, by using the constructed model of at least one observed object, a three-dimensional model rendering is performed at a position corresponding to the second position information to generate a target model. The three-dimensional object can be rendered using an existing three-dimensional model. A variety of model rendering methods can be used, which are not limited here.

[0093] In some embodiments, performing image recognition on the image information of the current frame to obtain image region information of each image of at least one observed object in the image information may include the following operations: processing the image information of the current frame using an image recognition model to obtain image region information of each image of at least one observed object in the image information, wherein the image recognition model includes at least two channels, and each of the at least two channels performs image recognition at different scales.

[0094] Among them, in order to improve the model to extract classification information of multi-level objects from the image (such as the classification information of the traffic light image includes but is not limited to: traffic light, lamp pole, lamp shell, lamp and other classification information), features of different depths can be extracted.

[0095] Figure 7 The structural diagram of the image recognition model according to the embodiment of the present application is schematically shown.

[0096] See also Figure 7 In the feature extraction process, three channel features can be used for feature extraction respectively. For example, the size of the convolution kernel (cov) of the first channel can be 3×3, and the number of channels is 256. The size of the convolution kernel of the second channel can be 5×5, and the number of channels is 256. The size of the convolution kernel of the third channel can be 1×1, and the number of channels is 256.

[0097] Figure 7 The symbol "×" represents concatenation, and the symbol "+" represents addition. The number of categories can be a preset number to prevent over-classification or insufficient classification. The activation function can be Softmax.

[0098] It should be noted that after performing two convolution operations on the input image (to extract the main features, the two convolutions here are not fixed, just for illustration, more or fewer convolutions can be performed), convolutions are performed separately using convolution kernels of different sizes. Convolution kernels of different sizes have different receptive field sizes. Large convolution kernels can extract the features of large targets, and small receptive fields can extract the features of small targets.

[0099] Concat function: concatenate the features of feature maps output by different convolution kernels to obtain the fused features. Then perform conv 1x1 operation, which has the advantage of effectively mixing the valid information in each branch feature map.

[0100] Figure 7 The residual structure in the network is based on the residual idea. The result obtained by conv 1x1 and conv 5x5 in the third branch is added to the mixed feature map of the first and second branches. The advantage of this is that it can prevent the deformation of the main information during the forward propagation of the mixed information, effectively ensuring the correctness of feature extraction. At the same time, it also enriches the representation ability of the network model (because different branches have different functions, when performing concat or add, the network will continuously adjust the parameters of each branch to make the network representation ability better).

[0101] Figure 7 The network structure shown can have the following advantages when applied to the scenario of high-precision map reconstruction.

[0102] For example, using different convolution kernels to convolve the feature map can enable the model to focus on target objects of different sizes. For example, finer structures such as lamp poles, lamp shells, and lamps can be further extracted from traffic lights, which helps to improve the accuracy of modeling.

[0103] For example, by fusing different feature maps, the model's expressiveness becomes richer and the extracted features are more conducive to the convergence of the model.

[0104] For example, the idea of ​​residual is used to ensure that the main information of the model is not deformed during forward propagation.

[0105] In some embodiments, determining the depth information of at least one observed object based on at least a difference between image region information of at least one observed object in a specified number of adjacent frames of the current frame may include the following operations.

[0106] Firstly, image region information of at least one observed object in a current frame and a specified number of adjacent frames is processed based on a simultaneous localization and mapping (SLAM) method to obtain a difference between the image region information of at least one observed object.

[0107] Then, the depth information of each of the at least one observation object is determined based on the difference between the image region information of the at least one observation object.

[0108] Among them, the SLAM problem can be described as: the robot starts to move from an unknown position in an unknown environment, locates itself according to the position and map during the movement, and builds an incremental map based on its own positioning to achieve autonomous positioning and navigation of the robot.

[0109] The entire SLAM can be roughly divided into the front end and the back end. The front end is equivalent to VO (visual odometer), which studies the transformation relationship between frames. First, extract the feature points of each frame image, use adjacent frame images to match the feature points, then use RANSAC to remove large noise, and then match to obtain a pose information (such as position and attitude). In addition, the attitude information provided by the inertial measurement unit (IMU) can also be used for filtering and fusion.

[0110] Graph-based SLAM is usually based on keyframes to establish relative transformation relationships between multiple nodes, such as affine transformation matrices, and continuously maintain key nodes to ensure the capacity of the graph, thereby reducing the amount of calculation while ensuring accuracy.

[0111] Figure 8 The following schematic diagram shows a method for determining depth information according to an embodiment of the present application.

[0112] See also Figure 8 , the world coordinate system may include an x-axis, a y-axis, and a z-axis. The area occupied by the observed object (e.g., a hexahedron) in the captured image in the world coordinate system (as shown by x1 and y1) can be expressed as Figure 8 When the coordinates of the camera relative to the observed object in the world coordinate system change (e.g., the traffic light does not change position, but as the vehicle moves, the camera moves closer to the traffic light), the area occupied by the image of the observed object in the captured image can be expressed as Figure 8 As shown in the dashed box in the figure below. It can be seen that the size of the area occupied by the image (as shown in x2 and y2) has changed. At this time, the corresponding relationship between the calibrated image size change rate (amount) and the relative distance change rate (amount) in the world coordinate system (such as the distance between the calibration object and the camera) and the displacement distance of the camera can be used to solve the problem of Figure 8The depth information corresponding to the size change rate (amount) shown. Among them, the displacement distance and the relative distance can be determined by the time difference between the two frames and the vehicle moving speed. The calibration process can be as follows: use a camera device to shoot an object with specified depth information (such as the size of the observed object along the optical axis direction of the lens is the specified depth information), when the shooting device moves from position 1 to position 2 (the moving distance is L), determine the image size change rate (amount) of the observed object in the captured image. Repeat the above process multiple times, or use observed objects with different depth information, and the above-mentioned first mapping relationship can be calibrated.

[0113] It should be noted that Figure 8 The x1, y1 and x2, y2 shown in are only exemplary and should not be construed as limiting the technical solution of the present application. For example, the depth information of the observed object can also be calculated based on the length change of a certain side (or maximum size, etc.) of the observed object in the image.

[0114] Another aspect of the present application also provides a device for reconstructing a high-precision map.

[0115] Fig. 9 A block diagram of an apparatus for reconstructing a high-precision map according to an embodiment of the present application is schematically shown.

[0116] See also Fig. 9 The device 900 for reconstructing a high-precision map may include: an information acquisition module 910, a target object acquisition module 920 and a high-precision map reconstruction module 930.

[0117] The information acquisition module 910 is used to acquire image information and current position information. The image information is acquired by a sensor mounted on the mobile platform and includes an image of the observed object.

[0118] The target object acquisition module 920 is used to match the current position information and the local area of ​​the image information in the high-precision map to obtain the target object corresponding to the observed object in the high-precision map.

[0119] The high-precision map reconstruction module 930 is used to use the model of the observed object as the model of the target object in the high-precision map to reconstruct the high-precision map.

[0120] In some embodiments, the target object obtaining module 920 includes: a first position information determining unit, a second position information determining unit, and a target object determining unit.

[0121] The first position information determining unit is used to determine the first position information of each image of at least one observed object in the image information.

[0122] The second location information determination unit is used to determine the second location information corresponding to the first location information and the current location information in the high-precision map based on the first mapping relationship.

[0123] A target object determination unit is used to determine the target object in the high-precision map based on the second position information.

[0124] In some embodiments, the first position information determining unit includes: an image recognition subunit and a depth information determining subunit.

[0125] The image recognition subunit is used to perform image recognition on the image information of the current frame to obtain image region information of each image of at least one observed object in the image information;

[0126] The depth information determination subunit is used to determine the depth information of at least one observed object based on at least a difference between image region information of at least one observed object in a specified number of adjacent frames of the current frame.

[0127] In some embodiments, the high-precision map reconstruction module 930 is specifically used to generate a target model corresponding to the model of at least one observed object in the high-precision map based on the second position information.

[0128] In some embodiments, the apparatus 900 may further include: an observation object model building module.

[0129] The observation object model construction module is used to construct a model of at least one observation object based on image area information of each image of at least one observation object in the image information and depth information of at least one observation object.

[0130] Accordingly, the high-precision map reconstruction module 930 may include: a first target model generation unit and a second target model generation unit.

[0131] The first target model generating unit is used to generate a target model for a two-dimensional target object by performing texture processing at a position corresponding to the second position information based on image area information of each image of at least one observed object in the image information.

[0132] The second target model generating unit is used to generate a target model for a three-dimensional target object by performing three-dimensional model rendering at a position corresponding to the second position information using a model of at least one observed object constructed respectively.

[0133] In some embodiments, the image recognition subunit is specifically used to process the image information of the current frame using an image recognition model to obtain image area information of each image of at least one observed object in the image information, wherein the image recognition model includes at least two channels, and each of the at least two channels performs image recognition at different scales.

[0134] In some embodiments, the depth information determination subunit is specifically used to achieve: processing the image area information of at least one observed object in the current frame and a specified number of adjacent frames based on a synchronous positioning and mapping method to obtain the difference between the image area information of at least one observed object; and determining the depth information of at least one observed object based on the difference between the image area information of at least one observed object.

[0135] Regarding the device 900 in the above embodiment, the specific manner in which each module, unit, and sub-unit performs operations has been described in detail in the embodiment of the method, and will not be elaborated again here.

[0136] Another aspect of the present application provides an electronic device.

[0137] Fig.10 A block diagram of an electronic device according to an embodiment of the present application is schematically shown.

[0138] See also Fig.10 , the electronic device 1000 includes a memory 1010 and a processor 1020 .

[0139] The processor 1020 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0140] The memory 1010 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. Among them, ROM can store static data or instructions required by the processor 1020 or other modules of the computer. The permanent storage device may be a readable and writable storage device. The permanent storage device may be a non-volatile storage device that does not lose the stored instructions and data even after the computer is powered off. In some embodiments, the permanent storage device uses a large-capacity storage device (such as a magnetic or optical disk, flash memory) as a permanent storage device. In some other embodiments, the permanent storage device may be a removable storage device (such as a floppy disk, optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as a dynamic random access memory. The system memory may store some or all instructions and data required by the processor at run time. In addition, the memory 1010 may include any combination of computer-readable storage media, including various types of semiconductor storage chips (such as DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, the memory 1010 may include a readable and / or writable removable storage device, such as a laser disc (CD), a read-only digital versatile disc (such as a DVD-ROM, a double-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (such as an SD card, a mini SD card, a Micro-SD card, etc.), a magnetic floppy disk, etc. The computer-readable storage medium does not include carrier waves and transient electronic signals transmitted wirelessly or wired.

[0141] The memory 1010 stores executable codes, and when the executable codes are processed by the processor 1020 , the processor 1020 can execute part or all of the methods described above.

[0142] In addition, the method according to the present application may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing some or all of the steps in the above method of the present application.

[0143] Alternatively, the present application can also be implemented as a computer-readable storage medium (or non-transitory machine-readable storage medium or machine-readable storage medium) on which executable code (or computer program or computer instruction code) is stored. When the executable code (or computer program or computer instruction code) is executed by a processor of an electronic device (or server, etc.), the processor executes part or all of the steps of the above-mentioned method according to the present application.

[0144] The embodiments of the present application have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for reconstructing a high-precision map, characterized in that: include: Acquire image information and current position information, wherein the image information is acquired by a sensor mounted on the mobile platform and includes an image of an observed object; Matching the current position information and the local area of ​​the image information in the high-precision map to obtain the target object corresponding to the observed object in the high-precision map, including: determining the first position information of each image of at least one of the observed objects in the image information, determining the second position information in the high-precision map corresponding to the first position information and the current position information based on a first mapping relationship, and determining the target object in the high-precision map based on the second position information; wherein, The determining, based on the first mapping relationship, the second position information in the high-precision map corresponding to the first position information and the current position information comprises: determining, based on the current position information, a section in the high-precision map that is perpendicular to the driving direction of the mobile platform, and determining the corresponding second position information from the section based on the first position information; The model of the observed object is used as the model of the target object in the high-precision map to reconstruct the high-precision map.

2. The method according to claim 1, characterized in that The determining of the first position information of each image of at least one of the observed objects in the image information comprises: for each frame of the acquired multiple frames of image information, Performing image recognition on the image information of the current frame to obtain image region information of each image of at least one of the observed objects in the image information; Depth information of at least one of the observed objects is determined based on at least a difference between image region information of at least one of the observed objects in a specified number of adjacent frames of the current frame.

3. The method according to claim 2, characterized in that The step of using the model of the observed object as the model of the target object in the high-precision map includes: Based on the second position information, a target model corresponding to each model of at least one of the observed objects is generated in the high-precision map.

4. The method according to claim 3, characterized in that Also includes: constructing a model of at least one of the observed objects based on image region information of each image of at least one of the observed objects in the image information and depth information of at least one of the observed objects; The generating a target model corresponding to a model of at least one of the observed objects in the high-precision map based on the second position information includes: For a two-dimensional target object, based on image region information of each image of at least one of the observed objects in the image information, texture processing is performed at a position corresponding to the second position information to generate the target model; For a three-dimensional target object, the target model is generated by performing three-dimensional model rendering at a position corresponding to the second position information using the constructed model of at least one of the observed objects.

5. The method according to claim 2, characterized in that: The performing image recognition on the image information of the current frame to obtain image region information of each image of at least one of the observed objects in the image information includes: The image information of the current frame is processed using an image recognition model to obtain image region information of at least one image of the observed object in the image information, wherein the image recognition model includes at least two channels, and each of the at least two channels performs image recognition at a different scale.

6. The method according to claim 2, characterized in that The determining of the depth information of at least one of the observed objects based on the difference between the image area information of at least one of the observed objects in the specified number of adjacent frames of the current frame comprises: Processing the image region information of at least one of the observed objects in the current frame and a specified number of adjacent frames based on a synchronous positioning and mapping method to obtain a difference between the image region information of at least one of the observed objects; Depth information of at least one of the observation objects is determined based on a difference between image region information of at least one of the observation objects.

7. A device for reconstructing a high-precision map, characterized in that: include: An information acquisition module, used to acquire image information and current position information, wherein the image information is acquired by a sensor mounted on the mobile platform and includes an image of the observed object; A target object acquisition module is used to match the local area of ​​the current position information and the image information in a high-precision map to obtain a target object corresponding to the observed object in the high-precision map, which includes: determining the first position information of each image of at least one of the observed objects in the image information, determining the second position information in the high-precision map corresponding to the first position information and the current position information based on a first mapping relationship, and determining the target object in the high-precision map based on the second position information, wherein, The determining, based on the first mapping relationship, the second position information in the high-precision map corresponding to the first position information and the current position information comprises: determining, based on the current position information, a section in the high-precision map that is perpendicular to the driving direction of the mobile platform, and determining the corresponding second position information from the section based on the first position information; A high-precision map reconstruction module is used to use the model of the observed object as the model of the target object in the high-precision map to reconstruct the high-precision map.

8. An electronic device, characterized in that: include: processor; as well as A memory having executable codes stored thereon, which, when executed by the processor, causes the processor to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: An executable code is stored thereon, and when the executable code is executed by a processor of an electronic device, the processor is caused to execute the method as claimed in any one of claims 1 to 6.

Images