Map generation method and device, equipment, storage medium and program product

Abstract

The embodiment of the invention provides a map generation method and device, equipment, a storage medium and a program product. The method comprises the following steps: acquiring a scene image of a scene in which a vehicle is located, which is acquired by image acquisition equipment at each moment in a current time period; determining element vector information corresponding to each scene image; constructing a back projection residual model according to the element vector information; and adjusting the element vector information based on a back projection residual model, and generating a scene map of a scene where the vehicle is located in the current time period. The method is used for achieving the effects of improving the map precision, improving the operation efficiency on low-computing-power equipment and remarkably reducing the consumption of computing resources.

Description

Technical Field

[0001] The present application relates to the field of map technology, and in particular to a map generation method, device, equipment, storage medium and program product. Background Art

[0002] During the driving process of the vehicle, based on navigation technology, the unknown environment in which the vehicle is located can be mapped.

[0003] In the prior art, the Global Positioning System (GPS) is used to determine the geographical location, speed and time information of a vehicle so as to draw a map of the environment in which the vehicle is located.

[0004] However, in the above process, in complex scenes such as cities or tunnels, signal transmission may be interfered with or blocked, resulting in inaccurate positioning and, in turn, low accuracy of the drawn map. Summary of the invention

[0005] The embodiments of the present application provide a map generation method, apparatus, device, storage medium and program product to improve map accuracy, enhance operating efficiency on low-computing power devices, and significantly reduce the consumption of computing resources.

[0006] In a first aspect, an embodiment of the present application provides a map generation method, comprising:

[0007] Obtaining scene images of the scene in which the vehicle is located captured by the image acquisition device at each moment in the current time period; and determining element vector information corresponding to each of the scene images; wherein the element vector information includes the projection position of each road element in all scene images in a preset coordinate system; the preset coordinate system is a three-dimensional coordinate system;

[0008] According to the element vector information, a back-projection residual model is constructed; wherein the back-projection residual model represents the projection error of all road elements from the preset coordinate system to the image coordinate system;

[0009] The element vector information is adjusted based on the back-projection residual model to generate a scene map of the scene where the vehicle is located in the current time period.

[0010] In a possible implementation manner, determining element vector information corresponding to each of the scene images includes:

[0011] Performing target detection processing on each of the scene images to obtain a detection result of each road element in each of the scene images; wherein the detection result includes position information of each of the road elements in an image coordinate system corresponding to each of the scene images;

[0012] The element vector information is determined according to the detection result of each of the road elements.

[0013] In a possible implementation, the method further includes:

[0014] Determine the lateral position information of the image acquisition device at each moment according to the detection results of each of the road elements;

[0015] Determine the longitudinal posture information of the image acquisition device at each moment according to the detection result of the road element and the lateral posture information of the image acquisition device at each moment; wherein the lateral posture information and the longitudinal posture information constitute the posture information of the image acquisition device;

[0016] Constructing a back-projection residual model according to the element vector information, including: constructing the back-projection residual model according to the element vector information and the posture information of the image acquisition device at each moment.

[0017] In a possible implementation, each of the road elements includes at least one linear element; determining the element vector information and the lateral position information of the image acquisition device at each moment according to the detection results of each of the road elements includes:

[0018] For each road element in the i-th scene image, according to the intrinsic parameters of the image acquisition device at the i-th moment, a back-projection process is performed on the two-dimensional polyline of the linear element in the i-th image coordinate system to obtain a projection polyline of the two-dimensional polyline in the i-th preset coordinate system;

[0019] Determine the linear element vector of the linear element in the element vector information according to the projection polyline of the two-dimensional polyline in the i-th preset coordinate system; wherein the linear element vector represents the projection position of the linear element in the i-th preset coordinate system;

[0020] According to the first initial vector of the linear element and the linear element vector of the linear element, the lateral posture information of the image acquisition device at the i-th moment is determined; wherein the first initial vector represents the initial position of the linear element in the i-th preset coordinate system; and the lateral posture information includes rotation parameters and translation parameters.

[0021] In a possible implementation manner, each of the road elements further includes at least one aerial element; and the method further includes:

[0022] For each aerial element in the i-th scene image, back-projection processing is performed on each target point according to the position information of each target point in the aerial element and the parameter information of the image acquisition device at the i-th moment, so as to obtain the projection ray of each target point in the i-th preset coordinate system; wherein the position information represents the position of the target point in the i-th image coordinate system; and the parameter information includes the intrinsic parameters, rotation parameters and translation parameters of the image acquisition device;

[0023] Determine the coordinate information of the intersection point of each projection ray in the i-th preset coordinate system as the aerial element vector of the aerial element in the element vector information; wherein the aerial element vector represents the projection position of the aerial element in the i-th preset coordinate system.

[0024] In a possible implementation, each of the road elements further includes at least one sign element; determining the longitudinal posture information of the image acquisition device at each moment according to the detection result of the road element and the lateral posture information of the image acquisition device at each moment includes:

[0025] For each sign element in the i-th scene image, projection processing is performed on the sign element according to the second initial vector of the sign element to obtain projection information of the sign element; wherein the second initial vector represents the initial position of the sign element in the i-th preset coordinate system; and the projection information represents the projection shape of the sign element in the i-th image coordinate system;

[0026] Determine each weight corresponding to the sign element according to the projection information of the sign element and the detection result of each other sign element except the sign element in the i-th scene image, wherein the weight represents the matching degree between the projection information corresponding to the sign element and the detection result of each other sign element;

[0027] Determine the detection result of other sign elements corresponding to the highest weight among the weights, as the target detection result corresponding to the sign element;

[0028] The longitudinal posture information of the image acquisition device at the i-th moment is determined according to the position information in the target detection result corresponding to the sign element, the projection information of the sign element, and the lateral posture information of the image acquisition device at the i-th moment.

[0029] In a possible implementation, a back-projection residual model is constructed according to the element vector information and the position information of the image acquisition device at each moment, including:

[0030] Performing target detection processing on each of the scene images to obtain a detection result of each road element in each of the scene images; wherein the detection result includes position information of each of the road elements in an image coordinate system corresponding to each of the scene images;

[0031] Determine the residual model of each road element according to the element vector information, the position information of the image acquisition device at each moment and the detection result of each road element; wherein the residual model represents the projection error of the road element from the image coordinate system to the preset coordinate system;

[0032] The back-projection residual model is constructed according to the residual model of each road element.

[0033] In a possible implementation, each of the road elements includes at least one sign element and at least one aerial element; determining a residual model of each of the road elements according to the element vector information, the position information of the image acquisition device at each moment, and the detection result of each of the road elements includes:

[0034] For each road element in the i-th scene image, determine first prediction information of the road element according to the posture information of the image acquisition device and the element vector information at the i-th moment; wherein the first prediction information represents the predicted projection position of the road element in the i-th image coordinate system;

[0035] A residual model of the road element at the i-th moment is determined according to the first prediction information of the road element and the position information of the road element in the i-th image coordinate system.

[0036] In a possible implementation manner, each of the road elements further includes at least one linear element; and determining the residual model of each of the road elements according to the element vector information and the detection results of each of the road elements includes:

[0037] For each linear element in the i-th scene image, determine the lateral residual in the residual model of the linear element according to the projection position of the linear element in the i-th preset coordinate system and the position information of the linear element in the i-th image coordinate in the element vector information; wherein the lateral residual represents the forward projection error of the linear element in the i-th image coordinate;

[0038] Determine the longitudinal residual in the residual model of the linear element according to the position information of the linear element in the i-th image coordinate system, the observation speed of the image acquisition device at the i-th moment, and the time difference information; wherein the time difference information represents the time interval between two adjacent moments; and the longitudinal residual represents the depth error between two points in the linear element;

[0039] Determine the height residual in the residual model of the linear element according to the position information of the linear element in the i-th image coordinate system and the installation height of the image acquisition device at the i-th moment; wherein the height residual represents the projection error of the point in the linear element in the height direction in the i-th image coordinate system;

[0040] According to the posture information of the image acquisition device at the i-th moment and the posture information of the image acquisition device at the i+1-th moment, the speed residual model in the residual model of the linear element is determined; wherein the speed residual model represents the motion error of the image acquisition device between the i-th moment and the i+1-th moment.

[0041] In a possible implementation manner, the linear element is a straight road or a curve; for each linear element in the i-th scene image, determining the lateral residual in the residual model of the linear element according to the projection position of the linear element in the i-th preset coordinate system and the position information of the linear element in the i-th image coordinate in the element vector information, including:

[0042] For each straight road in the i-th scene image, second prediction information of the straight road is determined according to the straight road vector of the straight road in the element vector information and the position information of the image acquisition device at the i-th moment; wherein the straight road vector represents the projection position of the edge point of the straight road in the i-th preset coordinate system; and the second prediction information includes the predicted projection position of the edge point of the straight road in the i-th bird's-eye view;

[0043] According to the second prediction information of the straight road and the position information of the straight road in the i-th image coordinate system, a straight road lateral residual in the lateral residual is determined; wherein the straight road lateral residual represents the projection error of the edge point of the straight road in the i-th bird's-eye view.

[0044] In a possible implementation, the method further includes:

[0045] For each curve in the i-th scene image, third prediction information of the curve is determined according to the curve vector of the curve in the element vector information and the position information of the image acquisition device at the i-th moment; wherein the curve vector represents the projection position of two points on the curve in the i-th preset coordinate system; and the third prediction information includes the predicted projection distance of the two points on the curve in the i-th bird's-eye view;

[0046] According to the third prediction information of the curve and the position information of two points on the curve in the i-th image coordinate system, the curve lateral residual in the lateral residual is determined; wherein the curve lateral residual represents the curve projection curvature error of the curve in the i-th bird's-eye view.

[0047] In a possible implementation, determining the velocity residual model in the residual model of the linear element according to the posture information of the image acquisition device at the i-th moment and the posture information of the image acquisition device at the i+1-th moment includes:

[0048] Determine the velocity plane residual in the velocity residual model according to the position information and the observed speed of the image acquisition device at the i-th moment and the position information of the image acquisition device at the i+1-th moment; wherein the velocity plane residual represents the movement speed of the image acquisition device between the i-th moment and the i+1-th moment;

[0049] The velocity height residual in the velocity residual model is determined according to the installation height of the image acquisition device at the i-th moment and the installation height of the image acquisition device at the i+1-th moment; wherein the velocity height residual represents the motion error of the image acquisition device in the height direction between the i-th moment and the i+1-th moment.

[0050] In a possible implementation, generating a scene map of the scene in which the vehicle is located in a current time period according to the element vector information and the back-projection residual model includes:

[0051] Minimizing the value of the back-projection residual model to obtain optimized element vector information and optimized posture information; wherein the optimized posture information includes the posture information of the image acquisition device at each moment after optimization;

[0052] According to the optimized posture information, the optimized element vector information is projected to generate the scene map.

[0053] In a second aspect, an embodiment of the present application provides a map generating device, including:

[0054] A determination module, used to obtain scene images of the scene where the vehicle is located, which are collected by the image acquisition device at each moment in the current time period; and determine the element vector information corresponding to each of the scene images; wherein the element vector information includes the projection position of each road element in all scene images in a preset coordinate system; the preset coordinate system is a three-dimensional coordinate system;

[0055] A construction module, used to construct a back-projection residual model according to the element vector information; wherein the back-projection residual model represents the projection error of all road elements from the preset coordinate system to the image coordinate system;

[0056] A generation module is used to adjust the element vector information based on the back-projection residual model to generate a scene map of the scene where the vehicle is located in a current time period.

[0057] In a possible implementation, the determination module is specifically used to: perform target detection processing on each of the scene images to obtain a detection result for each road element in each of the scene images; wherein the detection result includes position information of each of the road elements in an image coordinate system corresponding to each of the scene images; and determine the element vector information based on the detection result of each of the road elements.

[0058] In a possible implementation, the determination module is further specifically used to: determine the lateral posture information in the posture information of the image acquisition device at each moment according to the detection results of each of the road elements; determine the longitudinal posture information in the posture information of the image acquisition device at each moment according to the detection results of the road elements and the lateral posture information of the image acquisition device at each moment; wherein the lateral posture information and the longitudinal posture information constitute the posture information of the image acquisition device;

[0059] The construction module is specifically used to construct the back-projection residual model according to the element vector information and the posture information of the image acquisition device at each moment.

[0060] In a possible implementation, each of the road elements includes at least one linear element; the determination module is specifically used to: for each linear element in the i-th scene image, perform straight line fitting processing on the linear element according to the position information of the linear element in the i-th image coordinate system to obtain a two-dimensional broken line of the linear element in the i-th image coordinate system; perform back projection processing on the two-dimensional broken line according to the intrinsic parameters of the image acquisition device at the i-th moment to obtain a projection broken line of the two-dimensional broken line in the i-th preset coordinate system; determine the linear element vector of the linear element in the element vector information according to the projection broken line of the two-dimensional broken line in the i-th preset coordinate system; wherein the linear element vector represents the projection position of the linear element in the i-th preset coordinate system; determine the lateral posture information of the image acquisition device at the i-th moment according to the first initial vector of the linear element and the linear element vector of the linear element; wherein the first initial vector represents the initial position of the linear element in the i-th preset coordinate system; the lateral posture information includes rotation parameters and translation parameters.

[0061] In a possible implementation, each of the road elements also includes at least one aerial element; the determination module is further specifically used to: for each aerial element in the i-th scene image, perform back-projection processing on each target point according to the position information of each target point in the aerial element and the parameter information of the image acquisition device at the i-th moment, so as to obtain the projection ray of each target point in the i-th preset coordinate system; wherein the position information represents the position of the target point in the i-th image coordinate system; the parameter information includes the intrinsic parameters, rotation parameters and translation parameters of the image acquisition device; determine the coordinate information of the intersection of each of the projection rays in the i-th preset coordinate system, which is the aerial element vector of the aerial element in the element vector information; wherein the aerial element vector represents the projection position of the aerial element in the i-th preset coordinate system.

[0062] In a possible implementation, each of the road elements also includes at least one sign element; the determination module is further specifically used to: for each sign element in the i-th scene image, project the sign element according to the second initial vector of the sign element to obtain the projection information of the sign element; wherein the second initial vector represents the initial position of the sign element in the i-th preset coordinate system; the projection information represents the projection shape of the sign element in the i-th image coordinate system; according to the projection information of the sign element and the detection result of each other sign element except the sign element in the i-th scene image, determine each weight corresponding to the sign element, wherein the weight represents the degree of matching between the projection information corresponding to the sign element and the detection result of each other sign element; determine the detection result of the other sign element corresponding to the highest weight among the weights as the target detection result corresponding to the sign element; determine the longitudinal posture information of the image acquisition device at the i-th moment according to the position information in the target detection result corresponding to the sign element, the projection information of the sign element, and the lateral posture information of the image acquisition device at the i-th moment.

[0063] In a possible implementation, the construction module is specifically used to: perform target detection processing on each of the scene images to obtain a detection result for each road element in each of the scene images; wherein the detection result includes the position information of each of the road elements in the image coordinate system corresponding to each of the scene images; determine the residual model of each of the road elements based on the element vector information, the posture information of the image acquisition device at each moment and the detection result of each of the road elements; wherein the residual model represents the projection error of the road element when it is back-projected from the image coordinate system to the preset coordinate system; and construct the back-projection residual model based on the residual model of each of the road elements.

[0064] In a possible implementation, each of the road elements includes at least one sign element and at least one aerial element; a construction module is specifically used to: for each road element in the i-th scene image, determine the first prediction information of the road element based on the posture information of the image acquisition device and the element vector information at the i-th moment; wherein the first prediction information represents the predicted projection position of the road element in the i-th image coordinate system; determine the residual model of the road element at the i-th moment based on the first prediction information of the road element and the position information of the road element in the i-th image coordinate system.

[0065] In a possible implementation, each of the road elements further includes at least one linear element; the construction module is further specifically used to: for each linear element in the i-th scene image, determine the lateral residual in the residual model of the linear element according to the projection position of the linear element in the i-preset coordinate system in the element vector information and the position information of the linear element in the i-image coordinate; wherein the lateral residual represents the forward projection error of the linear element in the i-image coordinate; determine the longitudinal residual in the residual model of the linear element according to the position information of the linear element in the i-image coordinate system, the observation speed of the image acquisition device at the i-th moment and the time difference information; wherein the time difference information Characterizes the time interval between two adjacent moments; the longitudinal residual characterizes the depth error of two points in the linear element; determines the height residual in the residual model of the linear element according to the position information of the linear element in the i-th image coordinate system and the installation height of the image acquisition device at the i-th moment; wherein the height residual characterizes the projection error of the point in the linear element in the height direction in the i-th image coordinate system; determines the speed residual model in the residual model of the linear element according to the posture information of the image acquisition device at the i-th moment and the posture information of the image acquisition device at the i+1-th moment; wherein the speed residual model characterizes the motion error of the image acquisition device between the i-th moment and the i+1-th moment.

[0066] In a possible implementation, the linear element is a straight road or a curve; the construction module is further specifically used to: for each straight road in the i-th scene image, determine the second prediction information of the straight road according to the straight road vector of the straight road in the element vector information and the posture information of the image acquisition device at the i-th moment; wherein the straight road vector represents the projection position of the edge point of the straight road in the i-th preset coordinate system; the second prediction information includes the predicted projection position of the edge point of the straight road in the i-th bird's-eye view; determine the straight road lateral residual in the lateral residual according to the second prediction information of the straight road and the position information of the straight road in the i-th image coordinate system; wherein the straight road lateral residual represents the projection error of the edge point of the straight road in the i-th bird's-eye view.

[0067] In a possible implementation, the construction module is further specifically used to: for each curve in the i-th scene image, determine the third prediction information of the curve according to the curve vector of the curve in the element vector information and the posture information of the image acquisition device at the i-th moment; wherein the curve vector represents the projection position of two points on the curve in the i-th preset coordinate system; the third prediction information includes the predicted projection distance of the two points on the curve in the i-th bird's-eye view; determine the curve lateral residual in the lateral residual according to the third prediction information of the curve and the position information of the two points on the curve in the i-image coordinate system; wherein the curve lateral residual represents the curve projection curvature error in the i-th bird's-eye view.

[0068] In a possible implementation, the construction module is further specifically used to: determine the velocity plane residual in the velocity residual model based on the position information and observed speed of the image acquisition device at the i-th moment and the position information of the image acquisition device at the i+1-th moment; wherein the velocity plane residual represents the movement speed of the image acquisition device between the i-th moment and the i+1-th moment; determine the velocity height residual in the velocity residual model based on the installation height of the image acquisition device at the i-th moment and the installation height of the image acquisition device at the i+1-th moment; wherein the velocity height residual represents the motion error of the image acquisition device in the height direction between the i-th moment and the i+1-th moment.

[0069] In one possible implementation, the generation module is specifically used to: minimize the value of the back-projection residual model to obtain optimized element vector information and optimized pose information; wherein the optimized pose information includes the pose information of the image acquisition device at each moment after optimization; and project the optimized element vector information according to the optimized pose information to generate the scene map.

[0070] In a third aspect, an embodiment of the present application provides an electronic device, including: a memory, a processor;

[0071] The memory stores computer-executable instructions;

[0072] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.

[0073] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementations of the first aspect.

[0074] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementation methods of the first aspect.

[0075] The map generation method, device, equipment, storage medium and program product provided in the embodiment of the present application are to project each road element in the scene image of the scene where the vehicle is located at each moment into a preset coordinate system to obtain element vector information representing the projection position of each road element in the preset coordinate system, and to construct a back-projection residual model based on the element vector information to represent the projection error of all road elements from the preset coordinate system to the image coordinate system, so as to optimize the element vector information, and then obtain the scene map of the scene where the vehicle is located through the map drawing technology. Furthermore, by projecting each road element in the scene image into the preset coordinate system, element vector information is obtained, and based on the generated back-projection residual model, the element vector is accurately optimized to obtain a more accurate element vector, which, when applied to map generation, can achieve the effect of improving the accuracy of the drawn map; and by collecting scene images through a low-power rolling shutter camera and using a low-precision inertial navigation system during combined navigation, the operating efficiency on low-computing power devices can be improved, and the consumption of computing resources can be significantly reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0077] Figure 1 A schematic diagram of an application scenario provided for this application;

[0078] Figure 2 A flowchart of a map generation method provided in an embodiment of the present application;

[0079] Figure 3 A flowchart of another map generation method provided in an embodiment of the present application;

[0080] Figure 4 A flowchart of a map construction process provided in an embodiment of the present application;

[0081] Figure 5 A schematic diagram of the coordinates of a detection result frame provided in an embodiment of the present application;

[0082] Figure 6 A schematic diagram of a semantic tracking result provided in an embodiment of the present application;

[0083] Figure 7 A schematic diagram of a coordinate system conversion process provided in an embodiment of the present application;

[0084] Figure 8 A schematic diagram of a 2D intersection process provided in an embodiment of the present application;

[0085] Fig. 9 A schematic diagram of a semantic VO provided in an embodiment of the present application;

[0086] Fig.10 A schematic diagram of the vector optimization effect based on the urban canyon scene provided in an embodiment of the present application;

[0087] Fig.11 A schematic diagram of the vector optimization effect based on a common road scenario provided in an embodiment of the present application;

[0088] Fig.12 A schematic diagram of the vector optimization effect based on a high-speed scenario provided in an embodiment of the present application;

[0089] Fig.13 A schematic diagram of the vector optimization effect based on the gantry scenario provided in an embodiment of the present application;

[0090] Fig.14 A schematic diagram of the structure of a map generating device provided in an embodiment of the present application;

[0091] Fig.15 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0092] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0093] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0094] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0095] In addition, this application involves big data analysis of user information (including but not limited to personal biometrics, identity data, consumption data, asset data, electronic terminal operation data, etc.), and the use of artificial intelligence technology for automated decision-making, and a technical solution for making decisions that have a significant impact on personal rights and interests based on the results of automated decision-making. The application provides users with corresponding operation entrances for them to choose to agree or reject the results of automated decision-making; if the user chooses to reject, the expert decision-making process will be entered.

[0096] It should be noted that the present application can be used in the field of map technology, and can also be used in any field other than the field of map technology, and the application field of the present application is not limited.

[0097] Figure 1 A schematic diagram of an application scenario provided for this application, such as Figure 1 As shown, the specific application scenario of the present application includes an electronic device 101 and a sensor 102 in a vehicle. Relying on the Global Positioning System (GPS), the electronic device 101 obtains a signal from the sensor 102 to determine the vehicle's geographic location, vehicle speed and time information, so as to draw a map of the vehicle's environment.

[0098] In one example, based on Simultaneous Localization and Mapping (SLAM), the vehicle posture can be estimated in real time while mapping the surrounding environment.

[0099] Based on the above scenarios, it can be seen that when positioning and drawing maps through the global positioning system, there is a technical problem that signal transmission may be interfered or blocked in complex scenes such as cities or tunnels, resulting in inaccurate positioning, and then leading to low accuracy of the drawn maps.

[0100] The map generation method provided in the present application obtains element vector information by projecting each road element in the scene image into a preset coordinate system, and uses a technical means to accurately optimize the element vector based on the generated back-projection residual model, thereby solving the technical problem of low accuracy of maps drawn in complex scenes such as cities or tunnels.

[0101] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0102] Figure 2 A flowchart of a map generation method provided in an embodiment of the present application is shown as follows: Figure 2 As shown, the method includes:

[0103] 201. Obtain scene images of the scene in which the vehicle is located, which are captured by an image capture device at each moment in the current time period; and determine element vector information corresponding to each scene image; wherein the element vector information includes the projection position of each road element in all scene images in a preset coordinate system; the preset coordinate system is a three-dimensional coordinate system.

[0104] Exemplarily, the execution subject of this embodiment may be an electronic device in a vehicle, hereinafter referred to as the device. An image acquisition device in the vehicle, such as a monocular rolling shutter camera, collects scene images of the scene in which the vehicle is located at each moment in the current time period and transmits them to the device. Based on a preset map element recognition technology, the device performs recognition processing on each acquired scene image to obtain each road element in all scene images, including lane lines, traffic signs, poles, crosswalks, stop lines and other road elements, and uses a low-precision inertial navigation system's vectorization technology, such as by fitting polylines or Bezier curves in a preset coordinate system, so as to represent the boundary or center line as vector data, and obtain the projection position of each road element in the preset coordinate system, that is, the element vector information. The preset coordinate system is a three-dimensional coordinate system, such as the projection position of each road element in the three-dimensional camera coordinate system.

[0105] 202. Construct a back-projection residual model based on the element vector information; wherein the back-projection residual model represents the projection error of all road elements when back-projecting from a preset coordinate system to an image coordinate system.

[0106] Exemplarily, the device can process the element vector information based on the obtained element vector information through a preset back-projection residual model construction method such as a deep learning model; it can also determine the category of each road element, such as physical signs such as traffic signs and poles, and ground signs such as lane lines, crosswalks, stop lines, etc., and construct a back-projection residual model for the element vector information in combination with the category of each road element, so as to obtain the projection error of all road elements back-projected from the preset coordinate system to the image coordinate system.

[0107] For example, the device generates a factor graph based on the obtained element vector information and the category of each road element. In the factor graph, the edge between each node represents a factor, and the nodes in the factor graph are constrained by the residual function. The error between the observed value and the predicted value of node j in frame i can be represented by the residual function, that is, the back-projection residual model is obtained to optimize each node position and camera pose.

[0108] 203. Adjust the element vector information based on the back-projection residual model to generate a scene map of the scene where the vehicle is located in the current time period.

[0109] Exemplarily, the device optimizes the projection position of each road element in the element vector information in the preset coordinate system according to the obtained back-projection residual model, obtains the optimized projection position of each road element in the preset coordinate system, and then, based on the map drawing technology, further obtains the scene map of the scene in which the vehicle is located in the current time period.

[0110] In this embodiment, a map generation method is provided. On the one hand, each road element in the scene image is projected into a preset coordinate system to obtain element vector information, and based on the generated back-projection residual model, the element vector is accurately optimized to obtain a more accurate element vector. When applied to map generation, the accuracy of the drawn map can be improved. On the other hand, scene images are collected by a low-power rolling shutter camera, which can save 2 to 10 times the cost compared to a global shutter camera. A low-precision inertial navigation system is used during combined navigation, which can meet positioning requirements without significantly increasing hardware costs and adapt to the operation of low-computing power devices, thereby reducing the system's hardware dependence to adapt to a wider range of application scenarios. It can also improve the operating efficiency on low-computing power devices, so that high-precision positioning and mapping can be achieved in a variety of scenarios, and the consumption of computing resources can be significantly reduced.

[0111] Figure 3 A flowchart of another map generation method provided in an embodiment of the present application is shown as follows: Figure 3 As shown, the method includes:

[0112] 301. Obtain a scene image of a scene in which a vehicle is located, which is captured by an image capture device at each moment in a current time period.

[0113] Exemplarily, this step may refer to step 201 and will not be described in detail here.

[0114] 302. Perform target detection processing on each scene image to obtain a detection result of each road element in each scene image; wherein the detection result includes position information of each road element in an image coordinate system corresponding to each scene image.

[0115] Exemplarily, the device performs target detection processing on each scene image based on the map element target detection and tracking technology in a preset inertial measurement unit (IMU), and obtains the detection results of each road element in each scene image, including the position information of the road element in the image coordinate system corresponding to each scene image.

[0116] For example, Figure 4 A flowchart of a map construction process provided in an embodiment of the present application is shown as follows: Figure 4 As shown, for example, a deep learning network for common target detection is used to implement target detection and tracking processing for each acquired scene image based on specific semantic ground elements such as lane lines, traffic signs, poles, crosswalks, stop lines, etc., to obtain each road element in each scene image, and then through artificial intelligence (AI) image perception, AI semantic recognition is performed on each road element in each scene image, including high-precision traffic sign corner point detection, high-precision lane line edge detection, high-precision pole midpoint detection and other processes, to obtain the rectangular frame coordinates bbox (x, y, w, h) of the specific traffic target in each scene image, Figure 5 A schematic diagram of the coordinates of a detection result frame provided in an embodiment of the present application is shown in FIG. Figure 5 As shown in the figure, each image may contain multiple semantic target results, that is, multiple detection results. Each detection result is an object, which is composed of a bbox (x, y, w, h) representing the position. In the same image coordinate system, it represents the horizontal and vertical coordinates of the upper left corner of the detection box and the width and height of the detection result box. Figure 4 ,After target detection, the traditional target tracking network is used to track three types of ,semantic elements, including aerial object element tracking, linear element tracking, and ground point element tracking. ,The detected target area is used as the element position initialization. Figure 6 A schematic diagram of a semantic tracking result provided in an embodiment of the present application is shown in FIG. Figure 6As shown, after each target semantic element is identified in each frame of the image, the identification information corresponding to each target semantic element can be marked and displayed, for example Figure 6 The aerial object elements include semantic elements 1, 2, 3, 7, and 8, the linear elements include semantic elements 4, 5, and 6, and the ground point elements include semantic elements 9, 10, and 11. The detected target semantic elements are associated between consecutive frames. According to the identification information corresponding to each target semantic element, it is determined which target semantic elements are the same. An inertial navigation system (INS) track is maintained for each tracked target semantic element. The position information and category of each tracked target semantic element in the video sequence are recorded, and the track is associated with the identification information of each target semantic element one by one to obtain the track of each target semantic element in the video sequence, that is, the semantic tracking results including the target position, category, and semantic elements of all frame scene images where different road elements appear are obtained through the network to obtain the final detection result.

[0117] 303. Determine element vector information based on the detection results of each road element.

[0118] Exemplarily, the device back-projects each road element in each scene image from the corresponding image coordinate system to the corresponding preset coordinate system based on the position information of each road element in the image coordinate system corresponding to each scene image, that is, the detection result, to obtain the projection position of each road element in the corresponding preset coordinate system, that is, the element vector information, and then obtains a semantic vector with relatively low accuracy. In addition, for each road element in each scene image, based on the position information of the road element in the image coordinate system corresponding to each scene image and the projection position of the road element in the preset coordinate system corresponding to each scene image.

[0119] In one example, step 303 further includes the following steps:

[0120] The first step is to determine the lateral position information of the image acquisition device at each moment according to the detection results of each road element.

[0121] The second step is to determine the longitudinal posture information of the image acquisition device at each moment according to the detection results of the road elements and the lateral posture information of the image acquisition device at each moment; wherein the lateral posture information and the longitudinal posture information constitute the posture information of the image acquisition device.

[0122] Exemplarily, the device projects each road element in each scene image from the corresponding image coordinate system to the corresponding preset coordinate system according to the position information of each road element in the image coordinate system corresponding to each scene image, i.e., the detection result, to obtain the projection position of each road element in the corresponding preset coordinate system, i.e., the element vector information, and, for each road element in each scene image, based on the projection model of the preset image acquisition device, the position information of the road element in the image coordinate system corresponding to each scene image and the projection position of the road element in the preset coordinate system corresponding to each scene image are calculated and processed, so that the lateral posture information of the image acquisition device at the corresponding moment of each scene image can be determined to complete the lateral positioning. The device can calculate and process the lateral posture information of the image acquisition device at the corresponding moment of each scene image and the position information of each road element in the image coordinate system corresponding to the scene image through a preset (Perspective-n-Point, referred to as PnP) algorithm, and obtain the longitudinal posture information in the posture information of the image acquisition device at each moment, for example, the height of the image acquisition device at each moment. Among them, during the driving process of the vehicle, the lateral movement of the vehicle is usually more frequent and significant than the longitudinal movement. The change range of the lateral posture is larger, while the change range of the longitudinal posture is relatively small. The change of the lateral posture has a greater impact on the feature points in the image captured by the image acquisition device in the vehicle. Therefore, in the posture determination process of the image acquisition device, the lateral posture (also called horizontal posture or azimuth) is determined first, and then the longitudinal posture (also called pitch angle or vertical posture) is determined. On the one hand, it can help to quickly narrow the search range and reduce the calculation complexity. On the other hand, it can reduce the accumulation of errors and improve the stability and accuracy of the image data calculation system.

[0123] In one example, constructing a back-projection residual model according to the element vector information includes: constructing the back-projection residual model according to the element vector information and the posture information of the image acquisition device at each moment.

[0124] Exemplarily, the device can process the element vector information and the position information of the image acquisition device at each moment through a preset back-projection residual model construction method such as geometric correction or a deep learning model based on the obtained element vector information; it can also determine and combine the category of each road element, such as physical signs such as traffic signs and poles, and ground signs such as lane lines, crosswalks, and stop lines, and construct a back-projection residual model for the position information of the image acquisition device at each moment of the element vector information in combination with the category of each road element, so as to obtain the projection error of all road elements back-projected from the preset coordinate system to the image coordinate system.

[0125] In one example, each road element includes at least one linear element; the first step includes:

[0126] Step 1: For each linear element in the i-th scene image, a straight line fitting process is performed on the linear element according to the position information of the linear element in the i-th image coordinate system to obtain a two-dimensional polyline of the linear element in the i-th image coordinate system.

[0127] Step 2: According to the intrinsic parameters of the image acquisition device at the i-th moment, the two-dimensional polyline is back-projected to obtain the projection polyline of the two-dimensional polyline in the i-th preset coordinate system.

[0128] Step three: determine the linear element vector of the linear element in the element vector information according to the projection polyline of the two-dimensional polyline in the i-th preset coordinate system; wherein the linear element vector represents the projection position of the linear element in the i-th preset coordinate system.

[0129] Step 4: Determine the lateral posture information of the image acquisition device at the i-th moment based on the first initial vector of the linear element and the linear element vector of the linear element; wherein the first initial vector represents the initial position of the linear element in the i-th preset coordinate system; and the lateral posture information includes rotation parameters and translation parameters.

[0130] Specifically, each road element is classified, including at least one linear element, such as a lane line. For each linear element in the i-th scene image, a detection result is obtained through target detection. Using the image coordinate system corresponding to the i-th scene image, i.e., the position information in the i-th image coordinate system, each linear element is subjected to a linear fitting algorithm to obtain a two-dimensional polyline of each linear element in the i-th image coordinate system. According to the time corresponding to the i-th scene image, i.e., the intrinsic parameters (such as focal length, optical center, etc.) of the image acquisition device at the i-th time, the two-dimensional polyline of each linear element in the i-th image coordinate system is back-projected to obtain the projection polyline of the two-dimensional polyline corresponding to each linear element in the i-th preset coordinate system. The device obtains the projection position of the linear element in the i-th preset coordinate system based on the projection polyline corresponding to each linear element, and then obtains the projection position of all linear elements in the element vector information in the i-th preset coordinate system, i.e., the linear element vector. Based on the i-th scene image obtained by the image acquisition device, the initial position of each linear element in the i-th scene image in the i-th preset coordinate system, that is, the first initial vector, can be obtained, such as the initial position in the camera coordinate system. Based on the preset algorithm, the first initial vector of each linear element and the linear element vector of the linear element are calculated and processed to obtain the lateral posture information of the image acquisition device at the i-th moment, including the rotation parameters and translation parameters in the external parameters of the image acquisition device, such as the rotation matrix R and the translation vector T.

[0131] For example, combining Figure 4 , Single-frame vectorization of linear elements based on external parameters. For linear elements such as lane lines, single-frame reconstruction is performed based on the lane lines extracted by semantic segmentation in the target detection algorithm of the original image. A straight line fitting algorithm is used to generate a two-dimensional broken line based on semantic segmentation. Then, the four parameters obtained by camera calibration, pitch angle pitch, roll angle roll, yaw angle yaw, and vehicle height h, are used to back-project the fitted two-dimensional broken line into three-dimensional space to obtain a three-dimensional broken line. The three-dimensional broken line is aligned with the high-precision map to obtain the rotation matrix R and translation matrix T. For example, Figure 7 A schematic diagram of a coordinate system conversion process provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, the camera-based projection model is represented by homogeneous coordinates. In the three-dimensional world, a point P in the camera coordinate system (Oc, Xc, Yc, Zc) is i (x i ,y i ,z i ) Τ It can be converted into a two-dimensional coordinate (u) on the image plane such as the image physical coordinate system (O, X, Y) through the camera internal and external parameter matrix i ,v i ) Τ , where the camera’s internal and external parameters include the camera’s internal parameter matrix K and external parameter matrix [R|T], which represent the camera’s intrinsic parameters (such as focal length f, optical center, etc.) and the camera’s rotation and translation relationship, as well as the height h; the camera’s internal parameter matrix K is usually expressed as Among them, f x and f y is the focal length of the camera in the x and y directions, c x and c y is the pixel coordinate of the optical center. The external parameters of the camera are composed of the rotation matrix R and the translation vector T. The three-dimensional point P i (x i ,y i ,z i ) Τ The projection on the image plane can be calculated by the following matrix operation p i =K(RT)P i , through the normalization of homogeneous coordinates, the three-dimensional point is projected onto the two-dimensional image plane, and we get Among them, p x ,p y ,p z is the result of the above matrix operation.

[0132] In one example, each road element also includes at least one aerial element; the first step of step 303 also includes:

[0133] Step 1: For each aerial element in the i-th scene image, back-project each target point according to the position information of each target point in the aerial element and the parameter information of the image acquisition device at the i-th moment, to obtain the projection ray of each target point in the i-th preset coordinate system; wherein the position information represents the position of the target point in the i-th image coordinate system; the parameter information includes the intrinsic parameters, rotation parameters and translation parameters of the image acquisition device.

[0134] Step 2: Determine the coordinate information of the intersection of each projection ray in the i-th preset coordinate system, which is the aerial element vector of the aerial element in the element vector information; wherein the aerial element vector represents the projection position of the aerial element in the i-th preset coordinate system.

[0135] Specifically, all road elements also include one or more aerial elements. For each aerial element in the i-th scene image, such as a pole, based on the projection ray, that is, the position information, of each target point in each aerial element, such as the midpoint and the endpoint, in the i-th preset coordinate system, and then using the parameter information of the image acquisition device at the i-th moment, including the intrinsic parameters, rotation parameters and translation parameters of the image acquisition device, each target point in each aerial element is back-projected to the i-th preset coordinate system to obtain the projection ray of each target point in each aerial element in the i-th preset coordinate system; determine the intersection of the projection rays of each target point in each aerial element in the i-th preset coordinate system, and determine the coordinate information of the intersection in the i-th preset coordinate system as the projection position of each aerial element in the i-th preset coordinate system, that is, obtain the aerial element vector of each aerial element.

[0136] For example, combining Figure 4 ,Based on the vectorization of aerial features based on the intersection of two frames of 2D,for aerial features such as poles, captured by multi-view cameras, Figure 8 A schematic diagram of a 2D intersection process provided in an embodiment of the present application is shown as follows: Figure 8 As shown, in each scene image, the 2D coordinates of the midpoint of each rod-shaped object in the corresponding image coordinate system (including the X-axis parallel to the ground and the Z-axis perpendicular to the ground) are extracted by image processing methods, and the camera's intrinsic and extrinsic parameters are used for back projection to obtain the corresponding three-dimensional rays a, b, and c. These rays have theoretical intersections. With c as the baseline, the angles α, β, and γ obtained by the intersection of each ray are determined. The angles α, β, and γ obtained by the intersection of each ray are calculated through trigonometric functions to obtain the position of the intersection, which is the position of the rod-shaped object in three-dimensional space.

[0137] In one example, each road element also includes at least one sign element; the second step includes:

[0138] Step 1: For each sign element in the i-th scene image, project the sign element according to the second initial vector of the sign element to obtain the projection information of the sign element; wherein the second initial vector represents the initial position of the sign element in the i-th preset coordinate system; and the projection information represents the projection shape of the sign element in the i-th image coordinate system.

[0139] Step 2: Determine the weights corresponding to the sign elements based on the projection information of the sign elements and the detection results of each other sign element except the sign element in the i-th scene image, where the weight represents the degree of match between the projection information corresponding to the sign element and the detection results of each other sign element.

[0140] Step 3: Determine the detection results of other sign elements corresponding to the highest weight among all weights, which are the target detection results corresponding to the sign elements.

[0141] Step 4: Determine the longitudinal posture information of the image acquisition device at the i-th moment based on the position information in the target detection result corresponding to the sign element, the projection information of the sign element, and the lateral posture information of the image acquisition device at the i-th moment.

[0142] Specifically, all road elements also include at least one sign element, such as visual elements such as signs. For each sign element in the i-th scene image, based on the initial image acquisition and recognition processing, the initial position of each sign element in the i-th preset coordinate system, that is, the second initial vector, can be obtained. According to the second initial vector of each sign element, each sign element is projected to obtain the projection shape of the sign element in the i-th image coordinate system, that is, the projection information. According to the detection results obtained by the aforementioned target detection process, the detection results of each other sign element in the i-th scene image except the sign element are extracted, and the projection information of the sign element and the detection results of each other sign element are matched. Obtain the degree of match between the projection information corresponding to each sign element and the detection result of each other sign element, that is, the various weights corresponding to each sign element, determine the detection result of other sign elements with the highest weight among the various weights corresponding to each sign element, and determine the detection result as the target detection result corresponding to the sign element, and according to the preset PnP (Perspective-n-Point) algorithm, calculate and process the position information of other sign elements corresponding to the target detection result in the i-th image coordinate system, the projection shape of the sign element in the i-th image coordinate system, and the lateral posture information of the image acquisition device at the i-th moment, and obtain the longitudinal posture information of the image acquisition device at the i-th moment.

[0143] For example, when the horizontal positioning is completed, the visual elements such as signs and the initial vector of the signs are only one vertical deviation away from each other. Projection processing is performed, where d is the depth factor. The best matching relationship is determined by determining the weight between the shape of each semantic element projection and each rectangle of the target detection. After obtaining the matching relationship, a PnP problem is established, and the lateral positioning posture is substituted into the longitudinal posture to calculate, thereby reducing the longitudinal error.

[0144] 304. Perform target detection processing on each scene image to obtain a detection result of each road element in each scene image; wherein the detection result includes position information of each road element in an image coordinate system corresponding to each scene image.

[0145] Exemplarily, the device performs target detection processing on each scene image based on the map element target detection and tracking technology in a preset inertial measurement unit (IMU), and obtains the detection results of each road element in each scene image, including the position information of the road element in the image coordinate system corresponding to each scene image.

[0146] 305. Determine the residual model of each road element according to the element vector information, the position information of the image acquisition device at each moment and the detection result of each road element; wherein the residual model represents the projection error of the road element from the image coordinate system to the preset coordinate system.

[0147] Exemplarily, based on a preset residual model building method, the device models the projection position of each road element in the element vector information in the corresponding preset coordinate system, the posture information of the image acquisition device at each moment, and the position information of each road element in the image coordinate system corresponding to each scene image, to obtain a residual model of each road element to characterize the projection error of each road element when back-projected from the image coordinate system to the preset coordinate system.

[0148] For example, combining Figure 4 Based on Visual Odometry (VO) technology, VO is completed through 3D point feature modeling, 2D point feature modeling, linear feature modeling, speed modeling and other processes in various road elements, including traffic signs, poles, lane lines, and scenes without semantics. Comprehensive poles, signs, lines, crosswalks, stop lines, etc. Fig. 9 A semantic VO schematic diagram provided in an embodiment of the present application, such as Fig. 9As shown in the figure, each semantic element T in the map element is relatively evenly distributed in the image, which is conducive to improving the optimization accuracy. That is, the residual model of each road element can be obtained through the graph optimization modeling method. By introducing semantic elements and graph optimization modeling, the cumulative error is reduced and the positioning accuracy in complex outdoor scenes is significantly improved.

[0149] In one example, each road element includes at least one sign element and at least one aerial element; step 305 includes:

[0150] The first step of step 305 is to determine, for each road element in the i-th scene image, the first prediction information of the road element according to the posture information and element vector information of the image acquisition device at the i-th moment; wherein the first prediction information represents the predicted projection position of the road element in the i-th image coordinate system.

[0151] The second step of step 305 is to determine the residual model of the road element at the i-th moment according to the first prediction information of the road element and the position information of the road element in the i-th image coordinate system.

[0152] Exemplarily, each road element includes at least one sign element and at least one aerial element; for each sign element or each aerial element in the i-th scene image, the projection position of each sign element or each aerial element in the i-th preset coordinate system is extracted according to the element vector information, and the posture information of the image acquisition device at the i-th moment is used to perform predictive projection processing on each sign element or each aerial element in the i-th scene image through the aforementioned matrix operation process to obtain the predicted projection position of each sign element or each aerial element in the i-th image coordinate system, that is, the first prediction information; based on the preset residual model construction method, the first prediction information of each sign element or each aerial element and the position information of each sign element or each aerial element in the i-th image coordinate system are processed to obtain the residual model of each sign element or each aerial element at the i-th moment.

[0153] For example, for the back-projection residual modeling of the corner points of traffic signs, it is assumed that the actual observed position of a corner point of the traffic sign in the i-th frame image coordinate system is The theoretical projection position calculated based on the camera pose is (u i ,v i ), the residual model optimizes the consistency of the corner point in multiple frames by calculating the difference between the two. For rod-shaped objects (such as street light poles and traffic signal poles), we only consider the projection error on the horizontal plane and ignore the error in the height direction. Assume that the projection position of the pole in the image is u j The actual observation position is The residual model of the rod

[0154] In one example, each road element also includes at least one linear element; step 305 includes:

[0155] Step 1. For each linear element in the i-th scene image, determine the lateral residual in the residual model of the linear element based on the projection position of the linear element in the i-th preset coordinate system in the element vector information and the position information of the linear element in the i-th image coordinate; wherein the lateral residual represents the forward projection error of the linear element in the i-th image coordinate.

[0156] Step 2: Determine the longitudinal residual in the residual model of the linear element based on the position information of the linear element in the i-th image coordinate system, the observation speed of the image acquisition device at the i-th moment, and the time difference information; wherein the time difference information represents the time interval between two adjacent moments; and the longitudinal residual represents the depth error between two points in the linear element.

[0157] Step three, determine the height residual in the residual model of the linear element based on the position information of the linear element in the i-th image coordinate system and the installation height of the image acquisition device at the i-th moment; wherein the height residual represents the projection error of the point in the linear element in the height direction in the i-th image coordinate system.

[0158] Step 4: Determine the velocity residual model in the residual model of the linear element based on the posture information of the image acquisition device at the i-th moment and the posture information of the image acquisition device at the i+1-th moment; wherein the velocity residual model represents the motion error of the image acquisition device between the i-th moment and the i+1-th moment.

[0159] Exemplarily, each road element includes at least one linear element. For each linear element in the i-th scene image, the projection position of each linear element in the i-th preset coordinate system is extracted according to the element vector information, and based on the lateral residual generation model, the projection position of each linear element in the i-th preset coordinate system and the position information of the linear element in the i-image coordinate are processed to obtain the lateral residual in the residual model of each linear element to characterize the forward projection error of each linear element in the i-image coordinate; according to the detection result of each linear element, the position information of each linear element in the i-image coordinate system is extracted, and the position information of each linear element in the i-image coordinate system, the observation speed of the image acquisition device at the i-th moment, and the time between two adjacent moments are generated. The time interval, i.e., the time difference information, is calculated and processed to obtain the longitudinal residual in the residual model of each linear element to characterize the depth error of two points in each linear element; the position information of each linear element in the i-th image coordinate system and the installation height of the image acquisition device at the i-th moment are calculated and processed to obtain the height residual in the residual model of each linear element to characterize the projection error of the point in each linear element in the height direction in the i-th image coordinate system; in the case where there are no semantic elements in the scene image, the posture information of the image acquisition device at the i-th moment and the posture information of the image acquisition device at the i+1th moment can be calculated and processed to obtain the speed residual model in the residual model of the linear element to characterize the motion error of the image acquisition device between the i-th moment and the i+1th moment. Then, the residual model of each linear element and the residual model when there is no road element are obtained.

[0160] For example, lane lines, as linear semantic elements, require forward projection in multiple frames to ensure the consistency and accuracy of lane lines. The longitudinal residual measures the depth error between two points on the lane line. Assume that the actual world coordinates of the two points on the lane line are P bi and P bj , the observation speed of the camera is v, and the time difference between the two frames is dt, then the longitudinal residual model Where resolution is the resolution factor used for normalization. By minimizing the longitudinal error, the depth projection error of the lane line edge points in the multi-frame image can be optimized. The height residual of the lane line is used to measure the projection error of the lane line point in the height direction. The height residual model Among them, P b [2] represents the coordinate of the lane line point on the Z axis (height direction), cam h Indicates the installation height of the camera.

[0161] In one example, the linear element is a straight road or a curved road; step 1 of step 305 includes:

[0162] The first step is to determine, for each straight road in the i-th scene image, second prediction information of the straight road according to the straight road vector of the straight road in the element vector information and the posture information of the image acquisition device at the i-th moment; wherein the straight road vector represents the projection position of the edge point of the straight road in the i-th preset coordinate system; and the second prediction information includes the predicted projection position of the edge point of the straight road in the i-th bird's-eye view.

[0163] The second step is to determine the lateral residual of the straight road in the lateral residual according to the second prediction information of the straight road and the position information of the straight road in the i-th image coordinate system; wherein the lateral residual of the straight road represents the projection error of the edge point of the straight road in the i-th bird's-eye view.

[0164] Specifically, each linear element is divided into a straight road and a curved road. For each straight road in the i-th scene image, according to the element vector information, the projection position of the edge point of each straight road in the i-th preset coordinate system is extracted, that is, the straight road vector of each straight road, and the straight road vector of each straight road and the posture information of the image acquisition device at the i-th moment are calculated and processed to obtain the predicted projection position of the edge point of each straight road in the i-th bird's-eye view, that is, the second prediction information of each straight road. Based on the preset residual calculation method, the second prediction information of each straight road and the position information of each straight road in the i-th image coordinate system are calculated to obtain the straight road lateral residual in the lateral residual to characterize the projection error of the edge point of each straight road in the i-th bird's-eye view.

[0165] For example, the back-projection residual model of lane lines can be divided into three dimensions: horizontal, vertical, and vertical. Depending on the scene, lane lines can be divided into straight and curved. For straight roads, the horizontal residual is based on the projection error of the lane line edge points in the bird's-eye view. Assuming the actual observation position is The theoretical projection position is bev x , then the lateral residual

[0166]

[0167] In one example, the linear element is a straight road or a curved road; step 1 of step 305 further includes:

[0168] The third step is to determine, for each curve in the i-th scene image, the third prediction information of the curve based on the curve vector of the curve in the element vector information and the posture information of the image acquisition device at the i-th moment; wherein the curve vector represents the projection position of two points on the curve in the i-th preset coordinate system; and the third prediction information includes the predicted projection distance of the two points on the curve in the i-th bird's-eye view.

[0169] Step 4: Determine the lateral residual of the curve in the lateral residual according to the third prediction information of the curve and the position information of two points on the curve in the i-th image coordinate system; wherein the lateral residual of the curve represents the curvature error of the curve projection in the i-th bird's-eye view.

[0170] Specifically, for each straight road in the i-th scene image, the projection positions of the two points on each curve in the i-th preset coordinate system are extracted according to the element vector information, that is, the curve vector of each curve, and the curve vector of each curve and the posture information of the image acquisition device at the i-th moment are calculated and processed to obtain the predicted projection distance of the two points of each curve in the i-th bird's-eye view, that is, the third prediction information of each curve. Based on the preset residual calculation method, the third prediction information of each curve and the position information of the two points on each curve in the i-th image coordinate system are calculated to obtain the curve lateral residual in the lateral residual to characterize the curve projection curvature error of each curve in the i-th bird's-eye view.

[0171] For example, for a curve, the difference between the actual distance and the projected distance between two points on the lane line can be used to measure the curvature error of the curve. Assume that the positions of the two points on the lane line are L i and L j , then the lateral residual

[0172] In one example, step 4 of step 305 includes:

[0173] The first step is to determine the velocity plane residual in the velocity residual model based on the position information and observed speed of the image acquisition device at the i-th moment and the position information of the image acquisition device at the i+1-th moment; wherein the velocity plane residual represents the movement speed of the image acquisition device between the i-th moment and the i+1-th moment.

[0174] The second step is to determine the velocity height residual in the velocity residual model according to the installation height of the image acquisition device at the i-th moment and the installation height of the image acquisition device at the i+1-th moment; wherein the velocity height residual represents the motion error of the image acquisition device in the height direction between the i-th moment and the i+1-th moment.

[0175] Specifically, when calculating the velocity residual model, based on the preset velocity residual model generation method, the position information and observation speed of the image acquisition device at the i-th moment and the position information of the image acquisition device at the i+1-th moment are calculated to obtain the velocity plane residual in the velocity residual model to characterize the movement speed of the image acquisition device between the i-th moment and the i+1-th moment; based on the preset velocity residual model generation method, the installation height of the image acquisition device at the i-th moment and the installation height of the image acquisition device at the i+1-th moment are calculated to obtain the velocity height residual in the velocity residual model to characterize the movement error of the image acquisition device in the height direction between the i-th moment and the i+1-th moment.

[0176] For example, in a scene without semantic elements, the system optimizes the pose by the camera’s movement speed. The speed residual model is divided into plane residual and height residual. The plane residual is used to measure the camera’s movement speed between two frames. Assume that the camera’s positions in the two frames are T i and T j , the observation speed of the camera is v, then the plane residual Among them, ∥T i -T j ∥ represents the actual translation of the camera between the two frames, and v·dt is the theoretical translation distance of the camera. The height residual is used to measure the vertical motion error of the camera. Assuming that the height difference of the camera in the two frames is dT[2], the height residual model dT[2] is the change in the camera's height in the vertical direction.

[0177] 306. Construct a back-projection residual model based on the residual model of each road element.

[0178] For example, according to the residual model of each road element, the residual model of all road elements at each moment can be processed by fusion or accumulation to obtain the back-projection residual model. Multiple residual models make the system more robust in complex environments such as lighting changes, fast movement, and bad weather.

[0179] For example, combining Figure 4 Based on the graph optimization modeling method, the graph optimization goal is to minimize the error of each factor on the entire factor graph. Let G = (V, E) represent a factor graph, V = {v i} is a node in the graph, {R i ,T i} represents the camera pose and semantic element position P j ; E = {e ij} is an edge in the graph, indicating the co-viewing relationship of semantic elements in different frames or the motion association of cameras; e ijrepresents all the edges in the graph, i.e., the observed semantic element j in the i-th scene image or the velocity residual between adjacent frames, p ij is the weight of each edge, which is used to adjust the influence of different types of residuals. In the factor graph, each edge between nodes represents a factor, and the nodes in the factor graph are constrained by the residual function. ij represents the error between the observed value and the predicted value of node j in frame i. The goal of the factor graph model is to optimize each node position and camera pose to minimize the error of the entire factor graph, which can be expressed as

[0180]

[0181] 307. Minimize the value of the back-projection residual model to obtain optimized element vector information and optimized posture information; wherein the optimized posture information includes the posture information of the image acquisition device at each moment after optimization.

[0182] For example, in combination Figure 4 The device minimizes the value of the back-projection residual model based on minimizing the error, and continuously optimizes the element vector information and the posture information of the image acquisition device at each moment until the value of the back-projection residual model reaches the minimum value, thereby obtaining the optimized element vector information and the optimized posture information of the image acquisition device at each moment, that is, the optimized posture information.

[0183] 308. According to the optimized posture information, the optimized element vector information is projected to generate a scene map.

[0184] Exemplarily, the device projects the optimized element vector information based on the optimized posture information and in combination with the minimum back-projection residual model, and projects it onto all scene images, i.e., the common view frame, where each corresponding road element appears, to obtain high-precision vector information of each road element. Then, based on mapping technology, the vector information of each road element at each moment is mapped to generate a scene map of the scene in which the vehicle is located in the current time period.

[0185] For example, combining Figure 4 For traffic signs, the vector is projected to all valid frames of the tracking sequence; for poles, the vector is projected to all valid frames of the tracking sequence; for lane lines, the single frame vector is projected forward 3 frames. Through the back-projection residual model of each element, the vector is projected to the corresponding common view frame to optimize the pose and semantic vector of each frame. Fig.10 A schematic diagram of the vector optimization effect based on the urban canyon scene provided in an embodiment of the present application. Fig.11 A schematic diagram of the vector optimization effect based on a common road scenario provided in an embodiment of the present application. Fig.12 A schematic diagram of the vector optimization effect based on a high-speed scenario provided in an embodiment of the present application. Fig.13 A schematic diagram of the vector optimization effect based on the gantry scene provided in the embodiment of the present application, combined with Fig.10 , 11 As shown in Figures 12 and 13, in the urban canyon scene, the GPS signal is poor and the accuracy of the combined navigation trajectory is very poor, and the comparison effect before and after optimization is obvious; in the ordinary road scene, the combined navigation posture is abnormal, and the effect is significantly improved after optimization; in the high-speed scene, the track deviates to the opposite lane before optimization, and the relative accuracy of the pole arrangement is poor, and the relative accuracy is significantly improved after optimization; in the gantry scene, the error is large before optimization, and the relative relationship is corrected after optimization.

[0186] In this embodiment, based on the above embodiments, on the one hand, by introducing semantic elements and graph optimization modeling, the cumulative error is reduced and the positioning accuracy in complex outdoor scenes is significantly improved; on the other hand, low-cost sensor equipment is used to enable the system to run on low-computing power devices and adapt to a wider range of application scenarios; in addition, a variety of residual models enable the system to have higher robustness in complex environments such as lighting changes, rapid movement, and severe weather.

[0187] Fig.14 A schematic diagram of the structure of a map generating device provided in an embodiment of the present application is shown in FIG. Fig.14 As shown, the device comprises:

[0188] The determination module 401 is used to obtain the scene images of the scene where the vehicle is located captured by the image acquisition device at each moment in the current time period; and determine the element vector information corresponding to each scene image; wherein the element vector information includes the projection position of each road element in all scene images in a preset coordinate system; the preset coordinate system is a three-dimensional coordinate system;

[0189] A construction module 402 is used to construct a back-projection residual model according to the element vector information; wherein the back-projection residual model represents the projection error of all road elements from the preset coordinate system to the image coordinate system;

[0190] The generation module 403 is used to adjust the element vector information based on the back-projection residual model to generate a scene map of the scene where the vehicle is located in the current time period.

[0191] In one possible implementation, the determination module 401 is specifically used to: perform target detection processing on each scene image to obtain a detection result for each road element in each scene image; wherein the detection result includes position information of each road element in an image coordinate system corresponding to each scene image; and determine element vector information based on the detection result of each road element.

[0192] In a possible implementation, the determination module 401 is further specifically used to: determine the lateral posture information of the image acquisition device at each moment according to the detection results of each road element; determine the longitudinal posture information of the image acquisition device at each moment according to the detection results of the road element and the lateral posture information of the image acquisition device at each moment; wherein the lateral posture information and the longitudinal posture information constitute the posture information of the image acquisition device;

[0193] The construction module 402 is specifically used to construct a back-projection residual model according to the element vector information and the position and posture information of the image acquisition device at each moment.

[0194] In a possible implementation, the determination module 401 is specifically used to: for each linear element in the i-th scene image, perform straight line fitting processing on the linear element according to the position information of the linear element in the i-th image coordinate system to obtain a two-dimensional broken line of the linear element in the i-th image coordinate system; perform back projection processing on the two-dimensional broken line according to the intrinsic parameters of the image acquisition device at the i-th moment to obtain a projection broken line of the two-dimensional broken line in the i-th preset coordinate system; determine the linear element vector of the linear element in the element vector information according to the projection broken line of the two-dimensional broken line in the i-th preset coordinate system; wherein the linear element vector represents the projection position of the linear element in the i-th preset coordinate system; determine the lateral posture information of the image acquisition device at the i-th moment according to the first initial vector of the linear element and the linear element vector of the linear element; wherein the first initial vector represents the initial position of the linear element in the i-th preset coordinate system; the lateral posture information includes rotation parameters and translation parameters.

[0195] In a possible implementation, each road element also includes at least one aerial element; the determination module 401 is further specifically used to: for each aerial element in the i-th scene image, perform back-projection processing on each target point according to the position information of each target point in the aerial element and the parameter information of the image acquisition device at the i-th moment, so as to obtain the projection ray of each target point in the i-th preset coordinate system; wherein the position information represents the position of the target point in the i-th image coordinate system; the parameter information includes the intrinsic parameters, rotation parameters and translation parameters of the image acquisition device; determine the coordinate information of the intersection of each projection ray in the i-th preset coordinate system, which is the aerial element vector of the aerial element in the element vector information; wherein the aerial element vector represents the projection position of the aerial element in the i-th preset coordinate system.

[0196] In a possible implementation, each road element also includes at least one sign element; the determination module 401 is further specifically used to: for each sign element in the i-th scene image, project the sign element according to the second initial vector of the sign element to obtain projection information of the sign element; wherein the second initial vector represents the initial position of the sign element in the i-th preset coordinate system; the projection information represents the projection shape of the sign element in the i-th image coordinate system; according to the projection information of the sign element and the detection result of each other sign element except the sign element in the i-th scene image, determine each weight corresponding to the sign element, wherein the weight represents the degree of matching between the projection information corresponding to the sign element and the detection result of each other sign element; determine the detection result of the other sign element corresponding to the highest weight among the weights as the target detection result corresponding to the sign element; determine the longitudinal posture information of the image acquisition device at the i-th moment according to the position information in the target detection result corresponding to the sign element, the projection information of the sign element, and the lateral posture information of the image acquisition device at the i-th moment.

[0197] In one possible implementation, the construction module 402 is specifically used to: perform target detection processing on each scene image to obtain a detection result for each road element in each scene image; wherein the detection result includes the position information of each road element in the image coordinate system corresponding to each scene image; determine the residual model of each road element based on the element vector information, the posture information of the image acquisition device at each moment and the detection result of each road element; wherein the residual model represents the projection error of the road element from the image coordinate system to the preset coordinate system; and construct a back-projection residual model based on the residual model of each road element.

[0198] In a possible implementation, each road element includes at least one sign element and at least one aerial element; the construction module 402 is specifically used to: for each road element in the i-th scene image, determine the first prediction information of the road element according to the posture information and element vector information of the image acquisition device at the i-th moment; wherein the first prediction information represents the predicted projection position of the road element in the i-th image coordinate system; determine the residual model of the road element at the i-th moment according to the first prediction information of the road element and the position information of the road element in the i-th image coordinate system.

[0199] In a possible implementation, each road element also includes at least one linear element; the construction module 402 is further specifically used to: for each linear element in the i-th scene image, determine the lateral residual in the residual model of the linear element according to the projection position of the linear element in the i-preset coordinate system in the element vector information and the position information of the linear element in the i-image coordinates; wherein the lateral residual represents the forward projection error of the linear element in the i-image coordinates; determine the longitudinal residual in the residual model of the linear element according to the position information of the linear element in the i-image coordinate system, the observation speed of the image acquisition device at the i-th moment and the time difference information; wherein the time difference information The information represents the time interval between two adjacent moments; the longitudinal residual represents the depth error of two points in the linear element; according to the position information of the linear element in the i-th image coordinate system and the installation height of the image acquisition device at the i-th moment, the height residual in the residual model of the linear element is determined; wherein, the height residual represents the projection error of the point in the linear element in the height direction in the i-th image coordinate system; according to the posture information of the image acquisition device at the i-th moment and the posture information of the image acquisition device at the i+1-th moment, the speed residual model in the residual model of the linear element is determined; wherein, the speed residual model represents the motion error of the image acquisition device between the i-th moment and the i+1-th moment.

[0200] In a possible implementation, the linear element is a straight road or a curve; the construction module 402 is further specifically used to: for each straight road in the i-th scene image, determine the second prediction information of the straight road according to the straight road vector of the straight road in the element vector information and the posture information of the image acquisition device at the i-th moment; wherein the straight road vector represents the projection position of the edge point of the straight road in the i-th preset coordinate system; the second prediction information includes the predicted projection position of the edge point of the straight road in the i-th bird's-eye view; and determine the straight road lateral residual in the lateral residual according to the second prediction information of the straight road and the position information of the straight road in the i-th image coordinate system; wherein the straight road lateral residual represents the projection error of the edge point of the straight road in the i-th bird's-eye view.

[0201] In a possible implementation, the construction module 402 is further specifically used to: for each curve in the i-th scene image, determine the third prediction information of the curve according to the curve vector of the curve in the element vector information and the posture information of the image acquisition device at the i-th moment; wherein the curve vector represents the projection position of two points on the curve in the i-th preset coordinate system; the third prediction information includes the predicted projection distance of the two points on the curve in the i-th bird's-eye view; determine the lateral residual of the curve in the lateral residual according to the third prediction information of the curve and the position information of the two points on the curve in the i-th image coordinate system; wherein the lateral residual of the curve represents the curvature error of the curve projection in the i-th bird's-eye view.

[0202] In a possible implementation, construction module 402 is further specifically used to: determine the velocity plane residual in the velocity residual model based on the position information and observed speed of the image acquisition device at the i-th moment and the position information of the image acquisition device at the i+1-th moment; wherein the velocity plane residual represents the movement speed of the image acquisition device between the i-th moment and the i+1-th moment; determine the velocity height residual in the velocity residual model based on the installation height of the image acquisition device at the i-th moment and the installation height of the image acquisition device at the i+1-th moment; wherein the velocity height residual represents the motion error of the image acquisition device in the height direction between the i-th moment and the i+1-th moment.

[0203] In one possible implementation, the generation module 403 is specifically used to: minimize the value of the back-projection residual model to obtain optimized element vector information and optimized pose information; wherein the optimized pose information includes the pose information of the image acquisition device at each moment after optimization; and project the optimized element vector information according to the optimized pose information to generate a scene map.

[0204] The device of this embodiment can execute the technical solution in the above method. Its specific implementation process and technical principles are the same and will not be repeated here.

[0205] Fig.15 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Fig.15 As shown, the electronic device includes: a memory 501 and a processor 502; the memory 501 is a memory for storing instructions executable by the processor 502.

[0206] The processor 502 is configured to execute the method provided in the above embodiment.

[0207] The electronic device further includes a receiver 503 and a transmitter 504. The receiver 503 is used to receive instructions and data sent by other devices, and the transmitter 504 is used to send instructions and data to external devices.

[0208] The specific implementation process of the processor can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0209] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0210] An embodiment of the present application also provides a chip for executing instructions, which is used to execute the technical solution of the processing method in the above embodiment.

[0211] The embodiment of the present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed on a computer, the computer executes the technical solution of the processing method of the above embodiment.

[0212] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory, electrically erasable programmable read-only memory, erasable programmable read-only memory, programmable read-only memory, read-only memory, magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0213] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in a dedicated integrated circuit. Of course, the processor and the readable storage medium can also exist in a device as discrete components.

[0214] An embodiment of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When at least one processor executes the computer program, the technical solution of the processing method in the above embodiment can be implemented.

[0215] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0216] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0217] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0218] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0219] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A map generation method, characterized in that: include: Obtaining a scene image of the scene in which the vehicle is located, which is captured by the image acquisition device at each moment in the current time period; and determining the element vector information corresponding to each of the scene images; wherein the element vector information includes the projection position of each road element in all the scene images in a preset coordinate system; the preset coordinate system is a three-dimensional coordinate system; According to the element vector information, a back-projection residual model is constructed; wherein the back-projection residual model represents the projection error of all road elements from the preset coordinate system to the image coordinate system; The element vector information is adjusted based on the back-projection residual model to generate a scene map of the scene where the vehicle is located in the current time period.

2. The method according to claim 1, characterized in that Determining element vector information corresponding to each of the scene images includes: Performing target detection processing on each of the scene images to obtain a detection result of each road element in each of the scene images; wherein the detection result includes position information of each of the road elements in an image coordinate system corresponding to each of the scene images; The element vector information is determined according to the detection result of each of the road elements.

3. The method according to claim 2, characterized in that The method further comprises: Determine the lateral position information of the image acquisition device at each moment according to the detection results of each of the road elements; Determine the longitudinal posture information of the image acquisition device at each moment according to the detection result of the road element and the lateral posture information of the image acquisition device at each moment; wherein the lateral posture information and the longitudinal posture information constitute the posture information of the image acquisition device; Constructing a back-projection residual model according to the element vector information, including: constructing the back-projection residual model according to the element vector information and the posture information of the image acquisition device at each moment.

4. The method according to claim 3, characterized in that Each of the road elements includes at least one linear element; and according to the detection results of each of the road elements, determining the lateral position information of the image acquisition device at each moment includes: For each linear element in the i-th scene image, according to the intrinsic parameters of the image acquisition device at the i-th moment, a back-projection process is performed on the two-dimensional polyline of the linear element in the i-th image coordinate system to obtain a projection polyline of the two-dimensional polyline in the i-th preset coordinate system; Determine the linear element vector of the linear element in the element vector information according to the projection polyline of the two-dimensional polyline in the i-th preset coordinate system; wherein the linear element vector represents the projection position of the linear element in the i-th preset coordinate system; According to the first initial vector of the linear element and the linear element vector of the linear element, the lateral posture information of the image acquisition device at the i-th moment is determined; wherein the first initial vector represents the initial position of the linear element in the i-th preset coordinate system; and the lateral posture information includes rotation parameters and translation parameters.

5. The method according to claim 3, characterized in that: Each of the road elements also includes at least one sign element; determining the longitudinal posture information of the image acquisition device at each moment according to the detection result of the road element and the lateral posture information of the image acquisition device at each moment, including: For each sign element in the i-th scene image, projection processing is performed on the sign element according to the second initial vector of the sign element to obtain projection information of the sign element; wherein the second initial vector represents the initial position of the sign element in the i-th preset coordinate system; and the projection information represents the projection shape of the sign element in the i-th image coordinate system; Determine each weight corresponding to the sign element according to the projection information of the sign element and the detection result of each other sign element except the sign element in the i-th scene image, wherein the weight represents the matching degree between the projection information corresponding to the sign element and the detection result of each other sign element; Determine the detection result of other sign elements corresponding to the highest weight among the weights, as the target detection result corresponding to the sign element; The longitudinal posture information of the image acquisition device at the i-th moment is determined according to the position information in the target detection result corresponding to the sign element, the projection information of the sign element, and the lateral posture information of the image acquisition device at the i-th moment.

6. The method according to claim 3, characterized in that According to the element vector information and the position information of the image acquisition device at each moment, constructing the back-projection residual model, including: Performing target detection processing on each of the scene images to obtain a detection result of each road element in each of the scene images; wherein the detection result includes position information of each of the road elements in an image coordinate system corresponding to each of the scene images; Determine the residual model of each road element according to the element vector information, the position information of the image acquisition device at each moment and the detection result of each road element; wherein the residual model represents the projection error of the road element from the image coordinate system to the preset coordinate system; The back-projection residual model is constructed according to the residual model of each road element.

7. The method according to claim 6, characterized in that Each of the road elements includes at least one sign element and at least one aerial element; determining the residual model of each of the road elements according to the element vector information, the position information of the image acquisition device at each moment and the detection result of each of the road elements, including: For each road element in the i-th scene image, determine first prediction information of the road element according to the posture information of the image acquisition device and the element vector information at the i-th moment; wherein the first prediction information represents the predicted projection position of the road element in the i-th image coordinate system; A residual model of the road element at the i-th moment is determined according to the first prediction information of the road element and the position information of the road element in the i-th image coordinate system.

8. The method according to claim 6, characterized in that Each of the road elements further includes at least one linear element; determining a residual model of each of the road elements according to the element vector information, the position information of the image acquisition device at each moment and the detection result of each of the road elements, including: For each linear element in the i-th scene image, determine the lateral residual in the residual model of the linear element according to the projection position of the linear element in the i-th preset coordinate system and the position information of the linear element in the i-th image coordinate in the element vector information; wherein the lateral residual represents the forward projection error of the linear element in the i-th image coordinate; Determine the longitudinal residual in the residual model of the linear element according to the position information of the linear element in the i-th image coordinate system, the observation speed of the image acquisition device at the i-th moment, and the time difference information; wherein the time difference information represents the time interval between two adjacent moments; and the longitudinal residual represents the depth error between two points in the linear element; Determine the height residual in the residual model of the linear element according to the position information of the linear element in the i-th image coordinate system and the installation height of the image acquisition device at the i-th moment; wherein the height residual represents the projection error of the point in the linear element in the height direction in the i-th image coordinate system; According to the posture information of the image acquisition device at the i-th moment and the posture information of the image acquisition device at the i+1-th moment, the speed residual model in the residual model of the linear element is determined; wherein the speed residual model represents the motion error of the image acquisition device between the i-th moment and the i+1-th moment.

9. A map generating device, characterized in that: include: A determination module, used to obtain a scene image of the scene where the vehicle is located, which is captured by the image acquisition device at each moment in the current time period; and determining the element vector information corresponding to each of the scene images; wherein the element vector information includes the projection position of each road element in all the scene images in a preset coordinate system; the preset coordinate system is a three-dimensional coordinate system; A construction module, used to construct a back-projection residual model according to the element vector information; wherein the back-projection residual model represents the projection error of all road elements from the preset coordinate system to the image coordinate system; A generation module is used to adjust the element vector information based on the back-projection residual model to generate a scene map of the scene where the vehicle is located in a current time period.

10. An electronic device / computer-readable storage medium / computer program product, characterized in that: The electronic device comprises: a memory, a processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, so that the processor performs the method as described in any one of claims 1-8; and / or, the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to implement the method as described in any one of claims 1-8 when executed by the processor; and / or, the computer program product comprises a computer program, and the computer program implements the method as described in any one of claims 1-8 when executed by the processor.

Images