Pose optimization method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a pose optimization method and device, electronic equipment and storage medium. The pose optimization method comprises: obtaining a first image corresponding to a scene in which a first object is currently located and a first pose of the first object; obtaining a detected vector feature of the first image; obtaining a projected vector feature in a neighborhood range of the first pose in a vector map; matching the detected vector feature with the projected vector feature to obtain a matching result; minimizing a re-projection error determined based on the matching result to optimize the first pose and obtain a second pose of the first object. The present disclosure can effectively improve the accuracy of pose optimization.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer vision, and particularly relates to a pose optimization method and device, an electronic device, and a storage medium. BACKGROUND

[0002] Visual positioning technology is a popular research direction in the field of automatic driving and intelligent robots. Mainstream visual positioning methods are generally divided into two categories: global repositioning technology and pose optimization method based on sensor estimation of pose. Among them, the sensor pose optimization method combined with high-precision maps is one of the current research hotspots.

[0003] How to obtain stable high-precision vector features from images containing illumination changes and scene changes, how to realize data association between the current scene and the high-precision map, and how to perform pose optimization combined with high-precision map features are all problems that need to be solved in the visual positioning method based on high-precision maps. However, the above problems have not been effectively solved at present. SUMMARY

[0004] In order to solve at least one of the above technical problems, the present disclosure provides a pose optimization method, device, electronic device, and storage medium.

[0005] A first aspect of the present disclosure provides a pose optimization method, comprising:

[0006] obtaining a first image corresponding to a current scene in which a first object is located and a first pose of the first object;

[0007] obtaining a detection vector feature of the first image;

[0008] obtaining a projection vector feature within a neighborhood range of the first pose in a vector map;

[0009] matching the detection vector feature with the projection vector feature to obtain a matching result;

[0010] minimizing a re-projection error determined based on the matching result to optimize the first pose and obtain a second pose of the first object.

[0011] In some embodiments of the present disclosure, the detection vector feature of the first image includes a vector descriptor of each semantic object in the first image, and the vector descriptor is used to describe a semantic label, a shape point position, a center point position, a neighborhood feature position vector, and a direction vector of the semantic object.

[0012] In some embodiments of the present disclosure, the obtaining of the detection vector feature of the first image comprises:

[0013] perform target detection and multi-frame fusion on the first image of the continuous k frames to obtain features of each semantic object in the first image, the features of the semantic object including at least a semantic label of the semantic object, k being an integer greater than 1;

[0014] vectorize the features of each semantic object according to the semantic label of the semantic object to obtain a vector descriptor of each semantic object, thereby forming a detection vector feature of the first image.

[0015] In some embodiments of the present disclosure, the projection vector feature includes a projection vector descriptor of each reference object in a neighborhood range of the first pose in the vector map in an image coordinate system of the first image, the projection vector descriptor being used to describe a semantic label, a shape point position, a center point position, a neighborhood feature position vector and a direction vector of the reference object in the image coordinate system of the first image.

[0016] In some embodiments of the present disclosure, the obtaining of the projection vector feature in the neighborhood range of the first pose in the vector map comprises:

[0017] vectorizing the vector feature of each reference object in the neighborhood range of the first pose from the vector map;

[0018] projecting the vector feature of each reference object to an image coordinate system of the first image to obtain a projection vector feature of each reference object;

[0019] vectorizing the projection vector feature of each reference object to obtain a projection vector descriptor of each reference object.

[0020] In some embodiments of the present disclosure, the matching of the detection vector feature with the projection vector feature to obtain a matching result comprises:

[0021] filtering the projection vector feature through similarity between the detection vector feature and the projection vector feature, so that the projection vector feature and the detection vector feature have approximate consistency in vector feature space distribution; and,

[0022] performing graph matching on the detection vector feature and the filtered projection vector feature to obtain the matching result.

[0023] In some embodiments of the present disclosure, the filtering of the projection vector feature through similarity between the detection vector feature and the projection vector feature comprises:

[0024] calculating similarity in absolute distance between each semantic object in the detection vector feature and each reference object in the projection vector feature;

[0025] discard the reference object whose similarity is less than or equal to a predetermined similarity threshold, and retain the reference object whose similarity is greater than the similarity threshold, to obtain a filtered projection vector feature.

[0026] In some embodiments of the present disclosure, the matching of the detection vector feature with the filtered projection vector feature to obtain the matching result comprises:

[0027] matching the vector descriptor of each semantic object in the detection vector feature with the projection vector descriptor of each reference object in the filtered projection vector feature to obtain a matching matrix and a confidence matrix, wherein an element in the matching matrix is used to represent a matching relationship between each semantic object and each reference object, and an element in the confidence matrix is used to represent a matching confidence between each semantic object and each reference object.

[0028] In some embodiments of the present disclosure, the re-projection error is determined according to a semantic label of each semantic object in the detection vector feature, and a shape point re-projection error, an end point re-projection error and / or a direction vector re-projection error between the semantic object and each reference object in the filtered projection vector feature, wherein the shape point re-projection error is obtained according to a shape corner point position of the semantic object, a shape corner point position of the reference object, a matching relationship and a matching confidence between the semantic object and the reference object; and / or, the end point re-projection error is obtained according to an end point position of the semantic object, an end point position of the reference object, a matching relationship and a matching confidence between the semantic object and the reference object; and / or, the direction vector re-projection error is obtained according to a direction vector of the semantic object, a direction vector of the reference object, a matching relationship and a matching confidence between the semantic object and the reference object.

[0029] In some embodiments of the present disclosure, the re-projection error is represented by a predetermined kernel function of a logarithmic function.

[0030] In some embodiments of the present disclosure, a bundle adjustment method is used to minimize the re-projection error to optimize the first pose and obtain the second pose.

[0031] The second aspect of the present disclosure provides a pose optimization device, comprising:

[0032] an acquisition module configured to acquire a first image corresponding to a scene in which a first object is currently located and a first pose of the first object;

[0033] a detection module configured to acquire a detection vector feature of the first image;

[0034] a projection module, configured to acquire a projection vector feature in a neighborhood range of the first pose in the vector map;

[0035] a matching module, configured to match the detected vector feature and the projection vector feature to obtain a matching result;

[0036] an optimization module, configured to minimize a re-projection error determined based on the matching result to optimize the first pose, and obtain a second pose of the first object.

[0037] In some embodiments of the present disclosure, the detected vector feature of the first image comprises a vector descriptor of each semantic object in the first image, and the vector descriptor is used to describe a semantic label, a shape point position, a center point position, a neighborhood feature position vector and a direction vector of the semantic object.

[0038] In some embodiments of the present disclosure, the projection vector feature comprises a projection vector descriptor of each reference object in a neighborhood range of the first pose in the vector map in an image coordinate system of the first image, and the projection vector descriptor is used to describe a semantic label, a shape point position, a center point position, a neighborhood feature position vector and a direction vector of the reference object in the image coordinate system of the first image.

[0039] In some embodiments of the present disclosure, the matching module is specifically configured to: filter the projection vector feature by similarity between the detected vector feature and the projection vector feature, so that the projection vector feature and the detected vector feature have approximate consistency in vector feature space distribution; and perform graph matching on the detected vector feature and the filtered projection vector feature to obtain the matching result.

[0040] In some embodiments of the present disclosure, the matching module is specifically configured to: perform graph matching on the vector descriptor of each semantic object in the detected vector feature and the projection vector descriptor of each reference object in the filtered projection vector feature to obtain a matching matrix and a confidence matrix, wherein an element in the matching matrix is used to represent a matching relationship between each semantic object and each reference object, and an element in the confidence matrix is used to represent a matching confidence between each semantic object and each reference object.

[0041] In some embodiments of the present disclosure, the reprojection error is determined according to a semantic label of each semantic object in the detection vector feature, and a shape point reprojection error, an end point reprojection error and / or a direction vector reprojection error between the semantic object and each reference object in the filtered projection vector feature; wherein the shape point reprojection error is obtained according to a shape corner point position of the semantic object, a shape corner point position of the reference object, a matching relationship and a matching confidence between the semantic object and the reference object; and / or the end point reprojection error is obtained according to an end point position of the semantic object, an end point position of the reference object, a matching relationship and a matching confidence between the semantic object and the reference object; and / or the direction vector reprojection error is obtained according to a direction vector of the semantic object, a direction vector of the reference object, a matching relationship and a matching confidence between the semantic object and the reference object.

[0042] In some embodiments of the present disclosure, the reprojection error is represented by a predetermined kernel function of a logarithmic function.

[0043] In some embodiments of the present disclosure, the optimization module is specifically configured to minimize the reprojection error by using a bundle adjustment method to optimize the first pose and obtain the second pose.

[0044] A third aspect of the present disclosure provides an electronic device, comprising:

[0045] a memory, the memory storing execution instructions; and

[0046] a processor, the processor executing the execution instructions stored in the memory, so that the processor executes the above-mentioned pose optimization method.

[0047] A fourth aspect of the present disclosure provides a readable storage medium, the readable storage medium storing execution instructions, the execution instructions being executed by a processor to implement the above-mentioned pose optimization method.

[0048] The present disclosure has the following beneficial effects:

[0049] 1) Combining the absolute distance, relative distance, direction, semantics and neighborhood feature space distribution information between the semantic vector features, the uniqueness of the vector features is measured from multiple dimensions, and the robustness and accuracy of the vector feature matching are improved.

[0050] 2) Utilize the structural similarity between semantic vector features, including the absolute distance of features, relative distance, direction, semantics, neighborhood distribution information, realize the matching of semantic vector features. Compared with the descriptor information, the structured information of the feature is more robust, has a certain anti-interference ability to the environment, and can improve the accuracy of feature matching. At the same time, the vectorization of semantic features can improve the robustness of semantic features, and reduce the influence of unstable edge detection of neural network on feature matching.

[0051] 3) Combined with multi-dimensional vector descriptor, realize the spatial global consistency matching of lightweight semantic vector features, improve the accuracy and robustness of data association in complex environment and long-time environment, solve the problem of data association failure or error in complex scene and long-time environment.

[0052] 4) In addition, the present disclosure also utilizes the re-projection error function optimization of the first pose based on the multi-granularity matching of vector features and the matching confidence, further improves the accuracy of pose optimization. BRIEF DESCRIPTION OF DRAWINGS

[0053] The accompanying drawings illustrate exemplary embodiments of the present disclosure and together with the description, explain the principles of the present disclosure, in which the drawings are included to provide further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0054] Figure 1 is a flowchart of a pose optimization method of an embodiment of the present disclosure.

[0055] Figure 2 is a flowchart of obtaining a detection vector feature of an embodiment of the present disclosure.

[0056] Figure 3 is a flowchart of obtaining a projection vector feature of an embodiment of the present disclosure.

[0057] Figure 4 is a flowchart of matching a detection vector feature and a projection vector feature of an embodiment of the present disclosure.

[0058] Figure 5 is a flowchart of minimizing a re-projection error of an embodiment of the present disclosure.

[0059] Figure 6 is a structural schematic block diagram of a pose optimization device of an embodiment of the present disclosure in a hardware implementation of a processing system.

[0060] Reference numerals

[0061] 600 pose optimization device

[0062] 602 acquisition module

[0063] 604 detection module

[0064] 606 projection module

[0065] 608 matching module

[0066] 610 optimization module

[0067] 700 bus

[0068] 800 processor

[0069] 900 memory

[0070] 1000 other circuitry. DETAILED DESCRIPTION

[0071] The present disclosure will be further described in conjunction with the drawings and embodiments. It can be understood that the specific embodiments described herein are merely exemplary and are not intended to limit the disclosure. In addition, it should be noted that only parts related to the present disclosure are shown in the drawings for the purpose of description.

[0072] It should be noted that the embodiments and features in the embodiments of the present disclosure can be combined with each other without conflict. The technical solutions of the present disclosure will be described in detail below with reference to the drawings and in conjunction with the embodiments.

[0073] Unless otherwise specified, the exemplary embodiments / instances shown will be understood to provide exemplary features of various details that can implement the technical concepts of the present disclosure in practice. Therefore, unless otherwise specified, the features of various embodiments / instances can be additionally combined, separated, interchanged and / or rearranged without departing from the technical concepts of the present disclosure.

[0074] In the drawings, cross-hatching and / or shading are generally used to make the boundaries and regions of adjacent components clear. Thus, unless otherwise specified, the presence or absence of cross-hatching or shading does not convey or imply any preference or requirement for a specific material, material property, size, scale, commonality of the illustrated components, and / or any other characteristic, attribute, property, etc. of the components. In addition, in the drawings, the size and relative sizes of components can be exaggerated for clarity and / or descriptive purposes. When exemplary embodiments can be implemented differently, a specific process sequence can be performed in a different order than described. For example, two consecutively described processes can be performed substantially simultaneously or in an order opposite to that described. In addition, the same reference numbers represent the same components.

[0075] When a component is referred to as being “on”, “above”, “connected to” or “coupled to” another component, it can be directly on, directly connected to, or directly coupled to the other component, or intervening components can be present. When a component is referred to as being “directly on”, “directly connected to”, or “directly coupled to” another component, there are no intervening components present. By the term “connected” is meant to include physical or electrical connections, with or without intervening components.

[0076] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. Furthermore, to the extent that the terms “including”, “includes”, “having”, “has”, “a”, “an”, “one” or “said” are used in this specification and / or claims, they are intended to be inclusive (i.e. they mean “there is one or more”) and not to be exclusive (i.e. they do not mean “there is only one”). Additionally, it is also noted that the terms “generally”, “approximately” and other like terms are used herein as approximation terms and not as degree terms, such that any measurement, calculation, and / or provision values they describe will inherently have a certain amount of inaccuracy.

[0077] The related art will be briefly analyzed and described below.

[0078] Related Art 1: Chinese Patent Publication No. CN111982133 discloses a method and device for positioning a vehicle based on a high-precision map and electronic equipment, which involves a pose optimization method based on a high-precision map. The method predicts the semantic instance category and edge of the collected image sequence through a pre-trained deep learning model, then constructs a local high-precision map of the semantic elements of interest in the current scene based on the estimated pose of the current sensor, and matches it with the corresponding cloud high-precision map, and then calculates the positioning pose of the current vehicle in the cloud high-precision map using the Singular Value Decomposition (SVD) method for the matched semantic element boundary point set. The data association is performed based on the local search matching strategy of high-level semantic information, although the vector information of the high-precision map is combined, the matching method based on the lane line sampling points in the high-precision map is essentially a feature point-based matching, and the degree of vectorization of the features is low, and there is much noise in point matching. In addition, the semantic edge detection result based on deep learning is unstable, and the matching based on the edge point set will be affected by the edge detection result. Therefore, this scheme has the problems of low precision, poor accuracy of optimization results, and poor robustness.

[0079] Related Technology 2: Chinese Patent CN112085092A discloses a graph matching method and device based on spatiotemporal continuity constraint, which proposes a feature graph matching method based on SURF (Speeded Up Robust Features) descriptor. The scheme takes the spatiotemporal continuous video sequence as input, obtains the initial matching feature points through the homographic transformation relationship of interframe images, and then establishes the association graph with the initial matching relationship to convert the matching optimization problem of feature points into the graph matching optimization problem for solving. The scheme uses the continuity of video sequence, uses homographic constraint to provide a good initial state for the optimization process of graph matching, and considers the similarity between feature points and the similarity of the included angle formed by the edge of the matching point pair based on the descriptor matching of the graph, thereby improving the speed and accuracy of graph matching. Although the efficiency of graph matching is improved by providing a good initial value, the feature points based on the descriptor are large and sensitive, which still greatly restricts the speed and accuracy of graph matching. Therefore, the scheme has the defects of low precision, poor accuracy of optimization results, low processing efficiency, poor real-time performance, etc.

[0080] Related Technology 3: The paper named "Monocular Localization with Vector HD Map (MLVHM): A Low-Cost Method for Commercial" proposes a pose optimization method combined with high-definition map: first, the semantic point features and line features in the current scene are obtained by deep learning method, and then the improved random sample consensus algorithm (RANSAC) is used to realize data association and pose optimization. The scheme performs pose optimization based on randomly matched point pairs, and the optimization depends on the selection of outlier threshold and optimization error threshold, and only relies on the same semantic to randomly generate matching subsets, so there are more error matches in the generated subsets, which will lead to long time-consuming in the iterative solving process. Therefore, the scheme has the problems of low precision, poor result accuracy and poor robustness.

[0081] In the pose optimization scheme based on the high-definition map, since the semantic vector features of the high-definition map do not have corresponding descriptor information, the data association between the semantic vector features cannot be established by the descriptor matching method. Therefore, establishing the data association between the current scene and the high-definition map is an important link in the pose optimization process. In this regard, the related technology 1 uses the nearest neighbor matching to use the distance information of the vector features to take the features with the closest positions as the matching points. The related technology 3 uses the semantic information of the vector features to perform local search matching and RANSAC matching. It can be seen that the related technology 1 and the related technology 3 do not fully utilize the attributes and constraints of the vector semantic map, but only perform vector feature matching and data association based on distance or semantic information, and the matching effect and accuracy depend on prior constraint conditions (for example, the robustness of the semantic features, the overlap rate of the matching subgraphs, etc.), so these two schemes have the problems of low accuracy and poor accuracy of the optimization results. In addition, the related technology 1 and the related technology 3 both use the deep learning method to describe and extract semantic features, and the neural network has weak edge description capability, which is easily affected by the environment in actual application, and the detection result is unstable. Therefore, the robustness of the semantic features based on deep learning of these two schemes also needs to be improved. The related technology 2 is a graph-based feature matching method, which aims to use the second-order similarity between features to improve the accuracy of feature matching, but the number of features based on the descriptor is large, and the matching speed is slow, which cannot meet the real-time matching requirement.

[0082] In view of this, the present disclosure provides a pose optimization method, device, electronic equipment and storage medium, which can solve the robust data association problem in the pose optimization scheme combined with the high-definition map across long scenes, and realize a multi-granularity matching pose optimization scheme based on semantic vector features, with high positioning accuracy and strong adaptability.

[0083] The following describes an example embodiment of the present disclosure. Figures 1 to 6 The following describes an example embodiment of the present disclosure.

[0084] Figure 1 A flowchart of a pose optimization method in some embodiments of the present disclosure is shown. Referring to FIG. 1, the pose optimization method S10 can include the following steps. Figure 1

[0085] In step S12, a first image corresponding to a current scene in which a first object is located and a first pose of the first object are obtained.

[0086] ​The first image can be captured by a sensor such as a camera. The first image contains the content of the scene where the first object is located, but can not contain the first object itself. Here, the first object can be an object such as a vehicle, a robot, etc. that needs to be positioned in real time. Taking a vehicle as an example, the first image can be a front view image captured by a front view camera of the vehicle, which can be but is not limited to a monocular camera, a color (RGB) camera, a time of flight (TOF) camera, etc.

[0087] It should be noted that the "vehicle" can be but is not limited to various types such as a drone, a logistics vehicle, a household vehicle, a passenger vehicle, a subway, a high-speed rail, an aircraft, a ship, etc., and can be a hybrid vehicle, an electric vehicle, a fuel vehicle, etc. In addition, the present disclosure can be applicable to various scenarios that need to position an object in real time, and can be applicable to various environments such as indoor, outdoor and / or wild environments, etc.

[0088] It should be noted that the robot can be but is not limited to various types such as a logistics robot, a patrol robot, a flying robot, an industrial robot, etc.

[0089] In some embodiments, the first pose can be an initial pose directly obtained by a sensor such as an odometer, a global positioning system (GPS), an inertial measurement unit (IMU), etc., or can be a pose obtained by fusing data of multiple sensors such as an odometer, a GPS, an inertial measurement unit, etc. In addition, the first pose can be a pose of the first object itself, or can be a pose of a sensor fixedly installed on the first object such as a vehicle.

[0090] In step S14, a detection vector feature of the first image is obtained.

[0091] In some embodiments, the detection vector feature of the first image includes a vector descriptor of each semantic object in the first image, and the vector descriptor is used to describe a semantic label, a shape point position, a center point position, a neighborhood feature position vector and a direction vector of the semantic object.

[0092] In some embodiments, step S14 can include step S142 and step S144:

[0093] In step S142, target detection and multi-frame fusion are performed on the first images of the continuous k frames to obtain a feature of each semantic object in the first image, and the feature of each semantic object includes a semantic label of the semantic object.

[0094] Taking a road scene as an example, the semantic objects can be, but are not limited to, trees, road stakes, guardrails, street lamps, traffic lights, power poles, gantries, traffic signs such as signboards, identification plates, road marking lines such as lane lines, guide indication lines, stop lines, pedestrian crossings, and the like, and the features of the semantic objects can include, but are not limited to, one or more of the position, shape, direction, neighborhood features, and the like of the semantic objects in addition to the semantic labels.

[0095] In some embodiments, the target detection of the first image can be implemented based on various models such as a pre-trained deep learning network, a target detection network, a semantic segmentation network, a multi-task network based on a video sequence, and the like.

[0096] In some embodiments, step S142 can specifically include steps a1 and a2 as follows:

[0097] Step a1: performing target detection on the continuous k frames of the first image of the current scene to obtain the detection results of the continuous k frames of the first image, which can be represented as I(I0, I1,..., Ik-1, Ik). k-1 I0 represents the target detection result of the 0th frame of the first image, I1 represents the target detection result of the 1st frame of the first image, and Ik-1 represents the target detection result of the k-1th frame of the first image. Here, k is an integer greater than 1. k-1

[0098] Step a2: fusing the detection results of the continuous k frames of the first image by using a deep tracking algorithm such as DeepSORT to obtain the detection features of the first image, which can include the features of each semantic object in the first image, and the detection features V o of the first image can be represented as n+1 represents the number of semantic objects obtained by tracking and detecting the continuous k frames of the first image, represents the feature of the 0th semantic object, represents the feature of the 1st semantic object, and represents the feature of the nth semantic object.

[0099] Step S144: performing vector description on the features of each semantic object according to the semantic labels of the semantic objects to obtain the vector descriptors of each semantic object, thereby forming the detection vector features of the first image, which can include the vector descriptors of all semantic objects obtained after the target detection of the continuous k frames of the first image and the multi-frame fusion.

[0100] The vector description of the features refers to the description of the features in the form of position, shape, neighborhood feature position, semantics, and the like. For any one semantic object in the detection features V o of the first image, ​ its vector descriptor Different semantic labels are described as different vectors according to semantics.

[0101] In some embodiments, for semantic objects of straight line type such as street lamps, poles, stop lines, etc. its vector descriptor is represented as a shape point position vector center point position neighbor feature position vector and semantic label wherein, is the u-th neighbor feature position in the neighborhood range with the origin and radius r1, wherein, denotes the center point position of the u-th neighbor feature of denotes the position of the starting point, denotes the position of the end point.

[0102] In some embodiments, for semantic objects of arrow type such as guide sign lines, etc. its vector descriptor is represented as a shape point position vector center point position neighbor feature position vector direction vector of the arrow and semantic label wherein, denotes the p-th corner point position of denotes the position of the p-th corner point of

[0103] In some embodiments, for semantic objects of other types such as signboards, traffic lights, etc. its vector descriptor is represented as a shape point position center point position neighbor feature position vector and semantic label

[0104] In other words, the vector descriptor of each semantic object can be obtained according to the following formula (1) to obtain the detection vector features of the first image:

[0105]

[0106] wherein, denotes the vector descriptor of the semantic object denotes the starting point position of the semantic object denotes the end point position of the semantic object denotes the semantic label of the semantic object an end position of the semantic object, a shape point position of the semantic object, a position of a p-th corner point of the semantic object, a center point position of the semantic object, a neighborhood feature position vector of the semantic object, an arrow direction vector of the semantic object, a semantic label of the semantic object. a center point position of a u-th neighborhood feature in a neighborhood range with the origin as the center and a radius of r1, an arrow direction vector of the semantic object, a semantic label of the semantic object.

[0107] In step S16, a projection vector feature in a neighborhood range of the first pose in the vector map is obtained.

[0108] In some embodiments, the projection vector feature includes a projection vector descriptor of each reference object in the neighborhood range of the first pose in the vector map in the image coordinate system of the first image, and the projection vector descriptor is used to describe a semantic label, a shape point position, a center point position, a neighborhood feature position vector, and a direction vector of the reference object in the image coordinate system of the first image.

[0109] In some embodiments, the vector map stores information such as spatial positions, shapes, directions, and semantic labels of reference objects in a scene. For example, in a road scene, the reference objects in the vector map can include but are not limited to trees, road posts, guardrails, street lamps, traffic lights, power poles, gantries, traffic signs such as signboards and / or identification plates, and traffic marking lines such as lane lines, guide marks, stop lines, and pedestrian crossings. In specific applications, the vector map can be but is not limited to a high-definition map, a lightweight vector semantic map constructed manually or automatically, and the like.

[0110] In some embodiments, step S16 can include steps S162-S166 as follows:

[0111] In step S162, vector features of each reference object in the neighborhood range of the first pose (i.e., the scene where the first object is currently located) are parsed from the vector map;

[0112] Specifically, according to the first pose, features of reference objects in the neighborhood range of the first pose in the vector map are searched to obtain a reference object feature set X h corresponding to the first image of the current k-frame, h which is represented as​​​​​​ m+1 represents the number of reference objects, represents the features of the 0th reference object, Represents the characteristics of the first reference object, Represents the characteristics of the second reference object, represents the characteristics of the i-th reference object, Represents the features of the mth reference object. The features of each reference object can include information such as the spatial position, shape, direction, and semantic label of the reference object.

[0113] Here, the neighborhood range can be pre-set based on different application scenarios and actual application requirements. For example, the neighborhood range can be set to a fixed-size circular or rectangular area centered on the first pose. Generally, the neighborhood range covers the current scene. For example, in the case of a vehicle, the current scene may be the vehicle's forward-looking scene, and the vehicle's neighborhood range may include the vehicle's forward-looking area.

[0114] Step S164 , projecting the features of each reference object into the image coordinate system of the first image to obtain a projection vector feature of each reference object;

[0115] Specifically, for the reference object feature set X h Any reference object in The reference object can be obtained by projecting the position features (i.e., shape point positions, center point positions) of the reference object into the image coordinate system of the first image according to the camera internal and external parameters corresponding to the first image. Projected object in the image coordinate system of the first image characteristics.

[0116] For example, any reference object Center point location on the high-precision map The above projection can be achieved by the following formula (2): The projection of the shape point position is similar to the center point projection.

[0117]

[0118] in, express The projection point position on the image, R j , t j Represents a reference object The pose in the world coordinate system, R ext , t extrepresents the camera extrinsic parameter matrix corresponding to the first image (i.e., the extrinsic parameter matrix of the sensor that captured the first image, which can be obtained through camera calibration), that is, the transformation matrix from the world coordinate system to the camera coordinate system. K represents the camera intrinsic parameters corresponding to the first image (i.e., the intrinsic parameters of the sensor that captured the first image).

[0119] Thus, the projection feature set V can be obtained h , expressed as represents the features of the projected object of the 0th reference object, Represents the characteristics of the projection object of the first reference object, Represents the features of the projected object of the mth reference object.

[0120] Step S166 : performing vector description on the projection vector features of each reference object to obtain a projection vector descriptor of each reference object.

[0121] Similar to step S144, the projection vector descriptor of each reference object can be obtained according to the following formula (3):

[0122]

[0123] in, Represents a reference object Projection object The projection vector descriptor of Represents a reference object Projection object The starting position of the reference object The starting point of the reprojection point is Represents a reference object Projection object The end position of the reference object The end point reprojection point position, Represents a reference object Projection object The shape point position vector of the reference object The shape point reprojection point position, express The position of the qth corner point, Represents a reference object Projection object The center point position of the reference object The center reprojection point position, Represents a reference object Projection object The direction vector of the reference object The reprojection direction vector, Represents a reference object Projection object The neighborhood feature position vector of , where express is the center point of the uth neighborhood feature within the neighborhood range of the origin and radius r1, Represents a reference object Projection object semantic tags.

[0124] Step S18, matching the detected vector features with the projected vector features to obtain a matching result;

[0125] In some implementations, step S18 may include step S182 and step S184:

[0126] Step S182, coarse-grained matching: filter the projected vector features by the similarity between the detection vector features and the projected vector features. Specifically, the center point of the detection vector features is the origin, and the radius is In the neighborhood of , the similarity between the detection vector features and the projection vector features in the neighborhood is calculated, and the redundant parts in the projection vector features are filtered out so that the projection vector features and the detection vector features have approximate consistency in the vector feature space distribution.

[0127] In some implementations, step S182 may include the following steps b1 and b2:

[0128] Step b1, calculate the detection vector feature V o The similarity between each semantic object in the r2 neighborhood and the reference object in absolute distance.

[0129] In some implementations, for any semantic object, The semantic object can be calculated according to the following formula (4): The projection object of the reference object within the r2 neighborhood Similarity

[0130]

[0131] in, Representing semantic objects in Euclidean space and reference objects Projection object The center point distance difference.

[0132] Step b2: filtering the projection vector features according to the similarity obtained in step b1 and a preset similarity threshold, i.e., discarding reference objects whose similarity is less than or equal to the preset similarity threshold, and retaining reference objects whose similarity is greater than the similarity threshold, to obtain filtered projection vector features.

[0133] Specifically, for any projection object If the semantic object corresponding to the projected object can be detected, then there must be a semantic object in the neighborhood range r2. The semantic object and projection objects are matching pairs, which are close in spatial location and may even overlap. That is, semantic objects and projection objects The similarity error between the two should be small. and projection objects If the similarity error value is large, it can be considered that the two do not match. Therefore, a similarity threshold can be set in advance to filter out the parts of the projected vector features that do not match the semantic objects. In this way, the reference objects corresponding to semantic objects that are misdetected or missed in the projected vector features due to factors such as occlusion and lighting can be discarded, so that the detection vector features and the projected vector features have similar spatial distribution consistency.

[0134] Specifically, the filtered projection vector features It can be expressed as the following formula (5):

[0135]

[0136] Where m′ represents the number of projected vector features after filtering, Representing semantic objects and the reference object within the neighborhood range r2 Projection object The similarity of τ represents the similarity threshold, and the filtered V h′ and V o There is approximate consistency in spatial distribution.

[0137] Step S184, fine-grained matching: using the vector descriptor of the vector feature, the detection vector feature V o and the filtered projection vector feature V h′ Graph matching is performed to obtain matching results. Thus, fine-grained feature matching can be performed by combining the structural information between vector features, namely distance, direction, semantics, neighborhood distribution information, etc., to obtain one-to-one matching pairs between vector features.

[0138] Specifically, graph matching is performed between the vector descriptors of each semantic object in the detection vector features and the projected vector descriptors of each reference object in the filtered projected vector features to obtain a matching matrix and a confidence matrix. The elements in the matching matrix represent the matching relationship between each semantic object and each reference object, while the elements in the confidence matrix represent the matching confidence between each semantic object and each reference object. Because the vector descriptors contain structural information such as the semantic label, direction, position, and neighborhood characteristics of the vector features, graph matching can obtain a one-to-one correspondence between the semantic objects in the detection vector features and the reference objects in the projected vector features.

[0139] In some implementations, step S184 may include the following steps c1 to c4:

[0140] Step c1, establish detection vector feature V o The directed graph G o and the projection vector feature V h′ The directed graph G h′ ;

[0141] Specifically, based on the detection vector feature V o Create the corresponding directed graph G o ={V o , E o , W o}, V o Representation graph G o Node information, that is, each semantic object is regarded as a graph G o A node of E o Representation graph G o The side information, E o Including Figure G o The edge information between any two nodes W o Representation graph G o Weight information, W o Including Figure G o The weight of the edge between any two nodes in Represents the closeness between two nodes (i.e., the first-order similarity of the graph).

[0142] for If the node and nodes If there is an edge between otherwise

[0143] for Its corresponding weight Representation node and The intimacy between them. It can be expressed as a multidimensional vector as shown in the following formula (6), where Represents nodes respectively and Semantic tags, Represents nodes respectively and The center point position, Represents nodes respectively and The neighborhood feature position vector of .

[0144]

[0145] According to the above principle, we can also use the projection vector feature V h′ Create the corresponding directed graph G h′ ={V h′ , E h′ , W h′}, V h′ Representation graph G h′ Node information, E h′ Representation graph G h′ The side information, W h′ Representation graph G h′ The weight information.

[0146] Step c2, according to the directed graph G o and the directed graph G h′ , establish the corresponding association graph G a ={V a , E a , W a};

[0147] Among them, V a Representation graph G a Node information, Represents a directed graph G o Node and the directed graph G h′ Node Together they form a node.

[0148] E a Representation graph G a The side information, For any and The edge between Representation graph G o edge and Figure G h′ edge The second-order edge connection relationship between them.

[0149] W a Representation graph G a The weight information, For any Corresponding weight Represents graph G a Node and nodes Intimacy (i.e., subgraph G o edge and subgraph G h′ edge similarity between them, that is, the second-order similarity of the graph).

[0150] Among them, intimacy It can be obtained by the following formulas (7) to (9):

[0151]

[0152]

[0153]

[0154] Among them, ⊙ represents the exclusive OR operation. When the values ​​on both sides of ⊙ are equal, it is 1, otherwise it is 0. Represents graph G a Node and nodes The closeness between them, that is, the subgraph G o edge and subgraph G h′ edge The similarity between ξ. ij：pq The neighborhood weight parameter representing the closeness, when the neighborhood distribution of the edge is not similar, ξ ij：pq is very small, when the neighborhood distribution of the edge is very close, ξ ij：pq Very large. w1 represents the weight parameter for the first-order similarity, and w2 represents the weight parameter for the second-order similarity. w1 and w2 can be obtained through offline learning. Where a, b∈{1,2,…,v}, p={0,1,2,…,m′}, q={0,1,2,…,m′}.

[0155] In formulas (7) to (9), when ij = pq, Describes the similarity between nodes (i.e., the first-order similarity of the graph); when ij≠pq, Describes the similarity between edges (i.e., the second-order similarity of the graph). As can be seen from equations (7) to (9), the present disclosure sets the first-order similarity of the graph to zero and integrates the first-order similarity measurement into the second-order similarity of the graph. At the same time, it combines the absolute distance, relative distance, direction, and neighborhood feature space distribution relationship to measure the second-order similarity of the graph.

[0156] Step c3, based on the association graph G a Weight information W a , construct a weighted intimacy matrix W nm‘×nm’ .W nm‘×nm’ The rows of G are represented as a Node Column representation graph G a Node The corresponding element is the weight calculated by formula (7) Next, graph G o and Figure G h′ The matching problem is transformed into W nm‘×nm’ The quadratic assignment problem (QAP) of the closeness matrix, the problem of finding the optimal assignment matrix under constraints can be expressed as Equations (10) to (11), with the assignment matrix Q∈{0,1} nm′×1 represents the solution to the problem, for q ij ∈Q, if q ij =1, indicating graph G o Node and Figure G h′ Node is a matching pair, that is, a semantic object and projection objects is a pair of matching objects. On the contrary, q ij =0.

[0157] Q = argmax(Q T W nm′×nm′ Q) (10)

[0158]

[0159] In step c4, the Reweighted Random Walks for Graph Matching (RRWM) algorithm is used to solve the problem. The problem of finding the optimal assignment matrix is ​​approximated as a continuous optimization problem. The optimization equation is shown in Equation (12).

[0160] Q′=argmax(Q′ T W nm×nm Q′), stQ′∈[0,1]nm×1 (12)

[0161] Thus, through multiple iterations, the elements in the assignment matrix Q' tend to a stable probability value, and then the stable assignment matrix Q' is discretely mapped based on the constraints in formulas (10)-(11) to obtain the discrete assignment matrix Q, which is the matching result of the graph G o and the graph G h′ . For the elements q' ij in the assignment matrix Q' can be used to describe the matching confidence of the node and the node .

[0162] Here, the matrix Q contains elements q o indicating the matching relationship between each node in the graph G h′ and each node in the graph G ij . If the i-th node in the graph G o matches the j-th node in the graph G h′ , q ij = 1, and if the i-th node in the graph G o does not match the j-th node in the graph G h′ , q ij = 0.

[0163] Without prior constraints, the assignment matrix Q' is initialized as a dense graph of uniform probabilities, and the disclosure combines a deep learning method to obtain semantic information of the nodes corresponding to Q', and initializes the assignment matrix as a sparse probability graph with the semantic information as a constraint, which improves the efficiency of the calculation and improves the accuracy of the matching.

[0164] In step S110, the re-projection error determined based on the matching result is minimized to optimize the first pose and obtain the second pose of the first object.

[0165] In some embodiments, the re-projection error can be determined according to the semantic label of each semantic object in the detected vector feature, and the shape point re-projection error, the end point re-projection error and / or the re-projection error of the direction vector between the semantic object and each reference object in the filtered projection vector feature.

[0166] In some embodiments, the shape point re-projection error can be obtained according to the shape corner point position of the semantic object, the shape corner point position of the reference object, the matching relationship and the matching confidence of the semantic object and the reference object.

[0167] In some embodiments, the end point re-projection error can be obtained according to the end point position of the semantic object, the end point position of the reference object, the matching relationship and the matching confidence of the semantic object and the reference object.

[0168] In some embodiments, the re-projection error of the direction vector can be obtained according to the direction vector of the semantic object, the direction vector of the reference object, the matching relationship of the semantic object and the reference object, and the matching confidence.

[0169] In some embodiments, the step S110 can include steps S112 and S114 as follows:

[0170] In step S112, according to the semantic label and the matching result (i.e., the matching matrix Q and the confidence matrix Q'), the re-projection error expression E(V o , V h ) of the detection vector feature and the projection vector feature in the image coordinate system of the first image is determined.

[0171] Let the detection vector feature be represented as a sequence Let the projection vector feature be represented as a sequence The matching result between the two includes the matching matrix Q and the corresponding confidence matrix Q' described above. In some embodiments, any semantic object V o in V and any projection object V h′ in V The shape corner point projection error e ij in the image coordinate system of the first image can be obtained by the following formula (13):

[0172]

[0173] wherein q ij represents the matching relationship of the semantic object V and the projection object V , q' ij represents the matching confidence of the semantic object V and the projection object V , and h(v, u) represents the corresponding relationship of the corner points of the matched semantic object V and the projection object V . If h(v, u) = 1, it means that V and V are corresponding corner points, and if h(v, u) = 0, it means that V and V are not corresponding corner points. The corresponding relationship of the corner points between V and V can be inferred by the size ordering of the corner point coordinates in the pixel coordinate system. represents the shape corner point projection error of the semantic object V and the projection object V .

[0174] In some embodiments, for semantic objects with semantic labels of straight line type or line segment type, such as street lamps, poles, stop lines, etc. Considering the re-projection error of the endpoints of a line segment, such as the start point and / or the end point

[0175] For example, the endpoint re-projection error of such semantic objects in the image coordinate system of the first image can be calculated according to the following formula (14)

[0176]

[0177] wherein, denotes the semantic object and the start point projection error of the projection object denotes the semantic object and the end point projection error of the projection object

[0178] In some embodiments, for semantic objects with semantic labels of arrow type, such as guide sign lines, etc. Not only the re-projection error e ij of the shape corner point is considered, but also the re-projection direction error of the arrow direction vector is considered as a variable of pose optimization.

[0179] For example, the re-projection error of the direction vector can be obtained by the following formula (15):

[0180]

[0181] wherein, denotes the angle between the direction vector of the semantic object and the reference object, and the smaller the error of the two vectors is, the smaller the θ is.

[0182] If the semantic object and the projection object do not match, the value of the corresponding re-projection error will be very large, therefore, in some embodiments, the re-projection error between the detection vector feature V o and the projection vector feature V h E(V o , V h ) can be represented by a predetermined kernel function of a logarithmic function, so as to suppress the maximum value of the error, reduce the noise interference, and improve the calculation efficiency.

[0183] In some embodiments, the re-projection error E(V o , V h ) between the detection vector feature V o and the projection vector feature V h can be represented by the following formula (16):

[0184]

[0185] wherein λ1, λ2 and λ3 represent the dependence of the current pose optimization on the three kinds of re-projection errors, i.e. shape corner re-projection error e ij , end point re-projection error re-projection error of the direction vector. The dependence degree of the re-projection error of the direction vector on the current pose can be obtained through offline learning.

[0186] In the case of missed detection or false detection, the spatial distribution of the detection vector feature V o and the projection vector feature V h is inconsistent, and therefore, in the process of ensuring global consistent matching as much as possible, some unequal matching pairs can be wrongly generated. In the embodiments of the present disclosure, a robust kernel function is introduced to reduce the interference and influence of the pose optimization caused by the wrong matching pairs.

[0187] In step S114, the minimum value of the re-projection error expression is solved by using, for example, a bundle adjustment method or other similar algorithms to optimize the first pose, so as to obtain the second pose of the first object.

[0188] According to formula (2), the size of the re-projection error depends on the accuracy of the first pose. In an ideal case, i.e. the first pose is consistent with the true pose, the projection vector feature of the current reference object X i projected into the image coordinate system of the first image and the detection vector feature should be completely overlapped, i.e. the re-projection error E(V o , V h ) is 0. Therefore, the first pose can be optimized by minimizing the re-projection error to obtain a second pose closer to the true pose. Specifically, it can be represented as formula (17) below:

[0189]

[0190] wherein represents the second pose, E(V o , V h ) represents the re-projection error between the detection vector feature V o and the projection vector feature V h , and “argmin” represents the minimization operation.

[0191] In some embodiments, the bundle adjustment method can be used to minimize the re-projection error to optimize the first pose and obtain a more accurate second pose.

[0192] As can be seen from formula (15) to (16), according to the matching pairs and the corresponding matching confidence generated by the multi-granularity matching, the precise pose of the first object, that is, the second pose, can be calculated by fusing the multi-level reprojection errors of different semantic object, jointly optimizing the position error and the direction error based on the idea of nonlinear optimization, and optimizing the error function of the projection vector feature and the detection vector feature with the first pose as the initial value.

[0193] The above-mentioned pose optimization method of the present disclosure has the following beneficial effects:

[0194] 1) The uniqueness of the vector feature is measured from multiple dimensions by combining the absolute distance, relative distance, direction, semantics and neighborhood feature space distribution information between the semantic vector features, thereby improving the robustness and accuracy of the vector feature matching.

[0195] 2) The matching of the semantic vector features is realized by using the structural similarity between the semantic vector features, including the absolute distance, relative distance, direction, semantics and neighborhood distribution information of the features. Compared with the descriptor information, the structured information of the features is more robust and has certain anti-interference ability to the environment, which can improve the accuracy of the feature matching. At the same time, the vectorization of the semantic features can improve the robustness of the semantic features and reduce the influence of unstable edge detection of the neural network on the feature matching.

[0196] 3) The spatial global consistency matching of the lightweight semantic vector features is realized by combining the multi-dimensional vector descriptor, thereby improving the accuracy and robustness of the data association in the complex environment and the long-time environment, and solving the problem of data association failure or error in the complex scene and the cross-time scene.

[0197] 4) In addition, the first pose is optimized by using the reprojection error function based on the vector feature multi-granularity matching and the matching confidence, thereby further improving the accuracy of the pose optimization.

[0198] Figure 6 is a structural schematic block diagram of a pose optimization device of an embodiment of the present disclosure in a hardware implementation of a processing system.

[0199] Referring to Figure 6 As shown in the figure, the pose optimization device 600 can include:

[0200] The acquisition module 602 is configured to acquire a first image corresponding to a scene in which a first object is currently located and a first pose of the first object.

[0201] The detection module 604 is configured to acquire detection vector features of the first image.

[0202] The projection module 606 is configured to acquire projection vector features within a neighborhood range of the first pose in a vector map.

[0203] The matching module 608 is configured to match the detection vector features with the projection vector features to obtain a matching result.

[0204] The optimization module 610 is configured to minimize a re-projection error determined based on the matching result to optimize the first pose to obtain a second pose of the first object.

[0205] In some embodiments, the detection vector features of the first image include vector descriptors of each semantic object in the first image, and the vector descriptors are used to describe semantic labels, shape point positions, center point positions, neighborhood feature position vectors and direction vectors of the semantic objects.

[0206] In some embodiments, the projection vector features include projection vector descriptors of each reference object in a neighborhood range of the first pose in the vector map in an image coordinate system of the first image, and the projection vector descriptors are used to describe semantic labels, shape point positions, center point positions, neighborhood feature position vectors and direction vectors of the reference objects in the image coordinate system of the first image.

[0207] In some embodiments, the matching module 608 is specifically configured to: filter the projection vector features by similarity between the detection vector features and the projection vector features, so that the projection vector features have approximate consistency with the detection vector features in vector feature space distribution; and perform graph matching between the detection vector features and the filtered projection vector features to obtain the matching result.

[0208] In some embodiments, the matching module 608 is specifically configured to: perform graph matching between the vector descriptors of each semantic object in the detection vector features and the projection vector descriptors of each reference object in the filtered projection vector features to obtain a matching matrix and a confidence matrix, elements in the matching matrix are used to represent matching relationships between each semantic object and each reference object, and elements in the confidence matrix are used to represent confidence of matching between each semantic object and each reference object.

[0209] In some embodiments, the reprojection error is determined according to a semantic label of each semantic object in the detected vector feature, and a shape point reprojection error, an end point reprojection error and / or a direction vector reprojection error between the semantic object and each reference object in the filtered projected vector feature; wherein the shape point reprojection error is determined according to a shape corner point position of the semantic object, a shape corner point position of the reference object, a matching relationship and a matching confidence between the semantic object and the reference object; and / or, the end point reprojection error is determined according to an end point position of the semantic object, an end point position of the reference object, a matching relationship and a matching confidence between the semantic object and the reference object; and / or, the direction vector reprojection error is determined according to a direction vector of the semantic object, a direction vector of the reference object, a matching relationship and a matching confidence between the semantic object and the reference object.

[0210] In some embodiments, the reprojection error is represented by a predetermined kernel function of a logarithmic function.

[0211] In some embodiments, the optimization module 610 can be specifically configured to minimize the reprojection error by using a bundle adjustment method to optimize the first pose and obtain the second pose.

[0212] The pose optimization apparatus 600 can include corresponding modules for performing each or several steps in the above flowchart. Therefore, each or several steps in the above flowchart can be performed by corresponding modules, and the apparatus can include one or more of these modules. The modules can be one or more hardware modules specially configured to perform the corresponding steps, or implemented by a processor configured to perform the corresponding steps, or stored in a computer readable medium for implementation by a processor, or implemented by some combination.

[0213] The hardware structure can be implemented using a bus architecture. The bus architecture can include any number of interconnecting buses and bridges, depending on the specific application of the hardware and the overall design constraints. The bus 700 connects various circuits including one or more processors 800, memory 900 and / or hardware modules together. The bus 700 can also connect various other circuits 1000 such as peripheral devices, voltage regulators, power management circuits, external antennas, etc.

[0214] Bus 700 may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component Architecture (EISA) bus. Buses can be classified as address buses, data buses, control buses, and the like. For ease of illustration, the figure shows only one connecting line, but this does not imply that there is only one bus or only one type of bus.

[0215] Any process or method description in the flowchart or otherwise described herein can be understood to represent a module, fragment or portion of code including one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present disclosure includes alternative implementations in which the functions may not be performed in the order shown or discussed, including performing the functions in a substantially simultaneous manner or in a reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present disclosure belong. The processor performs the various methods and processes described above. For example, the method embodiments of the present disclosure can be implemented as a software program that is tangibly contained in a machine-readable medium, such as a memory. In some embodiments, part or all of the software program can be loaded and / or installed via a memory and / or a communication interface. When the software program is loaded into the memory and executed by the processor, one or more steps in the method described above can be performed. Alternatively, in other embodiments, the processor can be configured to perform one of the above methods in any other appropriate manner (e.g., by means of firmware).

[0216] The logic and / or steps represented in the flowchart or otherwise described herein may be embodied in any readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device).

[0217] For the purposes of this specification, a "read only memory" can be any apparatus that can contain, store, communicate, propagate or transport the program for use by or in connection with the instruction execution system, apparatus or device. More specific examples (a non-exhaustive list) of the read only memory include the following: an electrical connection having one or more wires (electrical apparatus), a portable computer diskette (magnetic apparatus), a random access memory (RAM), a read only memory (ROM), an erasable programmable read only memory (EPROM or Flash memory), an optical fiber device, and a portable compact disc read only memory (CD ROM). In addition, a read only memory can even be paper or another suitable medium upon which the program can be printed, as the program can be electronically captured, via for instance an optical scanner, then compiled, interpreted or otherwise processed in a suitable manner if necessary, and stored in the storage medium.

[0218] It should be understood that portions of the present disclosure can be implemented in hardware, software, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and as in another embodiment, any of the following technologies known in the art or their combinations can be used: discrete logic circuitry having logic gates for implementing logic functions on data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), etc.

[0219] Those skilled in the art of the present technology can understand that all or part of the steps of the above-mentioned embodiments can be completed by programs instructing related hardware, and the programs can be stored in a readable storage medium, and when executed, include one or a combination of steps of the method embodiments.

[0220] In addition, each functional unit in each embodiment of the present disclosure can be integrated in one processing module, or each unit can be physically present separately, or two or more units can be integrated in one module. The above integrated module can be realized in the form of hardware or in the form of a software functional module. The integrated module, if realized in the form of a software functional module and sold or used as an independent product, can also be stored in a readable storage medium. The storage medium can be a read only memory, a magnetic disk or an optical disk, etc.

[0221] The present disclosure also provides an electronic device comprising: a memory storing execution instructions; and a processor or other hardware module executing the execution instructions stored in the memory, so that the processor or other hardware module executes the above-mentioned gesture optimization method.

[0222] The present disclosure also provides a readable storage medium, wherein the readable storage medium stores execution instructions, and the execution instructions are executed by a processor to implement the above posture optimization method.

[0223] In the description of the present specification, the description of the terms "one embodiment / way", "some embodiments / ways", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment / way or example are included in at least one embodiment / way or example of the present application. In the present specification, it is not necessarily the same embodiment / way or example that the illustrative expressions of the above terms are described. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments / ways or examples in a suitable manner. In addition, a person skilled in the art can combine and combine the different embodiments / ways or examples described in the present specification and the features of the different embodiments / ways or examples without contradiction.

[0224] In addition, the terms "first", "second" are only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise specifically limited.

[0225] Those skilled in the art should understand that the above embodiments are only for the purpose of clearly illustrating the present disclosure, and are not intended to limit the scope of the present disclosure. Other changes or modifications can be made on the basis of the above disclosure, and these changes or modifications are still within the scope of the present disclosure.

Claims

1. A pose optimization method, characterized in that, The method comprises the following steps: obtaining a first image corresponding to a scene where a first object is currently located and a first pose of the first object; obtaining a detection vector feature of the first image; obtaining a projection vector feature in a neighborhood range of the first pose in a vector map, comprising: analyzing each reference object in the neighborhood range of the first pose from the vector map to obtain a vector feature of each reference object; projecting the vector feature of each reference object to an image coordinate system of the first image to obtain a projection vector feature of each reference object; and performing vector description on the projection vector feature of each reference object to obtain a projection vector descriptor of each reference object; the projection vector feature comprises a projection vector descriptor of each reference object in the neighborhood range of the first pose in the vector map in the image coordinate system of the first image, and the projection vector descriptor is used to describe a semantic label, a shape point position, a center point position, a neighborhood feature position vector and a direction vector of the reference object in the image coordinate system of the first image; matching the detection vector feature with the projection vector feature to obtain a matching result, comprising: filtering the projection vector feature by similarity between the detection vector feature and the projection vector feature, so that the projection vector feature and the detection vector feature have approximate consistency in vector feature space distribution; and performing graph matching on the detection vector feature and the filtered projection vector feature to obtain the matching result; the detection vector feature of the first image comprises a vector descriptor of each semantic object in the first image, and the vector descriptor is used to describe a semantic label, a shape point position, a center point position, a neighborhood feature position vector and a direction vector of the semantic object; minimizing a reprojection error determined based on the matching result to optimize the first pose to obtain a second pose of the first object; wherein the filtering of the projection vector feature by similarity between the detection vector feature and the projection vector feature comprises: calculating similarity of each semantic object in the detection vector feature and each reference object in the projection vector feature in absolute distance; discarding the reference object with similarity less than or equal to a predetermined similarity threshold, and retaining the reference object with similarity greater than the similarity threshold to obtain the filtered projection vector feature; the graph matching of the detection vector feature with the filtered projection vector feature to obtain the matching result comprises: performing graph matching on the vector descriptor of each semantic object in the detection vector feature and the projection vector descriptor of each reference object in the filtered projection vector feature to obtain a matching matrix and a confidence matrix, wherein an element in the matching matrix is used to represent a matching relationship between each semantic object and each reference object, and an element in the confidence matrix is used to represent a confidence of the matching between each semantic object and each reference object.

2. The pose optimization method of claim 1, wherein, the obtaining of the detection vector feature of the first image comprises: performing target detection and multi-frame fusion on the first image of continuous k frames to obtain a feature of each semantic object in the first image, wherein the feature of the semantic object at least comprises a semantic label of the semantic object, and k is an integer greater than 1. The feature of each semantic object is vector described according to the semantic label of the semantic object to obtain a vector descriptor of each semantic object, thereby forming a detection vector feature of the first image.

3. The pose optimization method of claim 1, wherein, The re-projection error is determined according to the semantic label of each semantic object in the detection vector feature, and shape point re-projection error, end point re-projection error and / or direction vector re-projection error between the semantic object and each reference object in the filtered projection vector feature; The shape point re-projection error is obtained according to the shape corner point position of the semantic object, the shape corner point position of the reference object, the matching relationship and matching confidence of the semantic object and the reference object; and / or, The end point re-projection error is obtained according to the end point position of the semantic object, the end point position of the reference object, the matching relationship and matching confidence of the semantic object and the reference object; and / or, The direction vector re-projection error is obtained according to the direction vector of the semantic object, the direction vector of the reference object, the matching relationship and matching confidence of the semantic object and the reference object.

4. The pose optimization method of claim 3, wherein, The re-projection error is represented by using a predetermined kernel function of a logarithmic function.

5. A pose optimization apparatus, characterized by, Comprise: An acquisition module is configured to acquire a first image corresponding to a current scene of a first object and a first pose of the first object; A detection module is configured to acquire a detection vector feature of the first image; A projection module is configured to acquire a projection vector feature in a neighborhood range of the first pose in a vector map, comprising: analyzing each reference object in the neighborhood range of the first pose from the vector map to obtain a vector feature of each reference object; projecting the vector feature of each reference object to an image coordinate system of the first image to obtain a projection vector feature of each reference object; and vector describing the projection vector feature of each reference object to obtain a projection vector descriptor of each reference object, wherein the projection vector feature comprises a projection vector descriptor of each reference object in the neighborhood range of the first pose in the vector map in the image coordinate system of the first image, and the projection vector descriptor is used to describe a semantic label, a shape point position, a center point position, a neighborhood feature position vector and a direction vector of the reference object in the image coordinate system of the first image; A matching module is configured to match the detection vector feature with the projection vector feature to obtain a matching result, comprising: filtering the projection vector feature by similarity between the detection vector feature and the projection vector feature, so that the projection vector feature and the detection vector feature have approximate consistency in vector feature space distribution; and performing graph matching on the detection vector feature and the filtered projection vector feature to obtain the matching result; wherein the detection vector feature of the first image comprises a vector descriptor of each semantic object in the first image, and the vector descriptor is used to describe a semantic label, a shape point position, a center point position, a neighborhood feature position vector and a direction vector of the semantic object; An optimization module is configured to minimize a re-projection error determined based on the matching result to optimize the first pose and obtain a second pose of the first object. The filtering of the projection vector features by detecting similarities between the vector features and the projection vector features comprises: calculating similarities between each semantic object in the detection vector features and each reference object in the projection vector features in terms of absolute distances; discarding reference objects with similarities less than or equal to a predetermined similarity threshold, and retaining reference objects with similarities greater than the similarity threshold, to obtain filtered projection vector features. The matching module is specifically configured to: perform graph matching on vector descriptors of each semantic object in the detection vector features and projection vector descriptors of each reference object in the filtered projection vector features, to obtain a matching matrix and a confidence matrix, wherein elements in the matching matrix are used to represent matching relationships between each semantic object and each reference object, and elements in the confidence matrix are used to represent confidence of matching between each semantic object and each reference object.

6. The pose optimization apparatus according to claim 5, wherein, The re-projection error is determined according to semantic labels of each semantic object in the detection vector features, and shape point-point re-projection errors, end-point re-projection errors and / or direction vector re-projection errors between the semantic object and each reference object in the filtered projection vector features. The shape point-point re-projection error is obtained according to shape corner point positions of the semantic object, shape corner point positions of the reference object, a matching relationship and a confidence of matching between the semantic object and the reference object; and / or, The end-point re-projection error is obtained according to end-point positions of the semantic object, end-point positions of the reference object, a matching relationship and a confidence of matching between the semantic object and the reference object; and / or, The direction vector re-projection error is obtained according to a direction vector of the semantic object, a direction vector of the reference object, a matching relationship and a confidence of matching between the semantic object and the reference object.

7. The pose optimization apparatus according to claim 6, wherein, The re-projection error is represented by using a predetermined kernel function of a logarithmic function.

8. An electronic device, comprising: The method comprises: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, so that the processor executes the pose optimization method in any one of claims 1 to 4.

9. A readable storage medium, characterized by, The readable storage medium stores execution instructions, and the execution instructions are executed by the processor to implement the pose optimization method in any one of claims 1 to 4.

Images