Target positioning method and electronic device

Abstract

The application discloses a target positioning method and electronic equipment, which is used for realizing target positioning based on key line segment detection, and improving the accuracy of target positioning by detecting key line segments with more obvious features. The method comprises the following steps: acquiring a to-be-detected image containing a target object; performing feature extraction on the to-be-detected image to obtain a feature image, determining an object position of the target object and a line segment position of a key line segment representing the posture of the target object according to the feature image; and determining the pose of the target object according to the object position and the line segment position, so as to position the target object.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent transportation, in particular to a target positioning method and an electronic device. BACKGROUND

[0002] At present, the target positioning method is mainly based on target detection or a key point-based scheme, which has the following disadvantages:

[0003] (1) In the scheme based on target detection, the 2-dimensional target detection frame can only position the overall position of the target in the image, and it is difficult to reflect the real position of the target in the 3-dimensional space, thereby causing a large error in the network prediction of the target position. The 3-dimensional target detection technology currently has problems such as great difficulty in manual annotation, poor prediction accuracy, high resource requirement, and difficulty in deploying embedded devices, and the current technology is not mature.

[0004] (2) In the scheme based on key points, since the key points of the target are easily occluded and have unobvious visual features, the data annotation is difficult and the quality of the annotated data is not high, thereby causing a large deviation in the detection result. SUMMARY

[0005] The present application provides a target positioning method and an electronic device, which are used for realizing target positioning based on key line segment detection, and improving the accuracy of target positioning by detecting key line segments with more obvious features.

[0006] In a first aspect, the present application provides a target positioning method, which comprises the following steps:

[0007] Obtaining a to-be-detected image containing a target object;

[0008] Performing feature extraction on the to-be-detected image to obtain a feature image, determining an object position of the target object and a line segment position of a key line segment representing the posture of the target object according to the feature image;

[0009] Determining the pose of the target object according to the object position and the line segment position, and positioning the target object.

[0010] The method of the present application using key line segment detection can more reasonably reflect the actual position of the target object in the three-dimensional space. Compared with other key point-based methods, the method of using key line segment detection can obtain more accurate detection results because the visual features are more obvious, thereby more accurately predicting the position of the target object, and achieving the purpose of positioning the target object.

[0011] As an optional implementation, the feature extraction on the to-be-detected image to obtain a feature image comprises:

[0012] extracting features from the to-be-tested image in different scale ranges to obtain features of multiple scales, wherein the features of different scales are used to represent different contents of the image;

[0013] fusing the features of multiple scales to obtain a feature image containing multi-scale features.

[0014] As an optional implementation, the line segment position of the key line segment representing the posture of the target object is determined in the following manner:

[0015] According to the feature image, each center point and a vector corresponding to each center point are determined, the center point representing the center of the key line segment, and the vector corresponding to the center point representing a vector of the center of the key line segment pointing to two end points;

[0016] According to each center point and a vector corresponding to each center point, a line segment position of the key line segment corresponding to each center point is determined.

[0017] As an optional implementation,

[0018] According to the feature image, an object position and an object category of the target object are determined;

[0019] When the object category is a preset category, a line segment position and a line segment category of a key line segment representing the posture of the target object are determined according to the feature image, the line segment category representing the position of the key line segment in the target object;

[0020] According to the object position of the target object, the line segment position and the line segment category of the key line segment, a pose of the target object is determined.

[0021] As an optional implementation, the object position of the target object and the line segment position of the key line segment representing the posture of the target object are determined in the following manner:

[0022] The to-be-tested image is input into a detection network, and an object position and an object category of the target object and a line segment position and a line segment category of the key line segment are output;

[0023] The detection network is obtained by training using a labeled sample image, wherein the sample image is labeled with an object position and an object category of the target object and a line segment position and a line segment category of the key line segment; the training process includes:

[0024] The labeled sample image is input into an initial detection network, a first loss function is determined according to the output object position of the target object and the labeled object position of the target object, and a second loss function is determined according to the output line segment position of the key line segment and the labeled line segment position of the key line segment;

[0025] determine a loss function according to the first loss function and the second loss function, adjust the network parameters of the initial detection network according to the loss function, and determine that the training is completed when a loss function value is less than a first threshold or a training iteration number reaches a second threshold, to obtain a trained detection network.

[0026] As an optional implementation, the detection network comprises a feature extraction network, a target detection network, and a line segment detection network.

[0027] The feature extraction network is configured to perform feature extraction on the input image to be detected to obtain a feature image.

[0028] The target detection network is configured to perform target detection on the feature image to output an object position and an object category of a target object.

[0029] The line segment detection network is configured to perform line segment detection on the feature image to output a line segment position and a line segment category of a key line segment.

[0030] As an optional implementation, the feature extraction network comprises a backbone network and a fusion network, and the backbone network comprises a plurality of groups of convolutional layers.

[0031] The backbone network is configured to perform feature extraction on the image to be detected to obtain a plurality of features, and one group of convolutional layers corresponds to one feature.

[0032] The fusion network is configured to fuse the plurality of features to obtain the feature image.

[0033] As an optional implementation, the line segment detection network is configured to perform line segment detection on the feature image to output each center point and a vector corresponding to each center point, and the second loss function is determined by the following steps.

[0034] For each center point, a first sub-loss is determined according to the output center point and the labeled center point, a second sub-loss is determined according to an angle of the vector corresponding to the center point and a labeled vector angle, and a third sub-loss is determined according to a distance of the vector corresponding to the center point and a labeled distance.

[0035] The second loss function is determined according to the first sub-loss, the second sub-loss, and the third sub-loss.

[0036] As an optional implementation, after the image to be detected containing the target object is obtained, the method further comprises:

[0037] The size and pixel data range of the image to be detected are normalized to obtain a normalized image to be detected, and the feature extraction network is configured to perform feature extraction on the normalized image to be detected to obtain the feature image.

[0038] In a second aspect, an electronic device is provided. The electronic device includes a processor and a memory. The memory is configured to store a program executable by the processor. The processor is configured to read the program in the memory and perform the following steps:

[0039] obtaining a to-be-tested image containing a target object;

[0040] performing feature extraction on the to-be-tested image to obtain a feature image, determining an object position of the target object and a line segment position of a key line segment representing a posture of the target object according to the feature image;

[0041] determining a pose of the target object according to the object position and the line segment position, for positioning the target object.

[0042] As an optional implementation, the processor is specifically configured to perform the following steps:

[0043] performing feature extraction on the to-be-tested image from different scale ranges to obtain features of multiple scales, wherein the features of different scales are used to represent different contents of the image;

[0044] fusing the features of multiple scales to obtain the feature image containing multi-scale features.

[0045] As an optional implementation, the processor is specifically configured to determine the line segment position of the key line segment representing the posture of the target object by the following steps:

[0046] determining each center point and a vector corresponding to each center point according to the feature image, wherein the center point represents a center of the key line segment, and the vector corresponding to the center point represents a vector of the center of the key line segment pointing to two end points;

[0047] determining the line segment position of the key line segment corresponding to each center point according to each center point and the vector corresponding to each center point.

[0048] As an optional implementation, the processor is specifically configured to perform the following steps:

[0049] determining an object position of the target object and an object category according to the feature image;

[0050] when the object category is a preset category, determining a line segment position of a key line segment representing a posture of the target object and a line segment category representing a position of the key line segment in the target object according to the feature image;

[0051] determining a pose of the target object according to the object position, the line segment position of the key line segment and the line segment category.

[0052] As an optional implementation, the processor is specifically configured to determine the object position of the target object and the line segment position of the key line segment representing the posture of the target object by:

[0053] inputting the to-be-tested image into the detection network to output the object position and the object category of the target object and the line segment position and the line segment category of the key line segment;

[0054] The detection network is obtained by training using a labeled sample image, wherein the sample image is labeled with the object position and the object category of the target object and the line segment position and the line segment category of the key line segment; the training process comprises:

[0055] inputting the labeled sample image into an initial detection network, determining a first loss function according to the output object position of the target object and the labeled object position of the target object, and determining a second loss function according to the output line segment position of the key line segment and the labeled line segment position of the key line segment;

[0056] determining a loss function according to the first loss function and the second loss function, adjusting network parameters of the initial detection network according to the loss function, and determining that the training is completed when the loss function value is less than a first threshold value or the number of training iterations reaches a second threshold value, to obtain the trained detection network.

[0057] As an optional implementation, the detection network comprises a feature extraction network, a target detection network and a line segment detection network.

[0058] The feature extraction network is used for feature extraction on the input to-be-tested image to obtain a feature image.

[0059] The target detection network is used for target detection on the feature image to output the object position and the object category of the target object.

[0060] The line segment detection network is used for line segment detection on the feature image to output the line segment position and the line segment category of the key line segment.

[0061] As an optional implementation, the feature extraction network comprises a backbone network and a fusion network; the backbone network comprises a plurality of groups of convolutional layers.

[0062] The backbone network is used for feature extraction on the to-be-tested image to obtain a plurality of features, and one group of convolutional layers corresponds to one feature.

[0063] The fusion network is used for fusion of the plurality of features to obtain the feature image.

[0064] As an optional implementation, the line segment detection network is used to detect line segments in the feature image and output each center point and the vector corresponding to each center point; the processor is specifically configured to determine the second loss function through the following steps:

[0065] For each center point, a first sub-loss is determined based on the output center point and the labeled center point; a second sub-loss is determined based on the angle of the vector corresponding to the center point and the angle of the labeled vector; and a third sub-loss is determined based on the distance between the vector corresponding to the center point and the distance between the labeled vector.

[0066] The second loss function is determined based on the first sub-loss, the second sub-loss, and the third sub-loss.

[0067] As an optional implementation, after acquiring the image to be tested containing the target object, the processor is further configured to execute:

[0068] The size and pixel data range of the image to be tested are normalized to obtain a normalized image to be tested, which is then used to extract features from the normalized image to obtain a feature image.

[0069] Thirdly, embodiments of the present invention also provide a target positioning device, the device comprising:

[0070] Image acquisition unit, used to acquire the test image containing the target object;

[0071] The key position detection unit is used to extract features from the image to be tested to obtain a feature image, and determine the object position of the target object and the line segment position of the key line segment representing the pose of the target object based on the feature image.

[0072] A pose determination unit is used to determine the pose of the target object based on the object position and the line segment position, and to locate the target object.

[0073] As an optional implementation, the key position detection unit is specifically used for:

[0074] Feature extraction is performed on the image under test from different scale ranges to obtain features at multiple scales, wherein features at different scales are used to characterize different contents of the image.

[0075] Features from multiple scales are fused to obtain a feature image containing features from multiple scales.

[0076] As an optional implementation, the key position detection unit is specifically used to determine the line segment position of the key line segment representing the pose of the target object in the following manner:

[0077] Based on the feature image, each center point and the corresponding vector are determined. The center point represents the center of the key line segment, and the vector corresponding to the center point represents the vector from the center of the key line segment to the two endpoints.

[0078] Based on each center point and its corresponding vector, determine the position of the key line segment corresponding to each center point.

[0079] As an optional implementation, the pose determination unit is specifically used for:

[0080] The object location and object category of the target object are determined based on the feature image;

[0081] When the object category is a preset category, the line segment position and line segment category of the key line segment representing the pose of the target object are determined according to the feature image, and the line segment category represents the position of the key line segment in the target object;

[0082] The pose of the target object is determined based on the object position, the line segment position of the key line segments, and the line segment type.

[0083] As an optional implementation, the key position detection unit is specifically used to determine the object position of the target object and the line segment position of the key line segment representing the pose of the target object in the following manner:

[0084] The image to be tested is input into the detection network, which outputs the object position and object category of the target object, as well as the line segment position and line segment category of the key line segment;

[0085] The detection network is trained using labeled sample images, where the sample images are labeled with the object location and object category of the target object, as well as the line segment location and line segment category of key line segments; the training process includes:

[0086] The labeled sample image is input into the initial detection network. The first loss function is determined based on the object position of the output target object and the object position of the labeled target object. The second loss function is determined based on the line segment position of the output key line segment and the line segment position of the labeled key line segment.

[0087] The loss function is determined based on the first loss function and the second loss function. The network parameters of the initial detection network are adjusted according to the loss function. When the loss function value is less than the first threshold or the number of training iterations reaches the second threshold, the training is determined to be complete, and the trained detection network is obtained.

[0088] As an optional implementation, the detection network includes a feature extraction network, an object detection network, and a line segment detection network;

[0089] The feature extraction network is used to extract features from the input image to be tested, and obtain a feature image;

[0090] The target detection network is used to detect targets in the feature image and output the object location and object category of the target object;

[0091] The line segment detection network is used to detect line segments in feature images and output the line segment positions and line segment categories of key line segments.

[0092] As an optional implementation, the feature extraction network includes a backbone network and a fusion network; the backbone network includes multiple sets of convolutional layers.

[0093] The backbone network is used to extract features from the image under test, obtaining multiple features, with one convolutional layer corresponding to one feature;

[0094] The fusion network is used to fuse multiple features to obtain a feature image.

[0095] As an optional implementation, the line segment detection network is used to detect line segments in the feature image and output each center point and its corresponding vector; the key position detection unit is specifically used to determine the second loss function through the following steps:

[0096] For each center point, a first sub-loss is determined based on the output center point and the labeled center point; a second sub-loss is determined based on the angle of the vector corresponding to the center point and the angle of the labeled vector; and a third sub-loss is determined based on the distance between the vector corresponding to the center point and the distance between the labeled vector.

[0097] The second loss function is determined based on the first sub-loss, the second sub-loss, and the third sub-loss.

[0098] As an optional implementation, after acquiring the image to be tested containing the target object, the image acquisition unit is further configured to:

[0099] The size and pixel data range of the image to be tested are normalized to obtain a normalized image to be tested, which is then used to extract features from the normalized image to obtain a feature image.

[0100] Fourthly, embodiments of the present invention also provide a computer storage medium having a computer program stored thereon, which, when executed by a processor, is used to implement the steps of the method described in the first aspect above.

[0101] These or other aspects of this application will become more apparent in the following description of embodiments. Attached Figure Description

[0102] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0103] Figure 1 A flowchart illustrating a specific implementation of a target localization method provided in this embodiment of the invention;

[0104] Figures 2A-2C A schematic diagram of a key line segment of a vehicle provided in an embodiment of the present invention;

[0105] Figure 3 A schematic diagram of a line segment representation method provided in an embodiment of the present invention;

[0106] Figures 4A-4B This invention provides a schematic diagram of key line segments annotated in a real-world scenario.

[0107] Figure 5 This is a schematic diagram of the structure of a detection network provided in an embodiment of the present invention;

[0108] Figure 6 This is a schematic diagram of the structure of a line segment detection network provided in an embodiment of the present invention;

[0109] Figure 7 A flowchart illustrating the implementation of a target localization method provided in an embodiment of the present invention;

[0110] Figure 8 A schematic diagram of an electronic device provided in an embodiment of the present invention;

[0111] Figure 9 This is a schematic diagram of a target positioning device provided in an embodiment of the present invention. Detailed Implementation

[0112] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0113] In this embodiment of the invention, the term "and / or" describes the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The character " / " generally indicates that the preceding and following associated objects have an "or" relationship.

[0114] The application scenarios described in the embodiments of this invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of this invention, and do not constitute a limitation on the technical solutions provided by the embodiments of this invention. Those skilled in the art will understand that with the emergence of new application scenarios, the technical solutions provided by the embodiments of this invention are also applicable to similar technical problems. In the description of this invention, unless otherwise stated, "multiple" means two or more.

[0115] Current target localization methods are mainly based on target detection or key point-based approaches, which have the following drawbacks:

[0116] (1) In object detection-based schemes, 2D object detection boxes can only locate the overall position of the object in the image, which is difficult to reflect the true position of the object in 3D space, resulting in a large error in the target position predicted by the network. At present, 3D object detection technology still has problems such as high difficulty in manual annotation, poor prediction accuracy, high resource requirements and difficulty in deploying embedded devices. The current technology is not mature.

[0117] (2) In the key point-based scheme, the key points of the target are easily occluded and the visual features are not obvious, which makes data annotation difficult and the quality of the annotation data is not high, resulting in a large deviation in the detection results.

[0118] Taking vehicle localization as an example, it plays a crucial role in smart parking, intelligent transportation, and traffic enforcement systems. However, in object detection-based solutions, 2D bounding boxes can only locate the vehicle's overall position in an image, failing to reflect its true position in 3D space, leading to significant errors in the network's predicted vehicle location. Furthermore, 3D object detection technology still faces numerous challenges and is not yet mature. In keypoint-based solutions, the easy occlusion and indistinct visual features of vehicle keypoints result in difficult data annotation and low-quality annotation data, leading to substantial deviations in detection results.

[0119] This embodiment provides a target localization method that accurately locates the position and attitude of a vehicle by detecting the position of the target object and the position of the key line segments representing the attitude of the target object, thereby improving the accuracy of target localization.

[0120] like Figure 1 As shown, the specific implementation flow of the target localization method provided in this embodiment is as follows:

[0121] Step 100: Obtain the image to be tested containing the target object;

[0122] In practice, the target objects in this embodiment include, but are not limited to, vehicles, people, animals, robots, etc. This embodiment does not impose too many limitations on them.

[0123] In practice, the image to be tested can be acquired through camera equipment, monitoring equipment, checkpoint equipment, etc. This embodiment does not impose too many restrictions on how to acquire the image to be tested.

[0124] In some embodiments, after acquiring the image to be tested containing the target object, the method further includes:

[0125] The size and pixel data range of the image to be tested are normalized to obtain a normalized image to be tested. This normalized image is then used to extract features from the normalized image to obtain a feature image. Based on the feature image, the object position of the target object and the line segment positions of the key line segments representing the pose of the target object are determined. Based on the object position and the line segment positions, the pose of the target object is determined.

[0126] In practice, the size of the image to be tested can be normalized to a fixed size, and the range of pixel data in the image to be tested can be normalized to 0 to 255.

[0127] Step 101: Extract features from the image to be tested to obtain a feature image, and determine the object position of the target object and the line segment position of the key line segment representing the pose of the target object based on the feature image;

[0128] In implementation, taking a vehicle as the target object as an example, this embodiment first defines the key line segments from different perspectives of the vehicle, such as... Figures 2A-2C As shown in the diagram, this embodiment provides a schematic diagram of a vehicle's key line segments. Figure 2A These are schematic diagrams of the car body from different perspectives. Figure 2B For the defined critical line segments of the vehicle, Figure 2C This refers to the corresponding position of the vehicle's critical line segment on the vehicle body. It should be noted that the vehicle critical line segment defined in this embodiment is merely an example; vehicle critical line segments defined based on the same principle are all within the scope of protection of this invention.

[0129] In some embodiments, a feature image is obtained by feature extraction from the image to be tested in the following manner:

[0130] Feature extraction is performed on the image under test from different scale ranges to obtain features at multiple scales, wherein features at different scales are used to characterize different contents of the image; the features at multiple scales are fused to obtain a feature image containing features at multiple scales.

[0131] This embodiment obtains a feature image containing multi-scale features by fusing features from multiple scales, which can improve the accuracy of detecting subsequent target objects and their key line segments. Since the feature image contains more comprehensive image information, it can improve detection accuracy.

[0132] In some embodiments, the position of the key line segment representing the pose of the target object is determined in the following manner:

[0133] Based on the feature image, each center point and its corresponding vector angle and distance are determined. The center point represents the center of the key line segment, the vector angle represents the angle of the vector from the center to the endpoint of the key line segment relative to a preset direction, and the distance represents the distance from the center to the endpoint of the key line segment.

[0134] Based on each center point and its corresponding vector angle and distance, determine the position of the key line segment corresponding to each center point.

[0135] During implementation, such as Figure 3 As shown in the diagram, this embodiment provides a schematic representation of a line segment. The left side represents a line segment, and the right side shows a method for representing line segments using a center point and a vector. Specifically, key line segments are represented in the following way:

[0136] The center point of the key line segment, and the vectors (vector angle and vector distance) pointing from the center point to the two ends (start point and end point).

[0137] Among them, key line segments at different locations are represented by different line segment categories. The line segment category is represented by the category of the center point of the line segment. For example, the line segment category is represented by the category of the center point (0~24), and thus the key line segments at different locations are represented.

[0138] In practice, the key line segment of a target object can be marked as: the position of the center point R (x R ,y R The category of the center point (C), and the angle θ of the vector pointing from the center point to one of its endpoints. s and distance d s The angle θ of the vector whose center point points to the other endpoint e and distance d e .

[0139] In some embodiments, this embodiment also needs to determine the object category and line segment category to determine whether the target object is the object to be detected, and the position of each key line segment in the target object. The specific steps are as follows:

[0140] The object location and object category of the target object are determined based on the feature image;

[0141] When the object category is a preset category, the line segment position and line segment category of the key line segment representing the pose of the target object are determined according to the feature image, and the line segment category represents the position of the key line segment in the target object;

[0142] The pose of the target object is determined based on the object position, the line segment position of the key line segments, and the line segment type.

[0143] Step 102: Determine the pose of the target object based on the object position and the line segment position, for positioning the target object.

[0144] This embodiment determines the overall position of the target object in the image by using the object position. For example, the coordinates of the four vertices of the smallest bounding rectangle of the target object are used as the object position, and the line segment positions of each key line segment of the target object are further determined, thereby determining the position and posture of the target object.

[0145] In practice, this embodiment can also detect the pose of the target object through a detection network. Specifically, the object position and the line segment positions of the key line segments representing the pose of the target object are determined in the following ways:

[0146] The image to be tested is input into the detection network, which outputs the object position and object category of the target object, as well as the line segment position and line segment category of the key line segment;

[0147] The detection network is trained using labeled sample images, wherein the sample images are labeled with the object location and object category of the target object, as well as the line segment location and line segment category of the key line segments;

[0148] In practice, taking vehicles as an example, sample images are labeled in the following way:

[0149] First, based on the predefined key line segments on the vehicle, annotate the bounding box (vehicle position) of the vehicle in the current sample image, i.e., the minimum bounding rectangle of the vehicle, and label the vehicle category. When annotating the key line segments, it's important to note that in real-world scenarios, the key line segments may exhibit a certain curvature. To address this, simply connect the two endpoints of the key line segment to form a line segment, such as... Figures 4A-4B As shown in the figure, this embodiment also provides a schematic diagram of key line segments annotated in a real scene. The annotated sample image can be used as a training set to be input into the detection network for training. By identifying the overall position of the vehicle and the position of the key line segments, the detection network can more accurately analyze the current position and attitude of the vehicle, provide accurate vehicle positioning, and thus improve the accuracy of subsequent processes.

[0150] In some embodiments, such as Figure 5 As shown in the diagram, this embodiment provides a schematic diagram of a detection network. The detection network in this embodiment specifically includes:

[0151] (1) Feature extraction network;

[0152] The feature extraction network is used to extract features from the input image to be tested, and obtain a feature image.

[0153] In some embodiments, the feature extraction network includes a backbone network and a fusion network; the backbone network includes multiple sets of convolutional layers; each set of convolutional layers includes one or more convolutional layers.

[0154] 1a) Backbone network;

[0155] The backbone network is used to extract features from the image under test, obtaining multiple features, with one convolutional layer corresponding to one feature.

[0156] In implementation, the backbone network can use typical neural networks, such as ResNet-50 and ResNet101, for feature extraction. The backbone network primarily extracts features from the image under test, and its output contains features at multiple scales, denoted as {F1, F2, ...}. Each scale has a different resolution and number of channels, representing the hierarchical features of the image from low to high levels. Taking ResNet-50 as an example, the feature outputs from stage 2 to stage 5 can be used as the output of this backbone network.

[0157] Optionally, the size and pixel data range of the image to be tested are normalized to obtain a normalized image to be tested. The normalized image to be tested is then input into the backbone network for feature extraction. In practice, the size of the image to be tested can be normalized to a fixed size, and the pixel data range in the image to be tested can be normalized to 0 to 255.

[0158] 1b) Converged networks;

[0159] The fusion network is used to fuse multiple features to obtain a feature image.

[0160] A fusion network is added after the backbone network to perform feature fusion. The fusion network structure can adopt a Feature Pyramid Network (FPN) structure, which can fuse the information extracted from each convolutional layer of the backbone network to generate multi-scale features. This fusion network mainly fuses the multi-scale features of the input, and the output is a feature image F∈R containing multi-scale information. H×W×N Where H is the height of the image to be tested, W is the width of the image to be tested, and N is the feature dimension.

[0161] (2) Target detection network;

[0162] The object detection network is used to detect objects on the feature image and outputs the object location and object category. This network detects the overall contour of the object (e.g., the bounding rectangle of a vehicle) and consists of a category discrimination branch and a location regression branch. After a multi-scale feature image F is input into the network, it outputs the category and location of several candidate detection boxes. The loss function generated by this network is denoted as L1.

[0163] In practice, multi-scale feature images are input into the object detection network for object bounding box classification and location regression tasks, generating vehicle candidate boxes. Taking vehicles as an example, the classification is whether it is a vehicle or not, and the object location is the coordinates of the four vertices of the vehicle candidate box.

[0164] (3) Line segment detection network;

[0165] The line segment detection network is used to detect line segments in feature images and output the line segment positions and line segment categories of key line segments.

[0166] In implementation, multi-scale feature images are input into a line segment detection network to predict the position and category of key line segments. For each target object in the test image, the network outputs the object's position and labels each key line segment of the target object in the test image. Based on the object position and the line segment positions of the key line segments, the network further infers the target object's specific pose and position information.

[0167] In some embodiments, the line segment detection network is used to detect line segments in the feature image and output each center point and its corresponding vector. In this implementation, a multi-scale feature image F is input into the line segment detection network to predict key line segment information of the target object. Figure 6 As shown in the diagram, this embodiment provides a schematic diagram of a line segment detection network. The structure of this line segment detection network is a cascaded hourglass neural network. The input of the line segment detection network is a feature image, and the output is divided into three branches, as shown below:

[0168] Branch 1: The location and category of the center point R. The category of the center point indicates the category of the critical line segment.

[0169] This branch outputs a heatmap with dimensions H×W×25. The value at each location in the heatmap represents the probability that the point is a key segment of a certain category. For example, the probability value at location (x, y) in the i-th layer is p. i(x,y). The probability value of each layer is compared with a threshold. If it is greater than the threshold, it indicates that the position is the center point of the key line segment and the category C(x,y) is the position of the corresponding layer number. The heatmap is composed of multiple small colored blocks, each of which represents a value. The heatmap uses color to represent numbers, making the data presentation more intuitive and the comparison more obvious. It is used to intuitively present the global expression changes of multiple genes in multiple samples and to show the clustering relationship of the expression levels of multiple genes in multiple samples.

[0170]

[0171] Where, p i (x,y) represents the probability value of position (x,y) in the i-th layer, and C(x,y) represents the category of the center point R = (x,y).

[0172] Branch 2: The angle (or direction) of the vector corresponding to the center point.

[0173] This branch is used to predict the direction of the key line segment, and outputs a feature map of size H×W×2, representing the angle θ of the vector from each center point to the two endpoints of the key line segment. s and θ e The angle value of the vector is the angle between the vector and a preset direction, such as the angle between the vector and the horizontal or vertical direction. Optionally, this angle value is normalized to the range [0, 2π].

[0174] Branch 3: Distance between the vectors corresponding to the center point.

[0175] This branch is used to predict the distance of key segments, and outputs a feature map of size H×W×2, representing the distance value d from each center point to the two endpoints of the key segment. s and d e Optionally, the distance value can be normalized to a preset range.

[0176] It should be noted that in this embodiment, one center point corresponds to one key line segment, and one center point corresponds to two vectors. Therefore, the angle of the vector corresponding to one center point is two, namely the angle value of the vector pointing from the center point to one endpoint and the angle value of the vector pointing from the center point to the other endpoint. Similarly, the distance of the vector corresponding to one center point is two, namely the distance value of the vector pointing from the center point to one endpoint and the distance value of the vector pointing from the center point to the other endpoint.

[0177] The training process of the detection network in this embodiment is as follows:

[0178] The labeled sample image is input into the initial detection network. The first loss function is determined based on the object position of the output target object and the object position of the labeled target object. The second loss function is determined based on the line segment position of the output key line segment and the line segment position of the labeled key line segment.

[0179] The loss function is determined based on the first loss function and the second loss function. The network parameters of the initial detection network are adjusted according to the loss function. When the loss function value is less than the first threshold or the number of training iterations reaches the second threshold, the training is determined to be complete, and the trained detection network is obtained.

[0180] In some embodiments, the line segment detection network is used to detect line segments in a feature image and output each center point and the vector corresponding to each center point; the second loss function is determined through the following steps:

[0181] For each center point, a first sub-loss is determined based on the output center point and the labeled center point; a second sub-loss is determined based on the angle of the vector corresponding to the center point and the angle of the labeled vector; and a third sub-loss is determined based on the distance between the vector corresponding to the center point and the distance between the labeled vector.

[0182] The second loss function is determined based on the first sub-loss, the second sub-loss, and the third sub-loss.

[0183] In implementation, the loss function in this embodiment includes four parts, as follows:

[0184] The first loss function, namely Loss1, is the loss function for the object location of the target object. It consists of the target category classification loss and the location regression loss. The target category loss is implemented by the cross-entropy loss function, and the location regression is implemented by the L2 loss function.

[0185] The second loss function includes the first sub-loss, the second sub-loss, and the third sub-loss, where:

[0186] The first sub-loss, namely the centroid loss function Loss2, is implemented by the cross-entropy loss function of the centroid's class.

[0187] The second sub-loss, namely the vector angle loss function Loss3, is implemented by the angle L2 loss function.

[0188] The third sub-loss, namely the vector distance loss function Loss4, is implemented by the distance loss function L2.

[0189] This embodiment employs a multi-task training method, and the overall loss function of the detection network is:

[0190] Loss sum=Loss1+Loss2+Loss3+Loss4.

[0191] In implementation, if a certain point (x) r ,y r If the predicted point is the center of a critical line segment, then the vector from the center point to one endpoint and the vector from the center point to the other endpoint are respectively:

[0192]

[0193]

[0194] Where, d s d represents the distance value of the vector from the center point to one endpoint of the critical line segment. e θ represents the distance value of the vector from the center point to the other endpoint of the critical line segment. s θ represents the angle value of the vector from the center point to one endpoint of the critical line segment. e This represents the angle value of the vector from the center point to the other endpoint of the critical line segment. This represents the vector from the center point to one endpoint of the critical line segment. This represents the vector from the center point to the other endpoint of the critical line segment.

[0195] The coordinates of the two endpoints of the predicted critical line segment are as follows:

[0196]

[0197]

[0198] Among them, (x s ,y s (x) represents the coordinates of one endpoint of the critical line segment. e ,y e (x) represents the coordinates of the other endpoint of the critical line segment. r ,y r ) represents the coordinates of the center point.

[0199] This embodiment obtains the vehicle's position and attitude by fitting and assembling the predicted positions of each key segment.

[0200] like Figure 7 As shown in the figure, this embodiment also provides an implementation flow of the target localization method, as follows:

[0201] Step 700: Obtain the image to be tested, and normalize the image to be tested to obtain the normalized image to be tested;

[0202] Step 701: Input the normalized image to be tested into the backbone network for feature extraction and output multiple features;

[0203] Step 702: Input multiple features into the fusion network to perform feature fusion and obtain a feature image;

[0204] Step 703: Input the feature image into the object detection network for object detection, and output the object location and object category of the target object;

[0205] Step 704: Input the feature image into the line segment detection network to perform line segment detection, and output the line segment position and line segment category of the key line segments;

[0206] Step 705: When the object category is a preset category, determine the pose of the target object based on the object position, the line segment position of the key line segment, and the line segment category.

[0207] This embodiment defines and uses key line segment information of the vehicle to detect and locate its position. Multiple key line segment information can be defined based on the actual shape of the vehicle for subsequent detection and recognition. A multi-task approach combining object detection and key line segment detection is used to detect key line segments and predict the vehicle's position. Compared to other current object detection-based localization methods, the key line segment detection method more accurately reflects the vehicle's actual position in three-dimensional space. Compared to other key point-based methods, due to the more obvious visual features, the key line segment detection method yields more accurate detection results, thus more accurately predicting the vehicle's position and achieving the goal of vehicle localization.

[0208] Based on the same inventive concept, this embodiment of the invention also provides an electronic device. Since this electronic device is the same as the electronic device in the method of this embodiment of the invention, and the principle of solving the problem by this electronic device is similar to that of this method, the implementation of this electronic device can refer to the implementation of the method, and the repeated parts will not be described again.

[0209] like Figure 8 As shown, the electronic device includes a processor 800 and a memory 801. The memory 801 stores programs executable by the processor 800. The processor 800 reads the programs from the memory 801 and performs the following steps:

[0210] Obtain the image to be tested that contains the target object;

[0211] Feature extraction is performed on the image to be tested to obtain a feature image. Based on the feature image, the object position of the target object and the line segment position of the key line segment representing the pose of the target object are determined.

[0212] The pose of the target object is determined based on the object position and the line segment position, and is used to locate the target object.

[0213] As an optional implementation, the processor 800 is specifically configured to perform:

[0214] Feature extraction is performed on the image under test from different scale ranges to obtain features at multiple scales, wherein features at different scales are used to characterize different contents of the image.

[0215] Features from multiple scales are fused to obtain a feature image containing features from multiple scales.

[0216] As an optional implementation, the processor 800 is specifically configured to determine the line segment positions of key line segments characterizing the pose of the target object in the following manner:

[0217] Based on the feature image, each center point and the corresponding vector are determined. The center point represents the center of the key line segment, and the vector corresponding to the center point represents the vector from the center of the key line segment to the two endpoints.

[0218] Based on each center point and its corresponding vector, determine the position of the key line segment corresponding to each center point.

[0219] As an optional implementation, the processor 800 is specifically configured to perform:

[0220] The object location and object category of the target object are determined based on the feature image;

[0221] When the object category is a preset category, the line segment position and line segment category of the key line segment representing the pose of the target object are determined according to the feature image, and the line segment category represents the position of the key line segment in the target object;

[0222] The pose of the target object is determined based on the object position, the line segment position of the key line segments, and the line segment type.

[0223] As an optional implementation, the processor 800 is specifically configured to determine the object position of the target object and the line segment position of the key line segment characterizing the pose of the target object by:

[0224] The image to be tested is input into the detection network, which outputs the object position and object category of the target object, as well as the line segment position and line segment category of the key line segment;

[0225] The detection network is trained using labeled sample images, where the sample images are labeled with the object location and object category of the target object, as well as the line segment location and line segment category of key line segments; the training process includes:

[0226] The labeled sample image is input into the initial detection network. The first loss function is determined based on the object position of the output target object and the object position of the labeled target object. The second loss function is determined based on the line segment position of the output key line segment and the line segment position of the labeled key line segment.

[0227] The loss function is determined based on the first loss function and the second loss function. The network parameters of the initial detection network are adjusted according to the loss function. When the loss function value is less than the first threshold or the number of training iterations reaches the second threshold, the training is determined to be complete, and the trained detection network is obtained.

[0228] As an optional implementation, the detection network includes a feature extraction network, an object detection network, and a line segment detection network;

[0229] The feature extraction network is used to extract features from the input image to be tested, and obtain a feature image;

[0230] The target detection network is used to detect targets in the feature image and output the object location and object category of the target object;

[0231] The line segment detection network is used to detect line segments in feature images and output the line segment positions and line segment categories of key line segments.

[0232] As an optional implementation, the feature extraction network includes a backbone network and a fusion network; the backbone network includes multiple sets of convolutional layers.

[0233] The backbone network is used to extract features from the image under test, obtaining multiple features, with one convolutional layer corresponding to one feature;

[0234] The fusion network is used to fuse multiple features to obtain a feature image.

[0235] As an optional implementation, the line segment detection network is used to detect line segments in the feature image and output each center point and the vector corresponding to each center point; the processor 800 is specifically configured to determine the second loss function through the following steps:

[0236] For each center point, a first sub-loss is determined based on the output center point and the labeled center point; a second sub-loss is determined based on the angle of the vector corresponding to the center point and the angle of the labeled vector; and a third sub-loss is determined based on the distance between the vector corresponding to the center point and the distance between the labeled vector.

[0237] The second loss function is determined based on the first sub-loss, the second sub-loss, and the third sub-loss.

[0238] As an optional implementation, after acquiring the image to be tested containing the target object, the processor 800 is further configured to:

[0239] The size and pixel data range of the image to be tested are normalized to obtain a normalized image to be tested, which is then used to extract features from the normalized image to obtain a feature image.

[0240] Based on the same inventive concept, this embodiment of the invention also provides a target positioning device. Since this device is the same as the device in the method of this embodiment of the invention, and the principle of the device in solving the problem is similar to that of the method, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0241] like Figure 9 As shown, the device includes:

[0242] Image acquisition unit 900 is used to acquire the image to be tested containing the target object;

[0243] The key position detection unit 901 is used to extract features from the image to be tested to obtain a feature image, and determine the object position of the target object and the line segment position of the key line segment representing the pose of the target object based on the feature image.

[0244] The pose determination unit 902 is used to determine the pose of the target object based on the object position and the line segment position, and to locate the target object.

[0245] As an optional implementation, the key position detection unit 901 is specifically used for:

[0246] Feature extraction is performed on the image under test from different scale ranges to obtain features at multiple scales, wherein features at different scales are used to characterize different contents of the image.

[0247] Features from multiple scales are fused to obtain a feature image containing features from multiple scales.

[0248] As an optional implementation, the key position detection unit 901 specifically determines the line segment position of the key line segment representing the pose of the target object in the following manner:

[0249] Based on the feature image, each center point and the corresponding vector are determined. The center point represents the center of the key line segment, and the vector corresponding to the center point represents the vector from the center of the key line segment to the two endpoints.

[0250] Based on each center point and its corresponding vector, determine the position of the key line segment corresponding to each center point.

[0251] As an optional implementation method,

[0252] The object location and object category of the target object are determined based on the feature image;

[0253] When the object category is a preset category, the line segment position and line segment category of the key line segment representing the pose of the target object are determined according to the feature image, and the line segment category represents the position of the key line segment in the target object;

[0254] The pose of the target object is determined based on the object position, the line segment position of the key line segments, and the line segment type.

[0255] As an optional implementation, the key position detection unit 901 specifically determines the object position of the target object and the line segment position of the key line segment representing the pose of the target object in the following manner:

[0256] The image to be tested is input into the detection network, which outputs the object position and object category of the target object, as well as the line segment position and line segment category of the key line segment;

[0257] The detection network is trained using labeled sample images, where the sample images are labeled with the object location and object category of the target object, as well as the line segment location and line segment category of key line segments; the training process includes:

[0258] The labeled sample image is input into the initial detection network. The first loss function is determined based on the object position of the output target object and the object position of the labeled target object. The second loss function is determined based on the line segment position of the output key line segment and the line segment position of the labeled key line segment.

[0259] The loss function is determined based on the first loss function and the second loss function. The network parameters of the initial detection network are adjusted according to the loss function. When the loss function value is less than the first threshold or the number of training iterations reaches the second threshold, the training is determined to be complete, and the trained detection network is obtained.

[0260] As an optional implementation, the detection network includes a feature extraction network, an object detection network, and a line segment detection network;

[0261] The feature extraction network is used to extract features from the input image to be tested, and obtain a feature image;

[0262] The target detection network is used to detect targets in the feature image and output the object location and object category of the target object;

[0263] The line segment detection network is used to detect line segments in feature images and output the line segment positions and line segment categories of key line segments.

[0264] As an optional implementation, the feature extraction network includes a backbone network and a fusion network; the backbone network includes multiple sets of convolutional layers.

[0265] The backbone network is used to extract features from the image under test, obtaining multiple features, with one convolutional layer corresponding to one feature;

[0266] The fusion network is used to fuse multiple features to obtain a feature image.

[0267] As an optional implementation, the line segment detection network is used to detect line segments in the feature image and output each center point and the vector corresponding to each center point; the key position detection unit 901 is specifically used to determine the second loss function through the following steps:

[0268] For each center point, a first sub-loss is determined based on the output center point and the labeled center point; a second sub-loss is determined based on the angle of the vector corresponding to the center point and the angle of the labeled vector; and a third sub-loss is determined based on the distance between the vector corresponding to the center point and the distance between the labeled vector.

[0269] The second loss function is determined based on the first sub-loss, the second sub-loss, and the third sub-loss.

[0270] As an optional implementation, after acquiring the image to be tested containing the target object, the image acquisition unit 900 is further configured to:

[0271] The size and pixel data range of the image to be tested are normalized to obtain a normalized image to be tested, which is then used to extract features from the normalized image to obtain a feature image.

[0272] Based on the same inventive concept, this disclosure provides a computer storage medium comprising: computer program code, which, when executed on a computer, causes the computer to perform any of the target localization methods described above. Since the principle by which the computer storage medium solves the problem is similar to that of the target localization method, the implementation of the computer storage medium can be referred to the implementation of the method, and repeated details will not be elaborated further.

[0273] In specific implementation, computer storage media can include: Universal Serial Bus Flash Drive (USB), portable hard drive, Read-Only Memory (ROM), Random Access Memory (RAM), magnetic disk or optical disk, and other storage media that can store program code.

[0274] Based on the same inventive concept, this disclosure also provides a computer program product, which includes computer program code that, when executed on a computer, causes the computer to perform any of the target localization methods discussed above. Since the principle by which the above-described computer program product solves the problem is similar to that of the target localization method, the implementation of the above-described computer program product can be referred to the implementation of the method, and repeated details will not be elaborated further.

[0275] Computer program products may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0276] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0277] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 Devices that specify the functions in one or more boxes.

[0278] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction device, which is implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0279] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 Figure 1 The steps of the function specified in one or more boxes.

[0280] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A target localization method, characterized in that, The method includes: Obtain the image to be tested that contains the target object; Feature extraction is performed on the image to be tested to obtain a feature image. Based on the feature image, the object position of the target object and the line segment positions of key line segments representing the target object's pose are determined. The line segment positions of the key line segments representing the target object's pose are determined as follows: Based on the feature image, each center point and its corresponding vector are determined. The center point represents the center of the key line segment, and the vector corresponding to the center point represents the vector pointing from the center of the key line segment to its two endpoints. Based on each center point and its corresponding vector, the line segment positions of the key line segments corresponding to each center point are determined. The object position of the target object and the line segment positions of the key line segments representing the target object's pose are determined as follows: The image to be tested is input into a detection network, which outputs the object position and object category of the target object. The detection network is trained using labeled sample images, which are labeled with the object position and object category of the target object, as well as the line segment position and line segment category of the key line segments. The training process includes: inputting the labeled sample images into the initial detection network; determining a first loss function based on the object position of the output target object and the labeled object position; determining a second loss function based on the line segment position of the output key line segments and the labeled key line segments; determining a loss function based on the first loss function and the second loss function; adjusting the network parameters of the initial detection network based on the loss function; and determining that training is complete when the loss function value is less than a first threshold or the number of training iterations reaches a second threshold, thus obtaining a trained detection network. The pose of the target object is determined based on the object position and the line segment position, and is used to locate the target object.

2. The method according to claim 1, characterized in that, The feature image is obtained by performing feature extraction on the image to be tested, including: Feature extraction is performed on the image under test from different scale ranges to obtain features at multiple scales, wherein features at different scales are used to characterize different contents of the image. Features from multiple scales are fused to obtain a feature image containing features from multiple scales.

3. The method according to claim 1, characterized in that, The object location and object category of the target object are determined based on the feature image; When the object category is a preset category, the line segment position and line segment category of the key line segment representing the pose of the target object are determined according to the feature image, and the line segment category represents the position of the key line segment in the target object; The pose of the target object is determined based on the object position, the line segment position of the key line segments, and the line segment type.

4. The method according to claim 1, characterized in that, The detection network includes a feature extraction network, an object detection network, and a line segment detection network; The feature extraction network is used to extract features from the input image to be tested, and obtain a feature image; The target detection network is used to detect targets in the feature image and output the object location and object category of the target object; The line segment detection network is used to detect line segments in feature images and output the line segment positions and line segment categories of key line segments.

5. The method according to claim 4, characterized in that, The feature extraction network includes a backbone network and a fusion network; the backbone network includes multiple sets of convolutional layers. The backbone network is used to extract features from the image under test, obtaining multiple features, with one convolutional layer corresponding to one feature; The fusion network is used to fuse multiple features to obtain a feature image.

6. The method according to claim 4, characterized in that, The line segment detection network is used to detect line segments in the feature image, outputting each center point and its corresponding vector; the second loss function is determined through the following steps: For each center point, a first sub-loss is determined based on the output center point and the labeled center point; a second sub-loss is determined based on the angle of the vector corresponding to the center point and the angle of the labeled vector; and a third sub-loss is determined based on the distance between the vector corresponding to the center point and the distance between the labeled vector. The second loss function is determined based on the first sub-loss, the second sub-loss, and the third sub-loss.

7. The method according to any one of claims 1 to 3, characterized in that, After acquiring the image to be tested containing the target object, the process further includes: The size and pixel data range of the image to be tested are normalized to obtain a normalized image to be tested, which is then used to extract features from the normalized image to obtain a feature image.

8. An electronic device, characterized in that, The electronic device includes a processor and a memory for storing a program executable by the processor, and the processor for reading the program in the memory and executing the steps of the method according to any one of claims 1 to 7.

9. A computer storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images