A feature extraction method, apparatus, computer device, and storage medium

Abstract

The embodiment of the present application discloses a feature extraction method, device, computer equipment and storage medium. The embodiment of the present application can sample a preset neighborhood range from an image to be processed to obtain at least one sampling area, wherein the position points of the sampling area include a sampling reference point and multiple neighborhood points; calculate the relative relationship between the sampling reference point and each neighborhood point in the sampling area; calculate the weight vector of each position point in the sampling area based on the relative relationship; extract the feature vector of each position point in the sampling area; calculate the output vector of the sampling area according to the weight vector and feature vector of each position point in the sampling area; and generate the feature map corresponding to the image to be processed according to the output vector of each sampling area. The embodiment of the present application improves the accuracy and convenience of feature extraction.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a feature extraction method, device, computer equipment and storage medium. Background Art

[0002] With the continuous development of artificial intelligence, target detection technology has become more and more mature and widely used. Among them, target detection can be to find the target of interest from the image and determine the location of the target. Feature extraction is a relatively important step in the target detection process. At present, the existing feature extraction method generally extracts features from the image through convolution operations in order to predict the location of the target based on the features. However, since the existing feature extraction method only extracts features from the image based on a single convolution operation, it cannot extract the feature information of the target well, which reduces the accuracy of feature extraction, thereby reducing the accuracy of target detection. Summary of the invention

[0003] The embodiments of the present application provide a feature extraction method, apparatus, computer device and storage medium, which can improve the accuracy and convenience of target detection.

[0004] To solve the above technical problems, the present application provides the following technical solutions:

[0005] The present application provides a feature extraction method, including:

[0006] Sampling a preset neighborhood range from the image to be processed to obtain at least one sampling area, wherein the position points of the sampling area include a sampling reference point and a plurality of neighborhood points;

[0007] Calculate the relative relationship between the sampling reference point and each neighboring point in the sampling area;

[0008] Calculate the weight vector of each position point in the sampling area based on the relative relationship;

[0009] Extracting feature vectors of each position point in the sampling area;

[0010] Calculate the output vector of the sampling area according to the weight vector and the feature vector of each position point in the sampling area; and

[0011] A feature map corresponding to the image to be processed is generated according to the output vectors of each sampling area.

[0012] According to one aspect of the present application, a feature extraction device is also provided, including:

[0013] A sampling module, used for sampling a preset neighborhood range from the image to be processed to obtain at least one sampling area, wherein the position points of the sampling area include a sampling reference point and a plurality of neighborhood points;

[0014] A first calculation module, adapted to calculate the relative relationship between the sampling reference point and each neighboring point within the sampling area;

[0015] A second calculation module, used for calculating a weight vector of each position point in the sampling area based on the relative relationship;

[0016] An extraction module, used to extract the feature vector of each position point in the sampling area;

[0017] A third calculation module, configured to calculate an output vector of the sampling area according to the weight vector and the feature vector of each position point in the sampling area; and

[0018] A generation module is used to generate a feature map corresponding to the image to be processed according to the output vector of each sampling area.

[0019] According to one aspect of the present application, a computer device is also provided, including a processor and a memory, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, any feature extraction method provided in the embodiments of the present application is executed.

[0020] According to one aspect of the present application, a storage medium is also provided, wherein the storage medium is used to store a computer program, and the computer program is loaded by a processor to execute any feature extraction method provided in the embodiments of the present application.

[0021] The embodiments of the present application can sample a preset neighborhood range from an image to be processed to obtain multiple sampling areas, wherein the position points of the sampling areas include a sampling reference point and multiple neighborhood points; and calculate the relative relationship between the sampling reference point and each neighborhood point in the sampling area, and then calculate the weight vector of each position point in the sampling area based on the relative relationship, and extract the feature vector of each position point in the sampling area; calculate the output vector of the sampling area according to the weight vector and the feature vector of each position point in the sampling area; and generate a feature map corresponding to the image to be processed according to the output vector of each sampling area, thereby improving the accuracy and convenience of feature extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 is a schematic diagram of a scenario in which the feature extraction method provided in an embodiment of the present application is applied;

[0024] Figure 2 It is a flowchart of the feature extraction method provided in the embodiment of the present application;

[0025] Figure 3 is a schematic diagram of a depth image provided by an embodiment of the present application;

[0026] Figure 4 is a schematic diagram of depth image generation provided by an embodiment of the present application;

[0027] Figure 5 is a schematic diagram of extracting a feature map through a meta-convolution operator provided in an embodiment of the present application;

[0028] Figure 6 It is a schematic diagram of sampling area generation provided in an embodiment of the present application;

[0029] Figure 7 is a schematic diagram of sampling reference points and neighborhood points within a sampling area provided in an embodiment of the present application;

[0030] Figure 8 It is a schematic diagram of detecting an image to be processed to obtain multiple candidate frames and determining a target frame from the multiple candidate frames provided by an embodiment of the present application;

[0031] Fig. 9 It is a structural diagram of the target detection model training provided in the embodiment of the present application;

[0032] Fig.10 It is a schematic diagram of an unmanned vehicle automatically detecting and avoiding obstacles provided by an embodiment of the present application;

[0033] Fig.11 is a schematic diagram of a feature extraction device provided in an embodiment of the present application;

[0034] Fig.12 It is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0036] In the following description, the terms "first\second" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0037] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0038] The embodiments of the present application provide a feature extraction method, apparatus, computer equipment and storage medium.

[0039] The feature extraction method provided in the embodiment of the present application can be implemented by the terminal or the server alone, or can be implemented by the terminal and the server together. For example, the terminal alone undertakes the feature extraction method described below, or the terminal can send a feature extraction request for an image (including an image whose features are to be extracted) to the server, and the server can execute the feature extraction method according to the received feature extraction request, and send a feature map corresponding to the image to the terminal.

[0040] The computer device for feature extraction provided in the embodiments of the present application can be various types of terminal devices or servers, wherein the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms, but is not limited to this. The terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a camera, a wearable device, or a vehicle-mounted terminal, etc. The vehicle-mounted terminal can include an unmanned vehicle, but is not limited to this. The terminal and the server can be directly or indirectly connected by wired or wireless communication, and the embodiments of the present application are not limited here.

[0041] See also Figure 1 , Figure 1A schematic diagram of a scenario for applying the feature extraction method provided in an embodiment of the present application. The feature extraction method can be applied to a feature extraction device, which can be specifically integrated in a computer device such as a terminal or a server. For example, the computer device can sample a preset neighborhood range from an image to be processed to obtain at least one sampling area, and the position points of the sampling area include a sampling reference point and multiple neighborhood points; calculate the relative relationship between the sampling reference point and each neighborhood point in the sampling area, and calculate the weight vector of each position point in the sampling area based on the relative relationship; then, the feature vector of each position point in the sampling area can be extracted, and the output vector of the sampling area can be calculated according to the weight vector and feature vector of each position point in the sampling area; and generate a feature map corresponding to the image to be processed according to the output vector of each sampling area, thereby improving the accuracy and convenience of feature extraction.

[0042] It should be noted that Figure 1 The scenario diagram of the feature extraction method application shown is only an example. The feature extraction method application and scenario described in the embodiment of the present application are intended to more clearly illustrate the technical solution of the embodiment of the present application, and do not constitute a limitation on the technical solution provided in the embodiment of the present application. Ordinary technicians in this field can know that with the evolution of the application of feature extraction methods and the emergence of new business scenarios, the technical solution provided in the embodiment of the present application is also applicable to similar technical problems.

[0043] It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments.

[0044] In this embodiment, the description will be made from the perspective of a feature extraction device, which can be specifically integrated into a computer device such as a terminal or a server.

[0045] See also Figure 2 , Figure 2 : is a flow chart of a feature extraction method provided by an embodiment of the present application. The feature extraction method may include:

[0046] S101 . Sampling a preset neighborhood range from an image to be processed to obtain at least one sampling area, wherein the position points of the sampling area include a sampling reference point and a plurality of neighborhood points.

[0047] The specific type of the image to be processed can be flexibly set according to actual needs. For example, the image to be processed can be a depth image, which can be obtained in any way, and the present disclosure does not limit its acquisition method. Figure 3 As shown, a depth image containing depth information can be collected by a laser radar, and the collected depth image is used as an image to be processed. Figure 3In the figure, the depth image is displayed from different perspectives, or the pre-stored image to be processed can be obtained from the database. The image to be processed may include three-dimensional point cloud data, which may be data of the surrounding environment collected by a laser radar and represented by a set of sparse three-dimensional space points, that is, each valid point on the image to be processed corresponds to a point in the point cloud data, so the depth image obtained by the laser radar is a representation method of the laser point cloud. It should be understood that the image to be processed may also be an image output by an intermediate layer of the network, such as a feature map (intermediate feature map) output by an intermediate layer of the network.

[0048] The image to be processed can be a tensor of a×b×c, where a can represent the number of laser radar lines, b can represent the number of sampling times of the laser radar rotating one circle, and c can represent the number of channels of input data. The values ​​of a, b and c can be flexibly set according to actual needs. For example, Figure 4 As shown, the image to be processed can be a 64×2650×8 tensor, that is, a=64, b=2650, c=8, 8 can represent 8 channels of input data, and these 8 channels can include: depth, reflection intensity, elongation, x-coordinate, y-coordinate, z-coordinate, azimuth, and inclination, etc.

[0049] The image to be processed may include a target object to be detected, which may include a vehicle, a building, a person, a plant, and an animal, etc., and the specific content is not limited here. And each position point of the image to be processed has attribute information.

[0050] In one embodiment, the image to be processed is a depth image, the depth image has multiple position points, each position point has attribute information, and the attribute information includes at least one of depth information, spatial coordinates, elongation, reflection intensity, azimuth, pitch angle, size of the real frame, and number of points in the real frame. Of course, other attribute information may also be included, and this application does not limit it.

[0051] A preset neighborhood range can be sampled from the image to be processed to obtain at least one sampling area. The points in the sampling area are called position points. The position points include a sampling reference point and multiple neighborhood points of the sampling reference point. For example, a convolution operation can be performed on the image to be processed by a meta-convolution operator. Specifically, Figure 5 As shown, an area within a preset neighborhood range can be sampled from the image to be processed to obtain multiple image sampling areas to be processed, wherein the preset neighborhood range can be flexibly set according to actual needs, such as a square, rectangle or other arbitrary shape, and the present disclosure does not impose specific restrictions on its shape and size. For example, a 3×3 or 5×5 sampling area can be sampled, for example, Figure 6As shown, a plurality of sampling areas can be obtained by sliding a window from the upper left corner of the image to be processed to the lower right corner of the image to be processed.

[0052] S102, calculating the relative relationship between the sampling reference point and each neighboring point in the sampling area.

[0053] After obtaining the sampling area, the relative relationship between the sampling reference point and the neighboring points in the sampling area can be obtained. The sampling reference point can be any point in the sampling area, and the present disclosure does not limit this. Figure 7 As shown, the center point of the sampling area (i.e., the sampling center point) can be used as the sampling reference point, and the points around the sampling reference point can be used as neighborhood points. Alternatively, if there are two points in the center of the sampling area, any one of them can be used as the reference point, and the other points are neighborhood points. In one embodiment, the relative relationship includes at least one of the relative position relationship and relative feature relationship between the sampling reference point and each position point in the sampling area; the relative position relationship includes relative Cartesian coordinates; and the relative feature relationship includes the association relationship between feature vectors.

[0054] For example, Figure 5 As shown, the three-dimensional position coordinates (x i ,y i , z i ), and obtain the three-dimensional position coordinates (x j ,y j , z j ), according to the difference between the three-dimensional position coordinates of the sampling reference point and the three-dimensional position coordinates of the neighboring point, the relative Cartesian coordinates between the sampling reference point and the neighboring point are determined as (x i -x j ,y i -y j , z i -z j ), the relative Cartesian coordinate is (x i -x j ,y i -y j , z i -z j ) as the relative relationship between the sampling reference point and the neighborhood point. Or, if the feature vector of the sampling reference point is (a1, a2, ..., a n ), the eigenvector of a neighborhood point is (b1, b2, ..., b n ), the relative feature relationship can be the difference between the corresponding dimensions of two feature vectors, for example (a1-b1, a2-b2, ..., a n -b n ).

[0055] S103: Calculate the weight vector of each position point in the sampling area based on the relative relationship.

[0056] In one embodiment, the weight vector of each position point in the sampling area is calculated by a multi-layer perceptron.

[0057] In order to improve the accuracy and convenience of obtaining the weight vector, the weight vector can be calculated by a multilayer perceptron (MLP). In one embodiment, the meta-convolution operator includes a multilayer perceptron, and obtaining the weight vector of each position point in the sampling area based on the relative relationship can include: calculating the weight vector of each position point in the sampling area based on the relative relationship by the multilayer perceptron. For example, the weight vector of each position point in the sampling area can be calculated by the multilayer perceptron based on the relative Cartesian coordinates between the sampling reference point and the neighborhood point in the sampling area. When the corresponding weight vectors are calculated for multiple sampling areas, a weight matrix can be generated according to the weight vectors corresponding to the multiple sampling areas. The relative Cartesian coordinates of the sampling reference point and the neighborhood point in three-dimensional space are calculated by the meta-convolution operator, and the relative Cartesian coordinates are input into a globally shared multilayer perceptron, thereby obtaining the weight vector of each neighborhood point.

[0058] It should be noted that the relative relationship of the sampling reference point to all the position points in the sampling area is calculated here. For the sampling reference point in the sampling area, the relative Cartesian coordinates of the sampling reference point relative to itself can be obtained, and the weight vector of the sampling reference point can be calculated based on the relative Cartesian coordinates by the multi-layer perceptron, so the weight vector of each position point in the sampling area can be calculated. In addition to learning the weight vector through the relative Cartesian coordinates of the sampling reference point and the neighborhood point in the sampling area, the meta-convolution operator can also use more information to learn the weight vector, such as the depth difference between the sampling points or the feature vector of the sampling point.

[0059] S104: Extract feature vectors of each location point in the sampling area.

[0060] After obtaining the sampling area, the feature vector of each position point in the sampling area can be extracted. For example, the pixel features of each position point in the sampling area can be extracted as the feature vector. The feature vector can include feature information of 8 channels such as depth, reflection intensity, elongation, x-coordinate, y-coordinate, z-coordinate, azimuth, and pitch angle, and can also be the features of the middle layer of the network. It should be noted that the multi-layer perceptron of the meta-convolution operator can output a weight vector of the same length as the feature vector. Each sampling point in the sampling area has such a weight vector, so that the meta-convolution operator can only learn the weight vector of the same length as the feature vector, reducing the memory burden.

[0061] S105. Calculate the output vector of the sampling area according to the weight vector and feature vector of each position point in the sampling area.

[0062] In one embodiment, according to the weight vector and feature vector of each position point in the sampling area, calculating the output vector of the sampling area may include: performing vector operations on the weight vector and the corresponding feature vector of each position point in the sampling area to obtain an intermediate vector corresponding to each position point; performing a fusion operation on the obtained multiple intermediate vectors to obtain a fusion vector; performing a convolution operation on the fusion vector to obtain an output vector. The output vector can be considered as the output vector of the sampling reference point. In one embodiment, the fusion operation is a pooling operation or a sequential head-to-tail splicing operation.

[0063] Specifically, after obtaining the weight vector and the feature vector, a weighted product operation can be performed. For example, the weight vector and the feature vector of each position point in the sampling area can be subjected to a vector operation. The vector operation can be a vector multiplication (i.e., element product). j and the eigenvector f j Multiply them together to get the intermediate vector, such as Figure 5 O1 to O9 in the window. Since the multilayer perceptron through the meta-convolution operator can output a weight vector of the same length as the feature vector, each sampling point in the sampling area has such a weight vector. Therefore, the feature vector at each sampling point in the window area and its corresponding weight vector can be multiplied by corresponding elements to obtain the intermediate vector.

[0064] Each sampling area can correspond to multiple intermediate vectors, and then a product aggregation operation can be performed. Specifically, multiple intermediate vectors can be fused to obtain a fused vector. In one embodiment, the fusion operation is a splicing operation or a pooling operation. For example, multiple intermediate vectors can be spliced ​​together by a meta-convolution operator (for example, the head and tail can be spliced ​​in sequence). It should be noted that the product aggregation method can also be other methods, such as adding the output eigenvalues ​​(i.e., intermediate vectors) of all sampling points, or performing maximum pooling, etc.

[0065] Then the output vector can be obtained based on the fused vector, for example, Figure 5 As shown, the fused vector can be sent to a 1×1 convolution to obtain the final convolution result, and the convolution result is used as the output vector.

[0066] S106. Generate a feature map corresponding to the image to be processed according to the output vector of each sampling area.

[0067] Each sampling area can obtain a corresponding output vector, and a feature map corresponding to the image is determined based on the output vector. For example, multiple output vectors corresponding to multiple sampling areas can be used as feature maps corresponding to the image.

[0068] This embodiment can better utilize the three-dimensional spatial information (such as relative Cartesian coordinates) on the depth image through the meta-convolution operator. For example, adding the three-dimensional coordinate relationship in the Cartesian coordinate system between the sampling reference point and the neighborhood point (i.e., relative Cartesian coordinates) is conducive to better extraction of local geometric features. The meta-convolution operator replaces the learnable parameters with feature vectors embedded with the geometric relationship between points, which can not only better extract local geometric information, but also improve the model's perception of the shape and geometric structure of local objects. That is, the meta-convolution operator uses the relative spatial coordinates between points to provide a geometric prior for the weight learning of each point, and generates a separate weight vector for each point, and performs subsequent feature fusion between points, thereby enhancing the perception of local geometric feature structures under the traditional convolution paradigm. At the same time, the accuracy of three-dimensional target detection is significantly improved while increasing the amount of parameters that need to be learned very little, which brings a significant accuracy improvement to the subsequent three-dimensional target detection based on depth images, and the accuracy improvement for small objects such as pedestrians is particularly obvious.

[0069] It should be noted that the meta-convolution operator can extract the corresponding feature map from any image, which can be the original image input to the model, or the feature map (intermediate feature map) output after processing by the intermediate layer of the network. The original image can be a training set image, an image to be tested, etc. (such as a depth image to be processed, a sample depth image, etc.); it can be a depth image or other images. The present application does not impose specific restrictions on the input image of the meta-convolution operator. All schemes that use models to extract feature maps can use the meta-convolution operator, especially to extract feature maps from depth images. The generated feature maps can be used in a variety of scenarios, and the specific content is not limited here. Among them, the model can include target detection models, classification models, segmentation models, and completion models. Each model has a meta-convolution operator of the present application, so that subsequent operations can be performed after the feature map is extracted by the meta-convolution operator.

[0070] In one embodiment, after generating a feature map corresponding to the image to be processed, the feature extraction method also includes: performing at least one of the following operations on the image to be processed according to the feature map: target detection operation, semantic segmentation operation, instance segmentation operation, image classification operation, and missing information completion operation in the image.

[0071] Among them, the target detection operation can be based on the feature map to detect the position and type of the target object in the image, the semantic segmentation operation can be based on the feature map to identify the semantics contained in the image, the instance segmentation operation can be based on the feature map to segment and extract the target in the image, the image classification operation can be based on the feature map to determine the type of image, the missing information completion operation can be based on the feature map to complete the missing information in the image, for example, if part of the depth information in the depth image is missing, the missing depth information can be predicted, and of course other missing information can also be completed.

[0072] The following will explain in detail how to use the feature map to detect targets or train the model after obtaining the feature map corresponding to the image.

[0073] In one embodiment, after generating a feature map corresponding to the image to be processed, the feature extraction method also includes: detecting the feature map according to attribute information, generating multiple detection frames within a preset range and the overlap between each detection frame and the corresponding real frame; merging multiple detection frames at least according to the overlap of each detection frame to obtain a target frame corresponding to the image to be processed (i.e., the image to be processed).

[0074] For example, Figure 8 As shown, the target detection model can be used to obtain other models to detect feature maps according to attribute information, and obtain multiple detection frames and the classification confidence corresponding to each detection frame. The classification confidence can be the overlap between the detection frame and the corresponding real frame of the same category. The overlap can be the intersection and union ratio of the detection frame and the corresponding real frame of the same category. For example, different categories can include vehicles or people.

[0075] In one embodiment, a corresponding detection frame is generated for each position point of the image to be processed, and feature maps of different resolutions are used to generate detection frames within a preset attribute range. The preset attribute range includes any one or more of a preset depth range, a reflection intensity range, an azimuth angle range, a pitch angle range, a frame size range, a density range of points within the frame, and an elongation rate range of points within the frame.

[0076] In one embodiment, the classification confidence is the overlap between the detection frame and the corresponding real frame of the same category. For example, the classification confidence of a vehicle is the overlap between the detection frame of the vehicle and the real frame of the corresponding vehicle. For another example, the classification confidence of a pedestrian is the overlap between the detection frame of the pedestrian and the real frame of the corresponding pedestrian.

[0077] In one embodiment, the overlap is the intersection-and-union ratio of the detection frame and the corresponding real frame of the same category, and the intersection-and-union ratio is the ratio of the intersection area and the union area of ​​two frames. For example, VFL can be used to predict the intersection-and-union ratio. Compared with other tasks (such as traditional 2D target detection), VFL predicts the intersection-and-union ratio. The effect of VFL on this task is very good. The reason is that: this task can be a dense prediction (each point will generate a frame), and there will be a large number of low regression quality frames (low intersection-and-union ratio). There are many high-scoring frames in these frames. In the process of non-maximum suppression, high-quality frames are easily eliminated by these low-quality (but high-scoring, which is the classification confidence) frames. Therefore, VFL is used to predict an intersection-and-union ratio as the classification confidence. In this way, the regression quality of the frame and its score are more consistent, and high-quality frames are more likely to be retained in non-maximum suppression.

[0078] In one embodiment, the classification confidence includes the degree of overlap, and multiple detection frames are merged according to the classification confidence of each detection frame to obtain a target frame corresponding to the image to be processed, including: selecting detection frames whose overlap is greater than a first threshold from multiple detection frames to obtain a first candidate frame; selecting a candidate frame with the highest overlap from the first candidate frame to obtain a second candidate frame; selecting a candidate frame whose overlap with the second candidate frame is greater than a second threshold from the first candidate frame to obtain a third candidate frame; performing weighted processing on the second candidate frame and the third candidate frame to obtain a new generated frame; removing the second candidate frame and the third candidate frame from the first candidate frame, and returning to perform the operation of selecting the candidate frame with the highest overlap from the first candidate frame, until all the candidate frames contained in the first candidate frame are processed, at which time all the new generated frames are the target frames corresponding to the image to be processed.

[0079] Weighted non-maximum suppression can be used to remove redundant frames and obtain target frames with better quality for the generated multiple detection frames, wherein weighted non-maximum suppression further improves the quality of the detected candidate frames. Since in the process of generating multiple detection frames, a detection frame can be predicted on each foreground point in the image to be processed, a large number of detection frames will be generated, and the error between the detection frames can be a zero-mean distribution, so a large number of detection frames can be weighted averaged according to their weight coefficients to generate a target frame with higher quality. Specifically, first, a detection frame with a degree of overlap (i.e., classification confidence) greater than a first threshold can be screened out from multiple detection frames to obtain a first candidate frame, wherein the first threshold can be flexibly set according to actual needs, and the specific value is not limited here. If the first threshold of the detection frame is greater than the first threshold, it means that the overlap of the detection frame is high, so the candidate frame with a degree of overlap greater than the first threshold can be screened out from the multiple detection frames as a subsequent detection operation, and the candidate frame with a degree of overlap less than or equal to the first threshold is removed. Then, the candidate frame with the highest degree of overlap can be screened out from the detection frame with a degree of overlap greater than the first threshold, and the detection frame with the highest degree of overlap is used as the second candidate frame. The second candidate frame may include one or more frames, and the overlap degree may be an intersection-and-union ratio. For example, a second candidate frame having an intersection-and-union ratio with the first candidate frame greater than an intersection-and-union ratio greater than 0.5 may be screened out.

[0080] Then, the candidate frames whose overlap with the second candidate frame is greater than the second threshold can be screened out from the first candidate frame to obtain the third candidate frame, wherein the second threshold can be flexibly set according to actual needs, and the specific value is not limited here. The second candidate frame and the third candidate frame are weighted to obtain a new generated frame, and the second candidate frame and the third candidate frame in the first candidate frame are eliminated, and the operation of screening out the candidate frame with the highest overlap from the first candidate frame is returned until all the candidate frames included in the first candidate frame are processed. At this time, all the new generated frames are the target frames corresponding to the image to be processed, and the target frame can be the area where the target object to be detected is located, and can also include the category to which the target object belongs.

[0081] It should be noted that for weighted non-maximum suppression (WNMS): the key point is that WNMS can fuse multiple detection frames to obtain a frame with better quality. The reasons why WNMS works better in this embodiment may include: (1) dense prediction (i.e., each point generates a frame) will generate a large number of detection frames, and fusing multiple detection frames can greatly reduce the error. (2) The error of the output of the target detection model relative to the true value is usually a zero-mean distribution, which provides a theoretical basis for using weighted averaging to reduce the error.

[0082] In one embodiment, weighted processing of the second candidate box and the third candidate box may include: determining the weight coefficient of each first candidate box, and performing weighted processing according to the weight coefficient of the second candidate box and the third candidate box. In one embodiment, the weight coefficient is determined according to at least one of the following parameters: classification confidence, regression confidence, distance relationship between boxes, feature relationship between boxes. Weighted processing refers to weighting the coordinate parameters of multiple boxes. Assuming that the coordinate parameters are expressed as center point coordinates and length, width and height, for example (x, y, z, w, h, l), the coordinate parameter value of the newly generated box is obtained by weighting each x, y, z, w, h, l parameter of the weighted box. Of course, the coordinate parameters can also be expressed in other ways, such as using multiple vertex coordinates, and this application is not limited to this. For example, the classification confidence of the second candidate box can be used as the weight coefficient of the second candidate box, and the classification confidence of the third candidate box can be used as the weight coefficient of the third candidate box. The center position coordinates and width and height of the newly generated box = the weight coefficient of the second candidate box * the center position coordinates and width and height of the second candidate box + the weight coefficient of the third candidate box * the center position coordinates and width and height of the third candidate box.

[0083] For another example, through weighted non-maximum suppression, the second candidate box (also called a high-score box) and the third candidate box (for example, a box with an intersection-over-union ratio greater than 0.5) and their weight coefficients can be weighted averaged to obtain a new higher-quality target box. The process can be expressed as follows:

[0084]

[0085] Among them, b0 can represent the second candidate box, b k It can represent the third candidate box, Can represent the newly generated box in the image to be processed, s k It can represent the weight coefficient of the kth box, Can represent indicator functions.

[0086] This embodiment can obtain an image to be processed, each position point of the image to be processed has attribute information, and perform feature extraction on the image to be processed to obtain feature maps of different resolutions; then the feature map can be detected according to the attribute information, and multiple detection frames within a preset attribute interval corresponding to each feature map and the classification confidence of each detection frame are generated. At this time, the multiple detection frames can be merged to obtain a target frame corresponding to the depth image to be detected, thereby achieving efficient and accurate detection of the target frame in the depth image to be detected, and eliminating the need for multi-perspective feature fusion, thereby improving the convenience and accuracy of target detection.

[0087] In one embodiment, after generating a feature map corresponding to the image, the feature extraction method also includes: downsampling the feature map according to a preset sampling step size to obtain a plurality of first sampling feature maps of different resolutions; upsampling the plurality of first sampling feature maps of different resolutions to the original resolution of the image to be processed, to obtain a plurality of second sampling feature maps of different resolutions.

[0088] After obtaining the feature map, the feature map can be downsampled according to the preset sampling step size to obtain multiple first sampling feature maps of different resolutions, wherein the preset sampling step size can be flexibly set according to actual needs. For example, the backbone network of the model (also referred to as the backbone network) is composed of multiple stages, each stage will perform upsampling or downsampling, and each stage of the backbone network can be composed of multiple residual networks, that is, the residual network in the backbone network can be used to perform upsampling or downsampling operations. For example, the feature map can be downsampled to 1 / 16 in succession according to a sampling step size of 2 through the backbone network of the initial target detection model (for example, downsampled according to step sizes Stride1, Stride2, Stride4, Stride8 and Stride16), and 5 first sampling feature maps of different resolutions are obtained: Res1, Res2, Res3, Res4 and Res5. Res is a different residual block. Then, the first sampling feature maps of multiple different resolutions can be upsampled to the original resolution of the image to be processed to obtain multiple second sampling feature maps, and the second sampling feature maps of different resolutions are feature maps of different resolutions. It should be noted that for the depth image acquired by the lidar, such as a 64×2650×8 tensor, since the height of the depth image is only 64, upsampling or downsampling can be performed only in the width direction.

[0089] In one embodiment, the image to be processed is a sample image in a training set, and the sample image has a pre-labeled real frame, and the real frame is assigned to a second sampling feature map of different resolutions according to its attribute information; wherein each second sampling feature map is used to generate a detection frame of a preset attribute interval, and a predicted overlap between each detection frame and a corresponding real frame of the same category.

[0090] The sample image may include attribute information of each position point, a true frame, and true classification confidence of the true frame, etc., wherein the true frame may be the area where the target object is located in the sample image, and may also include the category to which the target object belongs. The shape and size of the true frame may be flexibly set according to actual needs, for example, the true frame may be a rectangular area of ​​a quadrilateral. The attribute information may include at least one of depth information, spatial coordinate value, elongation, reflection intensity, azimuth, pitch angle, size of the true frame, and number of points in the true frame.

[0091] In one embodiment, feature maps of different resolutions are annotated with real frames, and the real frames in the feature maps are mapped in the following manner: determining the attribute interval where the attribute information of each real frame is located; obtaining the mapping relationship between different attribute intervals and different feature maps; and allocating each real frame to the corresponding feature map according to the mapping relationship.

[0092] For example, the attribute information is depth information, and is described in detail as follows. Fig. 9 As shown, target detection label assignment can be performed based on a feature pyramid of depth, that is, at this time, the real frame can be assigned to the corresponding feature map of different resolutions based on the depth information, so that it can be assigned to the feature map of different resolutions according to the depth information of the target object for prediction. Among them, assigning the real frame to the corresponding feature map can be to use the coordinates of the real frame as the label of the feature map. Specifically, in order to improve the convenience of mapping the real frame in the feature map, the target depth interval where the depth information of each position point in the sample image is located can be determined, and the target depth interval where the position point is located can be determined according to the depth information of the frame where the position point is located. For example, when the depth information of the frame where the position point A is located is 6m, the target depth interval where it is located is [0m, 15m), and the mapping relationship between the pre-set different depth intervals and different feature maps is obtained, and the mapping relationship can be flexibly set according to actual needs. Then, according to the mapping relationship between different depth intervals and different feature maps, the feature map matching the target depth interval can be determined as the feature map, and the real frame can be assigned to the feature map so as to predict through the feature map with the real frame label.

[0093] Due to the nature of perspective projection, the target object (such as an object) appears larger near and smaller far away in the depth image collected by the LiDAR. This algorithm not only takes into account the change in scale, but also considers that point clouds at different distances have different physical properties, such as density and reflection intensity. Therefore, the target object can be assigned to feature maps of different resolutions for training and prediction according to its depth information. For example, the target objects at [0m, 15m), [15m, 30m), and [30m, +inf) can be divided into feature maps obtained by sampling with a step size of 4, 2, and 1 for prediction. The depth-based feature pyramid alleviates the scale change problem in the depth image. In addition to alleviating the scale change problem, it also considers that points at different depths have different physical properties, thereby improving the effect and reliability of subsequent prediction of sample images based on feature maps.

[0094] It should be noted that objects of different depth ranges can be assigned to feature maps of different levels through Pyramid. But in fact, it can be classified according to many characteristics, such as the size of the object, reflectivity, etc. For example, in 2D detection, it can be assigned according to the size of the real detection box. The reason why this embodiment can be preferably allocated based on depth information is that the depth information can not only reflect the size of the object, but also reflect the different physical properties of the point cloud data at different distances. For example, the reflectivity generally decreases with the increase of depth, and the point cloud density also gradually decreases.

[0095] After obtaining the feature map, the feature map can be downsampled according to the preset sampling step size to obtain multiple first sampling feature maps of different resolutions, wherein the preset sampling step size can be flexibly set according to actual needs. For example, the backbone network of the model is composed of multiple stages, each stage will perform upsampling or downsampling, and each stage of the backbone network can be composed of multiple basic blocks (BasicBlock) in the residual network, that is, the residual network in the backbone network can be used to perform upsampling or downsampling operations. First, the feature map can be downsampled to 1 / 16 (that is, downsampled according to Stride1, Stride2, Stride4, Stride8 and Stride16) with a sampling step size of 2 through the backbone network of the initial target detection model to obtain 5 first sampling feature maps of different resolutions: Res1, Res2, Res3, Res4 and Res5. Then, the multiple first sampling feature maps of different resolutions can be upsampled to the original resolution of the sample image to obtain multiple second sampling feature maps, such as Fig. 9 In the example, five second sampling feature maps can be obtained: Agg1, Agg2, Agg3, Agg4, and Agg5. Agg is a different aggregation block. It should be noted that for the depth image acquired by the lidar, such as a 64×2650×8 tensor, since the height of the depth image is only 64, upsampling or downsampling can be performed only in the width direction.

[0096] In one embodiment, multiple prediction boxes (the prediction boxes may also be referred to as detection boxes) corresponding to each second sampling feature map are generated, as well as the predicted overlap between each prediction box and the corresponding real box of the same category; the initial target detection model is trained based on the predicted overlap and the corresponding real overlap to obtain a trained target detection model.

[0097] After obtaining the sample feature map corresponding to the sample image, the sample image can be predicted based on the sample feature map through the initial target detection model to obtain a detection frame (i.e., a prediction frame) and the prediction classification confidence of each detection frame, wherein the obtained detection frame may include multiple detection frames, which can be either a two-dimensional frame or a three-dimensional frame.

[0098] In one embodiment, the predicted classification confidence is the predicted overlap between the detection box and the corresponding real box of the same category, and the real classification confidence is the real overlap between the detection box and the corresponding real box of the same category.

[0099] In one implementation, the predicted overlap is the predicted intersection-over-union ratio between the detection frame and the corresponding real frame of the same category, and the real overlap is the real intersection-over-union ratio between the detection frame and the corresponding real frame of the same category.

[0100] For example, the intersection-and-union ratio between each generated detection frame and its corresponding real frame can be directly predicted as the predicted overlap, and the confidence of the predicted classification can be determined based on the predicted overlap. For example, the predicted overlap can be used as the confidence of the predicted classification. That is, the intersection-and-union ratio of the current detection frame and its corresponding real frame is predicted on the regression branch, and the predicted value of the intersection-and-union ratio is used as the predicted classification confidence. Since the intersection-and-union ratio is a physical quantity that directly reflects the regression accuracy, the prediction of the intersection-and-union ratio can make the classification accuracy as consistent as possible with the regression accuracy.

[0101] In one embodiment, the method for generating the predicted classification confidence of each detection frame may include: obtaining the intersection area and the union area between the detection frame and the real frame, calculating the ratio between the intersection area and the union area, and obtaining the intersection-union ratio between the detection frame and the real frame; determining the predicted overlap between the detection frame and the real frame based on the intersection-union ratio, and setting the predicted overlap as the predicted classification confidence corresponding to the detection frame.

[0102] Specifically, the intersection area A∩B and the union area A∪B between the detection box A and the true box B can be obtained, and the ratio between the intersection area and the union area can be calculated to obtain the intersection-union ratio IOU between the detection box and the true box = (A∩B) / (A∪B).

[0103] At this time, the predicted overlap between the detection box and the true box can be determined based on the intersection-and-union ratio. For example, the intersection-and-union ratio can be used as the predicted overlap between the detection box and the true box, or the intersection-and-union ratio can be weighted to obtain the predicted overlap, and the predicted overlap can be set as the predicted classification confidence corresponding to the detection box.

[0104] It should be noted that a core problem in the detection task is the inconsistency between regression quality and classification quality. This problem also exists in 3D detection based on depth images collected by LiDAR. This inconsistency will cause high-precision boxes to be eliminated due to low confidence in the subsequent non-maximum suppression. Therefore, the intersection-over-union ratio of the current detection box and its corresponding real box can be predicted on the regression branch, and the predicted value of the intersection-over-union ratio can be used as the predicted classification confidence. Since the intersection-over-union ratio is a physical quantity that directly reflects the regression accuracy, the prediction of the intersection-over-union ratio can make the classification accuracy as consistent as possible with the regression accuracy. The intersection-over-union ratio prediction alleviates the problem of inconsistency between the confidence of the detection box and its regression quality.

[0105] Calculate the true classification confidence between the detection frame and the corresponding real frame of the same category. For example, you can calculate the true classification confidence between the detection frame of a vehicle and the real frame of the corresponding vehicle type through manual annotation or other models, or calculate the true classification confidence between the detection frame of a pedestrian and the real frame of the corresponding pedestrian type.

[0106] After obtaining the predicted classification confidence and the true classification confidence, the initial target detection model can be trained based on the predicted classification confidence and the true classification confidence. For example, the initial target detection model can be iteratively trained according to the predicted classification confidence of the detection box corresponding to multiple sample images and the true classification confidence of the true box until the predicted classification confidence and the true classification confidence converge, so that the parameters of the initial detection model are adjusted to appropriate values ​​to obtain the target detection model.

[0107] In one embodiment, training an initial target detection model based on predicted overlap and actual overlap to obtain a target detection model may include: calculating a classification loss value between the predicted overlap and the actual overlap through a classification loss function; calculating a regression loss value between a predicted box and a real box through a regression loss function; determining a total loss value based on the classification loss value and the regression loss value; and training the initial target detection model based on the total loss value to obtain a trained target detection model.

[0108] For example, the classification loss value L between the predicted classification confidence and the true classification confidence can be calculated by the classification loss function iou :

[0109]

[0110] The prediction loss of the intersection-over-union ratio can be used as the classification loss, where q can represent the true intersection-over-union ratio (i.e., the true classification confidence), p can represent the predicted intersection-over-union ratio (i.e., the predicted classification confidence), and the specific values ​​of α and γ can be flexibly set according to actual needs and are not limited here.

[0111] And, calculating the regression loss value between the detection box and the real box through the regression loss function. In one embodiment, calculating the regression loss value between the detection box and the real box through the regression loss function may include: obtaining the area parameters and position parameters of the detection box; calculating the regression loss value between the detection box and the real box based on the area parameters, position parameters and the real box through the regression loss function.

[0112] The region parameters may include the center coordinates of the detection box and the length, width, and height of the detection box, and the position parameters may include the sine and cosine values ​​of the detection box orientation. For example, SmoothL1 may be used as the regression loss function:

[0113]

[0114] Among them, L reg It can represent the regression loss value, q1 can represent the detection box, p1 can represent the real box, Q contains the center coordinates, length, width, height, and sine and cosine values ​​of the detection box, etc. The regression target is the offset from the current point to its real box, including the length, width, height and orientation of the real box.

[0115] After obtaining the classification loss value and the regression loss value, the total loss value can be determined based on the classification loss value and the regression loss value.

[0116] In one embodiment, determining the total loss value based on the classification loss value and the regression loss value may include: adding the classification loss value and the regression loss value to obtain the total loss value; or determining a first weight value corresponding to the classification loss value and a second weight value corresponding to the regression loss value, and adding the product between the classification loss value and the first weight value and the product between the regression loss value and the second weight value to obtain the total loss value.

[0117] For example, total loss value = classification loss value L iou + regression loss value L reg , or, total loss value = first weight value * classification loss value L iou + second weight value * regression loss value L reg , wherein the specific values ​​of the first weight value and the second weight value can be flexibly set according to actual needs.

[0118] At this time, the initial model (initial target detection model) can be trained according to the total loss value, and the initial model can be iteratively trained according to the total loss values ​​corresponding to multiple sample images until the total loss value converges, so that the parameters of the initial model are adjusted to appropriate values, and the target model is obtained, so that the image can be detected through the target model.

[0119] It should be noted that in addition to training the model according to the regression branch and the classification branch in the above manner, the regression confidence can also be predicted through the regression branch of the model, and the initial model can be trained according to the predicted regression confidence and the true regression confidence to obtain the trained target model.

[0120] The embodiments of the present application can sample a preset neighborhood range from an image to be processed to obtain at least one sampling area, wherein the position points of the sampling area include a sampling reference point and multiple neighborhood points; and calculate the relative relationship between the sampling reference point and each neighborhood point in the sampling area, and then calculate the weight vector of each position point in the sampling area based on the relative relationship, and extract the feature vector of each position point in the sampling area; calculate the output vector of the sampling area according to the weight vector and the feature vector of each position point in the sampling area; and generate a feature map corresponding to the image to be processed according to the output vector of each sampling area, thereby improving the accuracy and convenience of feature extraction.

[0121] According to the method described in the above embodiment, the following will describe an exemplary application of the embodiment of the present application in an actual application scenario, for example, an application scenario of an unmanned vehicle. This embodiment is described in detail by taking the integration of a target detection device and a target detection model training device in an unmanned vehicle as an example, wherein the unmanned vehicle may be provided with a laser radar for collecting depth images, and a target detection model, etc.

[0122] 1. Model training.

[0123] First, the unmanned vehicle can train the initial target detection model according to the target detection model training method to obtain the target detection model. For example, the unmanned vehicle can collect multiple depth images through a laser radar, and mark the real boxes of the two-dimensional or three-dimensional space where the target objects such as people, vehicles, buildings, plants, and animals are located on the depth images containing the depth information of each position point, as well as the real classification confidence of the real boxes, and use the multiple depth images as images to be processed.

[0124] Then, the unmanned vehicle can perform operations such as sampling, weight acquisition, weighted product, and product aggregation on the image to be processed through the meta-convolution operator of the initial target detection model to extract the sample feature map corresponding to the image to be processed. For example, the meta-convolution operator can be used to sample an area within a preset neighborhood from the image to be processed to obtain the sampling area, and the relative Cartesian coordinates between the sampling reference point and the neighborhood point in the sampling area can be obtained. The multi-layer perceptron of the meta-convolution operator can be used to obtain the sample weight vector of each sampling point in the sampling area according to the relative Cartesian coordinates, and the sample feature vector of each sampling point in the sampling area can be extracted. At this time, the sample feature map corresponding to the image to be processed can be obtained according to the sample weight vector and the sample feature vector of each sampling point in the sampling area.

[0125] Secondly, the unmanned vehicle can downsample the sample feature map according to a preset sampling step size to obtain multiple first sampling feature maps with different resolutions, and upsample the multiple first sampling feature maps with different resolutions to the original resolution of the image to be processed to obtain multiple second sampling feature maps; based on the depth information of each position point in the image to be processed, the real frame is assigned to the corresponding second sampling feature map, and the second sampling feature map after the real frame is assigned is used as the target feature map corresponding to the image to be processed, so that it can be assigned to feature maps of different resolutions for prediction according to the depth information of the target object.

[0126] At this point, the unmanned vehicle can predict the image to be processed based on the target feature map through the initial target detection model to obtain at least one two-dimensional or three-dimensional detection frame. The predicted intersection-over-union ratio between the detection frame and the true frame is obtained, and the predicted intersection-over-union ratio is used as the predicted classification confidence corresponding to the detection frame.

[0127] Finally, the unmanned vehicle can train the initial target detection model based on the predicted classification confidence and the true classification confidence to obtain the target detection model. For example, the classification loss value between the predicted classification confidence and the true classification confidence (i.e., the classification loss value between the predicted intersection and union ratio and the true intersection and union ratio) can be calculated through the classification loss function, and the regression loss value between the detection box and the true box can be calculated through the regression loss function. The total loss value is determined according to the classification loss value and the regression loss value. The initial target detection model is iteratively trained according to the total loss value corresponding to the multiple images to be processed until the total loss value converges, so that the parameters of the initial detection model are adjusted to appropriate values, and the target detection model is obtained. The target detection model is used to detect the depth image to be detected, thereby improving the accuracy and reliability of the target detection model training.

[0128] 2. Application of the model.

[0129] Unmanned vehicles can collect depth images through lidar as depth images to be detected, extract feature maps corresponding to the depth images to be detected through the meta-convolution operator of the target detection model, detect the depth images to be detected based on the feature maps, and obtain multiple two-dimensional or three-dimensional candidate boxes and the classification confidence corresponding to each candidate box.

[0130] Then, the unmanned vehicle can use the weighted non-maximum suppression algorithm to filter out detection frames with a degree of overlap greater than a first threshold from multiple detection frames to obtain a first candidate frame; filter out the candidate frame with the highest degree of overlap from the first candidate frame to obtain a second candidate frame, and filter out the candidate frame with a degree of overlap between the first candidate frame and the second candidate frame greater than a second threshold to obtain a third candidate frame; perform weighted processing on the second candidate frame and the third candidate frame to obtain a new generated frame; remove the second candidate frame and the third candidate frame from the first candidate frame, and return to execute the operation of filtering out the candidate frame with the highest degree of overlap from the first candidate frame, until all the candidate frames contained in the first candidate frame are processed, at which time all the new generated frames are the target frames corresponding to the depth image to be detected, and the target frame is the location of the detected target object, thereby improving the accuracy and convenience of target detection.

[0131] For example, Fig.10 As shown, the unmanned vehicle can quickly and accurately detect obstacles (i.e., target objects) around the driving section based on the collected depth images, and avoid obstacles in a timely manner. In the autonomous driving scenario, the unmanned vehicle can obtain depth images through lidar, and directly and efficiently detect the precise position of objects in the environment from the depth images, and realize the unmanned vehicle's large-scale perception of the environment, thereby improving the safety of autonomous driving.

[0132] In order to better implement the feature extraction method provided in the embodiment of the present application, the embodiment of the present application also provides a device based on the above feature extraction method. The meanings of the terms are the same as those in the above feature extraction method, and the specific implementation details can refer to the description in the method embodiment.

[0133] See also Fig.11 , Fig.11 A structural schematic diagram of a feature extraction device 300 provided in an embodiment of the present application, wherein the feature extraction device 300 may include a sampling module 301, a first calculation module 302, a second calculation module 303, an extraction module 304, a third calculation module 305, and a generation module 306, etc.

[0134] The sampling module 301 is used to sample a preset neighborhood range from the image to be processed to obtain at least one sampling area, and the position points of the sampling area include a sampling reference point and a plurality of neighborhood points.

[0135] The first calculation module 302 is adapted to calculate the relative relationship between the sampling reference point and each neighboring point in the sampling area.

[0136] The second calculation module 303 is used to calculate the weight vector of each position point in the sampling area based on the relative relationship.

[0137] The extraction module 304 is used to extract the feature vector of each position point in the sampling area.

[0138] The third calculation module 305 is used to calculate the output vector of the sampling area according to the weight vector and the feature vector of each position point in the sampling area.

[0139] The generating module 306 is used to generate a feature map corresponding to the image to be processed according to the output vector of each sampling area.

[0140] In one embodiment, the third calculation module 305 can be specifically used to: perform vector operations on the weight vectors and corresponding feature vectors of each position point in the sampling area to obtain an intermediate vector corresponding to each position point; perform a fusion operation on the obtained multiple intermediate vectors to obtain a fusion vector; and perform a convolution operation on the fusion vector to obtain an output vector.

[0141] In one embodiment, the fusion operation is a pooling operation or a sequential head-to-tail concatenation operation.

[0142] In one embodiment, the relative relationship includes at least one of a relative position relationship and a relative feature relationship between the sampling reference point and each position point in the sampling area; the relative position relationship includes relative Cartesian coordinates; and the relative feature relationship includes an association relationship between feature vectors.

[0143] In one embodiment, the weight vector of each position point in the sampling area is calculated by a multi-layer perceptron.

[0144] In one embodiment, the image to be processed is a depth image, which has multiple position points, each position point has attribute information, and the attribute information includes at least one of depth information, spatial coordinates, elongation, reflection intensity, azimuth, pitch angle, size of the real frame, and the number of points in the real frame.

[0145] In one implementation, the feature extraction device 300 may further include:

[0146] The processing module (not shown in the figure) is used to perform at least one of the following operations on the image to be processed according to the feature map: target detection operation, semantic segmentation operation, instance segmentation operation, image classification operation, and completion operation of missing information in the image.

[0147] In one embodiment, the processing module can be specifically used to: detect the feature map according to the attribute information, generate multiple detection frames within a preset range and the overlap between each detection frame and the corresponding real frame; merge the multiple detection frames at least according to the overlap of each detection frame to obtain the target frame corresponding to the image to be processed.

[0148] In one implementation, the feature extraction device 300 may further include:

[0149] A sampling module (not shown in the figure) is used to downsample the feature map according to a preset sampling step size to obtain a plurality of first sampling feature maps of different resolutions; and upsample the plurality of first sampling feature maps of different resolutions to the original resolution of the image to be processed to obtain a plurality of second sampling feature maps of different resolutions.

[0150] In one embodiment, the image to be processed is a sample image in a training set, and the sample image has a pre-labeled real frame, and the real frame is assigned to a second sampling feature map of different resolutions according to its attribute information; wherein each second sampling feature map is used to generate a detection frame of a preset attribute interval, and a predicted overlap between each detection frame and a corresponding real frame of the same category.

[0151] In one implementation, the feature extraction device 300 may further include:

[0152] A training module (not shown in the figure) is used to generate multiple prediction boxes corresponding to each second sampling feature map, as well as the predicted overlap between each prediction box and the corresponding real box of the same category; the initial target detection model is trained based on the predicted overlap and the corresponding real overlap to obtain a trained target detection model.

[0153] In one embodiment, the training module can be specifically used to: calculate the classification loss value between the predicted overlap and the actual overlap through a classification loss function; calculate the regression loss value between the predicted box and the actual box through a regression loss function; determine the total loss value based on the classification loss value and the regression loss value; train the initial target detection model based on the total loss value to obtain a trained target detection model.

[0154] In the embodiment of the present application, the sampling module 301 can sample a preset neighborhood range from the image to be processed to obtain at least one sampling area, and the position points of the sampling area include a sampling reference point and multiple neighborhood points; and the first calculation module 302 calculates the relative relationship between the sampling reference point and each neighborhood point in the sampling area, and then the second calculation module 303 can calculate the weight vector of each position point in the sampling area based on the relative relationship, and the extraction module 304 extracts the feature vector of each position point in the sampling area; the third calculation module 305 calculates the output vector of the sampling area according to the weight vector and feature vector of each position point in the sampling area; and the generation module 306 generates a feature map corresponding to the image to be processed according to the output vector of each sampling area, thereby improving the accuracy and convenience of feature extraction.

[0155] The present application also provides a computer device, which may be a terminal or a server, etc. Fig.12 As shown, it shows a schematic diagram of the structure of the computer device involved in the embodiment of the present application, specifically:

[0156] The computer device may include one or more processing core processors 501, one or more computer-readable storage media memories 502, a power supply 503, an input unit 504 and other components. Those skilled in the art will appreciate that Fig.12 The computer device structure shown in the figure does not constitute a limitation on the computer device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. Among them:

[0157] The processor 501 is the control center of the computer device. It uses various interfaces and lines to connect various parts of the entire computer device. By running or executing software programs and / or modules stored in the memory 502 and calling data stored in the memory 502, it executes various functions of the computer device and processes data, thereby monitoring the computer device as a whole. Optionally, the processor 501 may include one or more processing cores; preferably, the processor 501 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 501.

[0158] The memory 502 can be used to store software programs and modules. The processor 501 executes various functional applications and data processing by running the software programs and modules stored in the memory 502. The memory 502 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 502 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 502 may also include a memory controller to provide the processor 501 with access to the memory 502.

[0159] The computer device also includes a power supply 503 for supplying power to various components. Preferably, the power supply 503 can be logically connected to the processor 501 through a power management system, so as to manage charging, discharging, power consumption and other functions through the power management system. The power supply 503 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators and other arbitrary components.

[0160] The computer device may further include an input unit 504, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0161] Although not shown, the computer device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 501 in the computer device will load the executable files corresponding to the processes of one or more application programs into the memory 502 according to the following instructions, and the processor 501 will run the application programs stored in the memory 502, thereby realizing various functions, as follows:

[0162] A preset neighborhood range is sampled from an image to be processed to obtain at least one sampling area, wherein the position points of the sampling area include a sampling reference point and a plurality of neighborhood points; a relative relationship between the sampling reference point and each neighborhood point in the sampling area is calculated; a weight vector of each position point in the sampling area is calculated based on the relative relationship; a feature vector of each position point in the sampling area is extracted; an output vector of the sampling area is calculated based on the weight vector and the feature vector of each position point in the sampling area; and a feature map corresponding to the image to be processed is generated based on the output vector of each sampling area.

[0163] In the above embodiments, the description of each embodiment has its own focus. For the part that is not described in detail in a certain embodiment, please refer to the detailed description of the feature extraction method above, which will not be repeated here.

[0164] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in various optional implementations in the above-mentioned embodiments.

[0165] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments can be completed by computer instructions, or by controlling related hardware through computer instructions, and the computer instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. To this end, an embodiment of the present application provides a storage medium in which a computer program is stored, and the computer program may include computer instructions, and the computer program can be loaded by a processor to execute any feature extraction method provided in the embodiment of the present application.

[0166] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0167] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0168] Since the instructions stored in the storage medium can execute any one of the feature extraction methods provided in the embodiments of the present application, the beneficial effects that can be achieved by any one of the feature extraction methods provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0169] The above is a detailed introduction to a feature extraction method, device, computer equipment and storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A feature extraction method, characterized in that: include: Sampling a preset neighborhood range from the image to be processed to obtain at least one sampling area, wherein the position points of the sampling area include a sampling reference point and a plurality of neighborhood points; Calculate the relative relationship between the sampling reference point and each neighboring point in the sampling area; Calculating the weight vector of each position point in the sampling area based on the relative relationship, including calculating the weight vector of each position point in the sampling area based on the relative relationship between the sampling reference point and the neighborhood point in the sampling area by a multi-layer perceptron; Extracting feature vectors of each position point in the sampling area; Performing vector operations on the weight vectors of each position point in the sampling area and the corresponding feature vectors to obtain an intermediate vector corresponding to each position point; Performing a fusion operation on the obtained multiple intermediate vectors to obtain a fused vector; Performing a convolution operation on the fused vector to obtain an output vector of the sampling area; A feature map corresponding to the image to be processed is generated according to the output vectors of each sampling area.

2. The feature extraction method according to claim 1, characterized in that: The fusion operation is a pooling operation or a sequential head-to-tail splicing operation.

3. The feature extraction method according to claim 1, characterized in that: The relative relationship includes at least one of a relative position relationship and a relative characteristic relationship between the sampling reference point and each position point in the sampling area; The relative position relationship includes relative Cartesian coordinates; the relative feature relationship includes the association relationship between the feature vectors.

4. The feature extraction method according to claim 3, characterized in that: The sampling reference point is the center point of the sampling area.

5. The feature extraction method according to claim 1, characterized in that: The image to be processed is a depth image, which has multiple position points, each of which has attribute information, and the attribute information includes at least one of depth information, spatial coordinates, elongation, reflection intensity, azimuth, pitch angle, size of the real frame, and number of points in the real frame.

6. The feature extraction method according to any one of claims 1 to 5, characterized in that: After generating the feature map corresponding to the image to be processed, the feature extraction method further includes: At least one of the following operations is performed on the image to be processed according to the feature map: a target detection operation, a semantic segmentation operation, an instance segmentation operation, an image classification operation, and an operation of completing missing information in the image.

7. The feature extraction method according to claim 5, characterized in that: After generating the feature map corresponding to the image to be processed, the feature extraction method further includes: Detecting the feature map according to the attribute information, generating multiple detection frames within a preset range and the overlap between each detection frame and the corresponding real frame; The multiple detection frames are merged at least according to the overlap degree of each detection frame to obtain a target frame corresponding to the image to be processed.

8. The feature extraction method according to any one of claims 1 to 5, characterized in that: After generating the feature map corresponding to the image, the feature extraction method further includes: Downsampling the feature map according to a preset sampling step size to obtain a plurality of first sampling feature maps with different resolutions; The multiple first sampling feature maps with different resolutions are respectively upsampled to the original resolution of the image to be processed to obtain multiple second sampling feature maps with different resolutions.

9. The feature extraction method according to claim 8, characterized in that: The image to be processed is a sample image in a training set, the sample image has a pre-labeled real frame, and the real frame is assigned to the second sampling feature map of different resolutions according to its attribute information; Among them, each second sampling feature map is used to generate a detection frame of a preset attribute interval, and a predicted overlap between each detection frame and a corresponding real frame of the same category.

10. The feature extraction method according to claim 9, further comprising: Generate multiple prediction boxes corresponding to each second sampling feature map, and the prediction overlap between each prediction box and the corresponding real box of the same category; The initial target detection model is trained based on the predicted overlap and the corresponding true overlap to obtain a trained target detection model.

11. The feature extraction method according to claim 10, characterized in that: The training of the initial target detection model based on the predicted overlap and the actual overlap to obtain the target detection model includes: Calculate the classification loss value between the predicted overlap and the true overlap by using a classification loss function; Calculate the regression loss value between the predicted frame and the true frame through a regression loss function; Determine a total loss value according to the classification loss value and the regression loss value; The initial target detection model is trained according to the total loss value to obtain a trained target detection model.

12. A feature extraction device, characterized in that: include: A sampling module, used for sampling a preset neighborhood range from the image to be processed to obtain at least one sampling area, wherein the position points of the sampling area include a sampling reference point and a plurality of neighborhood points; A first calculation module, used to calculate the relative relationship between the sampling reference point and each neighboring point in the sampling area; A second calculation module is used to calculate the weight vector of each position point in the sampling area based on the relative relationship, including calculating the weight vector of each position point in the sampling area based on the relative relationship between the sampling reference point and the neighborhood point in the sampling area by a multi-layer perceptron; An extraction module, used to extract the feature vector of each position point in the sampling area; A third calculation module is used to perform vector operations on the weight vectors and corresponding feature vectors of each position point in the sampling area to obtain an intermediate vector corresponding to each position point; perform a fusion operation on the obtained multiple intermediate vectors to obtain a fusion vector; and perform a convolution operation on the fusion vector to obtain an output vector of the sampling area; A generation module is used to generate a feature map corresponding to the image to be processed according to the output vector of each sampling area.

13. A computer device, characterized in that: The method comprises a processor and a memory, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, the feature extraction method according to any one of claims 1 to 11 is executed.

14. A storage medium, characterized in that: The storage medium is used to store a computer program, and the computer program is loaded by a processor to execute the feature extraction method according to any one of claims 1 to 11.

Images