Suspended object grounding point extraction method, electronic device, vehicle and medium

Abstract

The embodiment of the invention provides a suspended object grounding point extraction method, an electronic device, a vehicle and a medium. A suspended object detection result and image feature information are determined according to an input image; and inputting the suspended object detection result and the image feature information into a grounding point extraction model to obtain suspended object grounding point information in the input image. Namely, the grounding point of the suspended object can be directly output through the grounding point extraction model, so that the efficiency of obtaining the grounding point of the suspended object is improved, and meanwhile, the precision of avoiding the suspended object is improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of automobiles, and particularly relates to a method for extracting a grounding point of a suspended object, an electronic device, a vehicle and a medium. BACKGROUND

[0002] Under the background of rapid development of automatic driving technology, accurately identifying the grounding point of a suspended object to avoid the suspended object is crucial to improve traffic safety and efficiency. In the prior art, the grounding point of a suspended object is obtained by analyzing the physical structure and logic of the suspended object in an input image, and a path between a target vehicle and the grounding point of the suspended object is calculated to achieve the purpose of avoiding the suspended object. However, in the prior art, the efficiency of obtaining the grounding point of the suspended object is not high. SUMMARY

[0003] Embodiments of the application provide a method for extracting a grounding point of a suspended object to solve the problem of low efficiency in obtaining the grounding point of the suspended object in the prior art.

[0004] In a first aspect, the embodiments of the application provide a method for extracting a grounding point of a suspended object, comprising:

[0005] determining a suspended object detection result and image feature information according to an input image;

[0006] inputting the suspended object detection result and the image feature information into a grounding point extraction model to obtain suspended object grounding point information in the input image.

[0007] Optionally, the grounding point extraction model obtains a target feature vector of the suspended object based on the suspended object detection result and the image feature information, and obtains the suspended object grounding point information based on the target feature vector of the suspended object, wherein the suspended object grounding point information comprises a position coordinate of the suspended object grounding point.

[0008] Optionally, obtaining the target feature vector of the suspended object based on the suspended object detection result and the image feature information comprises:

[0009] obtaining image feature information of the input image, wherein the image feature information comprises a full-image feature vector;

[0010] obtaining a suspended object detection result in the input image, wherein the suspended object detection result comprises an initial feature vector of the suspended object;

[0011] performing attention calculation on the full-image feature vector of the input image and the initial feature vector of the suspended object by a cross-attention module to obtain the target feature vector of the suspended object.

[0012] Optionally, the attention calculation on the full-image feature vector of the input image and the initial feature vector of the hovering object by the cross-attention module to obtain the target feature vector of the hovering object in the input image comprises:

[0013] For the full-image feature vector of the input image, a value vector matrix and a key vector matrix are obtained;

[0014] For the initial feature vector of the hovering object, a hovering object query vector matrix is obtained;

[0015] Based on the value vector matrix, the key vector matrix and the hovering object query vector matrix, the target feature vector of the hovering object is obtained.

[0016] Optionally, the target feature vector of the hovering object is obtained based on the value vector matrix, the key vector matrix and the hovering object query vector matrix, comprising:

[0017] Based on the key vector matrix and the hovering object query vector matrix, a hovering object target weight matrix is obtained;

[0018] Based on the hovering object target weight matrix and the value vector matrix, the target feature vector of the hovering object is obtained.

[0019] Optionally, the hovering object target weight matrix is obtained based on the key vector matrix and the hovering object query vector matrix, comprising:

[0020] The key vector matrix and the hovering object query vector matrix are point multiplied to obtain a hovering object initial weight matrix;

[0021] The hovering object initial weight matrix is normalized to obtain a hovering object target weight matrix.

[0022] Optionally, the hovering object target weight matrix is obtained by normalizing the hovering object initial weight matrix, comprising:

[0023] wherein z i represents the i-th element in the hovering object initial weight matrix; softmax() represents an activation function normalization operation; represents taking the exponential of each element z i ; C represents the number of elements; represents summing the exponentials of all elements as a normalization factor.

[0024] Optionally, the target feature vector of the hovering object is obtained based on the target feature vector of the hovering object, comprising:

[0025] obtain a probability value of a pixel point in each column of pixels in the input image belonging to the grounding point of the levitation object based on a target feature vector of the levitation object;

[0026] obtain a pixel point coordinate and a probability value of each column of pixels belonging to the grounding point of the levitation object through an Argmax function;

[0027] obtain the coordinate of the grounding point of the levitation object through threshold filtering.

[0028] Optionally, the grounding point extraction model is trained through a levitation object grounding point data set, wherein the levitation object grounding point data set includes a sample image, first annotation information of the sample image, and second annotation information of the sample image, the first annotation information is used for framing the levitation object in the sample image, and the second annotation information is used for annotating the grounding point of the levitation object projected on the ground in the sample image.

[0029] The second aspect of the embodiment of the application provides a levitation object grounding point extraction device, including:

[0030] An acquisition module is configured to determine a levitation object detection result and image feature information according to an input image.

[0031] A processing module is configured to input the levitation object detection result and the image feature information into a grounding point extraction model to obtain levitation object grounding point information in the input image.

[0032] Optionally, the processing module is specifically configured to acquire a target feature vector of the levitation object based on the levitation object detection result and the image feature information, and obtain the levitation object grounding point information based on the target feature vector of the levitation object, wherein the levitation object grounding point information includes a position coordinate of the grounding point of the levitation object.

[0033] Optionally, the acquisition module is specifically configured to acquire image feature information of the input image, and the image feature information includes a full-image feature vector; and acquire a levitation object detection result in the input image, and the levitation object detection result includes an initial feature vector of the levitation object.

[0034] The processing module is specifically configured to perform attention calculation on the full-image feature vector of the input image and the initial feature vector of the levitation object through a cross-attention module to obtain the target feature vector of the levitation object.

[0035] Optionally, the processing module is specifically configured to obtain a value vector matrix and a key vector matrix for a full image feature vector of the input image; obtain a floating object query vector matrix for an initial feature vector of the floating object; and obtain the target feature vector of the floating object based on the value vector matrix, the key vector matrix, and the floating object query vector matrix.

[0036] Optionally, the processing module is specifically configured to obtain a floating object target weight matrix based on the key vector matrix and the floating object query vector matrix; and obtain the target feature vector of the floating object based on the floating object target weight matrix and the value vector matrix.

[0037] Optionally, the processing module is specifically configured to perform point multiplication calculation on the key vector matrix and the floating object query vector matrix to obtain a floating object initial weight matrix; and perform normalization processing on the floating object initial weight matrix to obtain a floating object target weight matrix.

[0038] Optionally, the processing module is specifically configured to wherein, z i represents the i-th element in the floating object initial weight matrix; softmax() represents an activation function normalization operation; represents taking the exponential of each element z i ; C represents the number of elements; represents summing the exponents of all elements as a normalization factor.

[0039] Optionally, the processing module is specifically configured to obtain a probability value of a pixel point in each column of pixels in the input image belonging to a floating object contact point based on the target feature vector of the floating object; obtain a pixel point coordinate and a probability value of a pixel point with the largest probability of belonging to the floating object contact point in each column of pixels through an Argmax function; and obtain the coordinate of the floating object contact point through threshold filtering.

[0040] Optionally, the acquisition module is further configured to train the contact point extraction model through a floating object contact point data set, wherein the floating object contact point data set includes a sample image, first annotation information of the sample image, and second annotation information of the sample image, the first annotation information is used for framing the floating object in the sample image, and the second annotation information is used for annotating the contact point of the floating object projected on the ground in the sample image.

[0041] A third aspect of the embodiment of the present application provides an electronic device, including: a processor, the processor is used to be connected with a memory, the memory stores programs or instructions that can be run on the processor, and the programs or instructions are executed by the processor to realize the steps of the floating object contact point extraction method in the first aspect.

[0042] The fourth aspect of the embodiment of the present application provides a vehicle, which comprises the electronic device of the third aspect.

[0043] The fifth aspect of the embodiment of the present application provides a computer readable storage medium, wherein the readable storage medium stores programs or instructions, and the programs or instructions are executed by a processor to realize the steps of the floating object grounding point extraction method of the first aspect.

[0044] The sixth aspect of the embodiment of the present application provides a computer program product, wherein the program product is executed by a processor of a cloud server to realize the steps of the floating object grounding point extraction method of the first aspect.

[0045] The floating object grounding point extraction method, the electronic device, the vehicle and the medium provided by the embodiment of the present application can determine the floating object detection result and the image feature information according to the input image, input the floating object detection result and the image feature information into the grounding point extraction model, and obtain the floating object grounding point information in the input image. That is, the floating object grounding point can be directly output by the grounding point extraction model, so that the efficiency of obtaining the floating object grounding point is improved, and the precision of avoiding the floating object is improved. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 An effect diagram of a floating object grounding point provided by the embodiment of the present application;

[0047] Figure 2 A flow diagram of a floating object grounding point extraction method provided by the embodiment of the present application;

[0048] Figure 3 A flow diagram of another floating object grounding point extraction method provided by the embodiment of the present application;

[0049] Figure 4 A structure diagram of a grounding point extraction model provided by the embodiment of the present application;

[0050] Figure 5 A flow diagram of another floating object grounding point extraction method provided by the embodiment of the present application;

[0051] Figure 6 A flow diagram of another floating object grounding point extraction method provided by the embodiment of the present application;

[0052] Figure 7 A flow diagram of another floating object grounding point extraction method provided by the embodiment of the present application;

[0053] Figure 8A structural schematic diagram of a cross-attention module provided for an embodiment of the present application is shown in FIG. 1.

[0054] Figure 9 A flowchart of another method for extracting a grounding point of a floating object provided for an embodiment of the present application is shown in FIG. 6.

[0055] Figure 10 A flowchart of another method for extracting a grounding point of a floating object provided for an embodiment of the present application is shown in FIG. 6.

[0056] Figure 11 A flowchart of another method for extracting a grounding point of a floating object provided for an embodiment of the present application is shown in FIG. 6.

[0057] Figure 12 A flowchart of another method for extracting a grounding point of a floating object provided for an embodiment of the present application is shown in FIG. 6.

[0058] Figure 13 A structural schematic diagram of a device for extracting a grounding point of a floating object provided for an embodiment of the present application is shown in FIG. 7. DETAILED DESCRIPTION

[0059] The technical solutions in the embodiments of the present application will be clearly described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art belong to the scope of protection of the present application.

[0060] The terms "first", "second", and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second" are generally of a kind and do not limit the number of objects, for example, the first object can be one or more. In addition, "or" in the present application means at least one of the connected objects. For example, "A or B" covers three scenarios, namely, scenario one: including A and not including B; scenario two: including B and not including A; scenario three: including A and including B. The character " / " generally represents that the objects before and after are in an "or" relationship.

[0061] The term "indication" in the present application can be a direct indication (or explicit indication) or an indirect indication (or implicit indication). The direct indication can be understood as that the sender explicitly informs the receiver of specific information, operations to be performed or requested results, etc. in the sent indication. The indirect indication can be understood as that the receiver determines the corresponding information according to the indication sent by the sender, or judges and determines the operations to be performed or the requested results according to the judgment result.

[0062] In the prior art, the contact point of the overhanging object is obtained by segmenting the overhanging object from the image and then calculating the contact point of the overhanging object according to the shape structure or the light and shadow structure of the obstacle by using complex logic. However, the above method has the problems of complicated processing steps and long time consumption in obtaining the contact point of the overhanging object, resulting in low efficiency in obtaining the contact point of the overhanging object. The present application provides an overhanging object contact point extraction method, which constructs an overhanging object contact point extraction structure based on a target detection model and a cross attention module, so as to directly output a high-precision overhanging object contact point, reduce complexity, and improve the efficiency of vehicle avoidance of the overhanging object.

[0063] The overhanging object related to the embodiments of the present application refers to an object with part of the structure contacting the ground, or an object with the whole structure not on the ground but possibly affecting the smooth parking of a vehicle. The overhanging object with part of the structure contacting the ground includes but is not limited to an inclined column, a billboard, a signboard, etc. The overhanging object with the whole structure not on the ground includes but is not limited to a suspended fire hydrant box, a suspended vehicle head and tail, a hollow fence, and a suspended vehicle spare tire, etc. For the overhanging object scene with part of the structure contacting the ground, the contact point itself is the position contacting the ground. For the overhanging object scene with the whole structure not on the ground, the position information of the vertical projection of the overhanging object to the ground, i.e. the overhanging object contact point information, can be obtained by the overhanging object contact point extraction method provided by the present application.

[0064] Figure 1 An effect diagram of the overhanging object contact point provided by the embodiments of the present application is shown in Figure 1 As shown in the figure, the overhanging object 101 is a suspended fire hydrant box protruding from the wall surface and a suspended area between the suspended fire hydrant box and the ground, and the overhanging object contact point 102 is the projection point of the suspended fire hydrant box perpendicular to the ground.

[0065] For the convenience of description, in the following embodiments, the hovering object feature vector included in the hovering object detection result of the input image is described as an initial feature vector of the hovering object, the hovering object feature vector obtained by crossing attention from the initial feature vector of the hovering object and the full-image feature vector of the input image is described as a target feature vector of the hovering object, the hovering object weight matrix obtained by point multiplication of the key vector matrix and the hovering object query vector matrix is described as an initial hovering object weight matrix, the hovering object weight matrix obtained by normalization processing of the initial hovering object weight matrix is described as a target hovering object weight matrix, and the bounding box of the hovering object manually labeled is described as first labeling information, and the grounding point of the hovering object manually labeled is described as second labeling information.

[0066] The technical solutions of the hovering object grounding point extraction method of the present application will be described below with several specific embodiments as examples:

[0067] Figure 2 The flowchart of the hovering object grounding point extraction method provided by the embodiments of the present application is shown in FIG. Figure 2 The method of the embodiments of the present application is as follows:

[0068] S21: Determine the hovering object detection result and image feature information according to the input image.

[0069] The input image includes a hovering object. The detection result of the hovering object refers to the part including the hovering object identified from the input image, for example, the part from above the hovering object to the ground, which can represent the hovering object detection result through the feature information of the part. The image feature information refers to the feature information of the whole image of the input image.

[0070] The possible implementation manners of determining the hovering object detection result include but are not limited to edge detection algorithm, instance segmentation method based on deep learning, and three-dimensional reconstruction-based technology, etc., and the possible implementation manners of determining the image feature information include but are not limited to color feature extraction-based method, texture feature extraction-based method, space feature extraction-based method, and deep learning feature extraction-based method, etc.

[0071] S22: Input the hovering object detection result and the image feature information into a grounding point extraction model to obtain the hovering object grounding point information in the input image.

[0072] The grounding point extraction model can directly output the hovering object grounding point information in the input image, and the hovering object grounding point is the projection point of the hovering object perpendicular to the ground.

[0073] In the embodiment, the overhanging object detection result and the image feature information are determined according to an input image; the overhanging object detection result and the image feature information are input into a grounding point extraction model to obtain the grounding point information of the overhanging object in the input image. That is, the grounding point of the overhanging object can be directly output by the grounding point extraction model, so that the efficiency of obtaining the grounding point of the overhanging object is improved, and the precision of avoiding the overhanging object is improved.

[0074] Figure 3 Another flowchart of an overhanging object grounding point extraction method provided by the embodiment of the application is shown in Figure 3 Figure 2 Based on the embodiment shown in the embodiment, the overhanging object detection result and the image feature information are determined by a deep learning model, which includes an overhanging object detection module and a feature extraction module. A possible implementation of S21 is described as shown in Figure 3

[0075] S211: input an input image into the overhanging object detection module and the feature extraction module.

[0076] The overhanging object detection result can be obtained by the overhanging object detection module, and the image feature information can be obtained by the feature extraction module.

[0077] S212: obtain an overhanging object detection result by the overhanging object detection module.

[0078] The overhanging object detection module can be obtained by model training, such as a YOLOv8 target detection model. The YOLOv8 target detection network can enhance the detection ability of the model for objects of different sizes by multi-scale feature fusion. The scale and proportion of the anchor point can be automatically adjusted by a dynamic anchor point adjustment technology to match the target box of the predicted overhanging object, so as to accurately extract the bounding feature of the overhanging object and provide fast inference speed. The input image is input into the YOLOv8 target detection model to obtain the overhanging object detection result of the input image.

[0079] S213: obtain image feature information by the feature extraction module.

[0080] ​​The feature extraction module can be obtained through model training, such as a short-term dense concatenate (STDC) feature extraction model. The STDC feature extraction network is a lightweight real-time semantic segmentation algorithm network. Different stage feature maps are connected in cascade through a jump connection to enrich feature information and increase a receptive field, so that details in an image can be more accurately captured, and real-time and efficient application can be realized. The input image is input into the STDC feature extraction model, and image feature information of the input image can be obtained.

[0081] In this embodiment, the input image is input into the suspended object detection module and the feature extraction module. The suspended object detection result is obtained through the suspended object detection module, and the image feature information is obtained through the feature extraction module. Therefore, the suspended object detection result and the image feature information are determined, which helps to extract the suspended object grounding point information.

[0082] In the above embodiment, the execution order of S212 and S213 is not limited.

[0083] In this embodiment, the suspended object detection result and the image feature information are input into the grounding point extraction model, and a target feature vector of the suspended object can be obtained. The position information of the suspended object grounding point can be obtained based on the target feature vector of the suspended object. Next, a technical solution realized based on the grounding point extraction model is described in detail.

[0084] Figure 4 A structure diagram of a grounding point extraction model provided in this embodiment is shown. Optionally, a structure of the grounding point extraction model can include a cross-attention module, a max pooling layer, a resize tool, a Conv2D layer, an ArgMax function, and a threshold filter. The input of the cross-attention module is an initial feature vector of a suspended object in the input image and a full-image feature vector of the input image, which is used for cross-attention calculation and outputs a target feature vector of the suspended object in the input image. The max pooling layer is used for down-sampling, that is, reducing the spatial size of the feature map to reduce the calculation complexity of the model. The resize tool is used to change the size of the feature map to match the size of the input image. The Conv2D layer is used to learn the spatial information of the feature map and output the same size as the input. The ArgMax function is used to find the pixel position of the input image with the maximum probability of belonging to the suspended object grounding point. The threshold filter is used to preset a probability threshold to remove pixel points below the probability value to reduce the false detection of the suspended object grounding point. Through the processing of the grounding point extraction model, the position coordinates of the suspended object grounding point can be obtained.

[0085] Figure 4 Each module in the system of the illustrated embodiment can be used to execute... Figures 5 to 11 The technical solution of the embodiment of the method for extracting the grounding point of a suspended object shown herein, its implementation principle and technical effects are described in the following references. Figures 5 to 11 Detailed description of the illustrated embodiment.

[0086] Figure 5 This is a flowchart illustrating another method for extracting the grounding point of a suspended object provided in an embodiment of this application. Figure 5 Is Figure 2 Based on this, a further description of a possible implementation of S22 is provided, such as... Figure 5 As shown, the method of this application embodiment is as follows:

[0087] S221: The grounding point extraction model obtains the target feature vector of the suspended object based on the detection result of the suspended object and the image feature information.

[0088] Among them, such as Figure 4 As shown, specifically, the detection results of the suspended object and the image feature information are input into the cross-attention module in the grounding point extraction model to perform cross-attention calculation, which can obtain the target feature vector of the suspended object. The target feature vector of the suspended object is the feature map of the region of the suspended object that stands out in the entire input image.

[0089] S222: Based on the target feature vector of the suspended object, obtain the grounding point information of the suspended object.

[0090] The grounding point extraction model is a regression-based prediction model. The regression-based prediction model can predict the target output based on one or more input features. That is, the grounding point extraction model can obtain the grounding point information of the suspended object based on the target feature vector of the suspended object.

[0091] In this embodiment, the grounding point extraction model obtains the target feature vector of the suspended object based on the detection result of the suspended object and the image feature information; the grounding point extraction model obtains the target feature vector of the suspended object based on the detection result of the suspended object and the image feature information, thereby realizing the determination of the grounding point of the suspended object through the feature vector of the suspended object.

[0092] Figure 6 This is a flowchart illustrating another method for extracting the grounding point of a suspended object, provided in an embodiment of this application. Figure 6 Is Figure 5 Based on this, a further description of a possible implementation of S221 is provided, such as... Figure 6 As shown, the method of this application embodiment is as follows:

[0093] S2211: Obtain image feature information of the input image, the image feature information comprising: a full-image feature vector.

[0094] Based on the above embodiments, one possible implementation of obtaining the image feature information is based on an STDC feature extraction model, and the obtained is the feature of the entire image of the input image, i.e., including the full-image feature vector.

[0095] For example, assuming that the input image obtained by the surround-view camera has a size of 1024x1024x3, where 1024x1024 is the width and height of the input image, and 3 represents three color channels, and the full-image feature vector output by the STDC feature extraction network has a size of 128x64x64, where 128 represents the number of full-image feature maps, and 64x64 represents the width and height of each full-image feature map.

[0096] S2212: Obtain a hanging object detection result in the input image, the hanging object detection result comprising: an initial feature vector of the hanging object.

[0097] Based on the above embodiments, one possible implementation of obtaining the hanging object detection result is based on a YOLOv8 target detection model, and the obtained is the feature of all hanging objects in the input image, i.e., including the initial feature vector of the hanging object.

[0098] For example, based on the foregoing example, assuming that the input image obtained by the surround-view camera has a size of 1024x1024x3, and the initial feature vector of the hanging object output by the YOLOv8 target detection network has a size of Nx128x64x64, where N represents the number of hanging objects in the input image, 128 represents the number of initial feature maps of the hanging object, and 64x64 represents the width and height of each initial feature map of the hanging object.

[0099] S2213: Perform attention calculation on the full-image feature vector of the input image and the initial feature vector of the hanging object in the input image through a cross-attention module to obtain a target feature vector of the hanging object in the input image.

[0100] The cross-attention (Cross-Attention) is a variant of the attention mechanism, which is used for interactive learning between different sequences, and allows a query (Query, Q) sequence to focus on different parts of another key (Key, K) value (Value, V) sequence.

[0101] Specifically, the input of cross attention is the query vector in the query sequence and the key vector and value vector in the key-value sequence. The similarity score between each query vector and all key vectors is calculated by dot product, scaled dot product and cosine similarity, i.e. the similarity between each element in each query vector and each element in each key vector is calculated, which can be expressed as:

[0102]

[0103] wherein e i,j represents the similarity score between each query vector and all key vectors, q i represents the query vector, k i represents the key vector, d k represents the dimension of the key vector, divided by is to scale the dot product result to avoid excessive gradient in the subsequent softmax function.

[0104] The obtained similarity score is normalized by the softmax activation function to form a probability distribution, so as to obtain a set of attention weight matrices, which can be expressed as softmax(e i,j ); the obtained attention weight and the value vector are weighted and summed to obtain a new vector, that is, after cross attention calculation, each position in the query sequence will obtain a new vector as output, which integrates relevant information from the key-value sequence, which can be expressed as:

[0105] o i =∑ j softmax(e i,j )v j

[0106] wherein o i represents the output new vector, v i represents the value vector, and ∑ j · represents the weighted sum operation.

[0107] It can be understood that in order to improve the attention degree of the grounding point extraction model to the floating object in the input image, the obtained full image feature vector and the initial feature vector of the floating object are input into the cross attention module to perform cross attention, so as to obtain a more prominent feature vector of the floating object in the input image, i.e. the target feature vector of the floating object.

[0108] Specifically, the initial feature vector of the floating object is converted into a floating object query vector matrix, and the global feature vector is converted into a key vector matrix and a value vector matrix. Then, cross attention calculation is performed based on each floating object query vector, key vector, and value vector to obtain a floating object feature vector that fuses global feature vector information, i.e., the target feature vector of the floating object.

[0109] In the embodiment, the image feature information of the input image is obtained, and the image feature information includes a global feature vector. The floating object detection result in the input image is obtained, and the floating object detection result includes an initial feature vector of the floating object. The global feature vector of the input image and the initial feature vector of the floating object are subjected to attention calculation by a cross attention module to obtain the target feature vector of the floating object. Thus, the attention degree of the floating object feature is improved.

[0110] In the above embodiment, the sequence of steps S2211 and S2212 is not limited.

[0111] Figure 7 A flowchart of another floating object grounding point extraction method provided by the embodiment is shown in the following figure, Figure 7 which is Figure 5 based on the above, further describes a possible implementation of S2213, as shown in the following figure, Figure 7 the method of the embodiment is as follows:

[0112] S22131: Obtain a value vector matrix and a key vector matrix for the global feature vector of the input image.

[0113] Before the cross attention module performs cross attention calculation on the global feature vector and the initial feature vector of the floating object, the input feature vector can be converted by two-dimensional convolution to convert the global feature vector of the input image and the initial feature vector of the floating object into a form suitable for attention calculation.

[0114] Specifically, Figure 8 A structure diagram of a cross attention module provided by the embodiment is shown in the following figure, Figure 8 The global feature vector of the input image is input into a group of two-dimensional convolution layers, and a group of vector matrices, i.e., the value vector matrix, is output. The global feature vector of the input image is input into another group of two-dimensional convolution layers, and a group of vector matrices, i.e., the key vector matrix, is output.

[0115] Based on the foregoing example, assuming that the full-image feature vector size is 128x64x64, the value vector matrix size output by one set of two-dimensional convolution layers is 1024x64x64, and the key vector matrix size output by another set of two-dimensional convolution layers is also 1024x64x64, where 1024 represents that the number of value vectors in the value vector matrix is 1024, and the number of key vectors in the key vector matrix is also 1024, and 64x64 represents the spatial dimension of the value vectors and the key vectors.

[0116] S22132: Obtain a suspended object query vector matrix for the initial feature vector of the suspended object in the input image.

[0117] As shown in Figure 8 , the initial feature vector of the suspended object in the input image is input into still another set of two-dimensional convolution layers, and a set of vector matrices output by the still another set of two-dimensional convolution layers is the suspended object query vector matrix.

[0118] Based on the foregoing example, assuming that the initial feature vector of the suspended object has a size of Nx128x64x64, and the suspended object query vector matrix output by still another set of two-dimensional convolution layers has a size of Nx1024x64x64, where N represents the number of suspended objects in the input image, 1024 represents the number of suspended object query vectors in the suspended object query vector matrix, and 64x64 represents the spatial dimension of the suspended object query vectors.

[0119] S22133: Obtain a target feature vector of the suspended object in the input image based on the value vector matrix, the key vector matrix, and the suspended object query vector matrix.

[0120] The cross-attention module performs cross-attention calculation based on the value vector matrix, the key vector matrix, and the suspended object query vector matrix. Specifically, as shown in Figure 8 , a suspended object target weight matrix is obtained by calculating the dot product of each suspended object query vector and each key vector and then performing normalization, and then the obtained suspended object target weight matrix is multiplied by each value vector to obtain a target feature vector of the suspended object that fuses the full-image feature vector information.

[0121] In this embodiment, the value vector matrix and the key vector matrix are obtained for the full-image feature vector of the input image, the suspended object query vector matrix is obtained for the initial feature vector of the suspended object in the input image, and the target feature vector of the suspended object in the input image is obtained based on the value vector matrix, the key vector matrix, and the suspended object query vector matrix, so that the highlighting of the suspended object feature is realized through cross-attention calculation.

[0122] Figure 9A flowchart of another method for extracting a suspension object grounding point provided by an embodiment of the present application is shown in FIG. 6, Figure 9 is based on Figure 7 Further, a possible implementation of S22133 is described as shown in FIG. 3. Figure 9 The method of the embodiment of the present application is as follows:

[0123] S901: Obtain a suspension object target weight matrix based on the key vector matrix and the suspension object query vector matrix.

[0124] The dot product of the key vector and the suspension object query vector is calculated to obtain an attention weight matrix representing the similarity score of the key vector and the suspension object query vector, that is, a suspension object initial weight matrix. The suspension object initial weight matrix is normalized by a softmax function to convert the similarity score into a probability distribution, and an attention weight matrix with a similarity score of 1 in each row is obtained, that is, the suspension object target weight matrix.

[0125] S902: Obtain a target feature vector of the suspension object in the input image based on the suspension object target weight matrix and the value vector matrix.

[0126] The matrix multiplication of the suspension object target weight matrix and the value vector matrix is performed, that is, the value vector is aggregated based on the attention weight of the suspension object, and a suspension object feature vector fused with the related information of the value vector is calculated by weighted summation, that is, the target feature vector of the suspension object.

[0127] In this embodiment, the suspension object target weight matrix is obtained based on the key vector matrix and the suspension object query vector matrix, and the target feature vector of the suspension object in the input image is obtained based on the suspension object target weight matrix and the value vector matrix. Thus, the features of the suspension object are more accurately obtained.

[0128] Figure 10 A flowchart of another method for extracting a suspension object grounding point provided by an embodiment of the present application is shown in FIG. 6, Figure 10 is based on Figure 9 Further, a possible implementation of S901 is described as shown in FIG. 4. Figure 10 The method of the embodiment of the present application is as follows:

[0129] S9011: Calculate the dot product of the key vector matrix and the suspension object query vector matrix to obtain a suspension object initial weight matrix.

[0130] Wherein, the dot product calculation of the key vector matrix and the query vector matrix is to measure the similarity between the query and the key, therefore, each element in the obtained initial weight matrix of the suspended object is the similarity score of the suspended object query vector and the key vector.

[0131] For example, assuming that the key vector matrix is a 3x2 matrix, denoted as The suspended object query vector matrix is a 2x2 matrix, denoted as The dot product calculation of K and Q obtains a 2x3 initial weight matrix of the suspended object, denoted as Wherein each element is the similarity score of the suspended object query vector and the key vector.

[0132] S9012: Normalizing the initial weight matrix of the suspended object to obtain a target weight matrix of the suspended object.

[0133] Wherein, the normalization processing of the initial weight matrix of the suspended object by the softmax activation function converts the similarity score into a probability distribution, and the obtained target weight matrix of the suspended object can reflect the importance of each key vector to the suspended object query vector, and the softmax activation function is denoted as:

[0134]

[0135] Wherein, z i represents the i-th element in the initial weight matrix of the suspended object, and i≥1; softmax(·) represents the activation function normalization operation; represents the normalization operation of taking the exponential of each element z i to make all elements positive; C represents the number of elements; represents the sum of all elements after taking the exponential as a normalization factor.

[0136] Specifically, for each element z i in the initial weight matrix of the suspended object, the exponential value is calculated, the sum of the exponential values of each row is calculated, and the exponential value of each element in each row is divided by the sum of each row to obtain the normalized value. Based on the foregoing example, the softmax normalization calculation is performed on the initial weight matrix of the suspended object , and the obtained target weight matrix of the suspended object can be approximately Wherein the sum of each row element is 1.

[0137] In this embodiment, the initial weight matrix of the suspended object is obtained by performing a dot product between the key vector matrix and the query vector matrix of the suspended object; the initial weight matrix of the suspended object is then normalized to obtain the target weight matrix of the suspended object. Thus, the attention score of the suspended object is obtained, which helps to accurately acquire the features of the suspended object.

[0138] Figure 11 This is a flowchart illustrating another method for extracting the grounding point of a suspended object, provided in an embodiment of this application. Figure 11 Is Figure 5 Based on this, a further description of a possible implementation of S222 is provided, such as... Figure 11 As shown, the method of this application embodiment is as follows:

[0139] S2221: Based on the target feature vector of the suspended object in the input image, obtain the probability value of each pixel in each column of the input image belonging to the ground point of the suspended object.

[0140] Among them, such as Figure 4 As shown, the target feature vector of the suspended object is downsampled by the max pooling module, which reduces the spatial dimension of the suspended object feature map. Then, the downsampled suspended object feature map is adjusted to a feature map with the same spatial dimension as the input image by Resize. At this time, each pixel value in each column of the suspended object feature map represents the probability value of that pixel point belonging to the ground point of the suspended object in that column. The larger the value, the greater the probability that the pixel point belongs to the ground point of the suspended object in that column of pixels.

[0141] For example, assuming the spatial dimension of the input image is 1024×1024, and the target feature vector of the suspended object input to the max pooling module is 1024×64×64, that is, in the 1024 suspended object feature maps, the size of each suspended object feature map is 64×64. After the max pooling module, the size of the target feature vector of the suspended object becomes 1024×32×32, that is, the size of each suspended object feature map becomes 32×32. After resizing, a suspended object feature map of size 1024×1024 is obtained, which is the same size as the input image. At this time, there are a total of 1024 columns of pixels in the suspended object feature map, and each column of pixels has one pixel as the suspended object ground point, so there will be a total of 1024 suspended object ground points and probability values.

[0142] S2222: Using the Argmax function, obtain the coordinates and probability value of the pixel in each column that has the highest probability of being a ground point of a suspended object.

[0143] Wherein, the Argmax function is used to output the maximum value of the input value and the corresponding index. Through the ArgMax function, the pixel point coordinates with the maximum probability of belonging to the grounding point of the floating object and the probability value can be found in each column of pixels.

[0144] For example, based on the foregoing example, the floating area feature after resizing Figure 1 There are 1024 columns of pixels, so the input image has a total of 1024 probability values of the grounding point of the floating object. Through the ArgMax function, the pixel point position coordinates with the maximum probability of belonging to the grounding point of the floating object and the probability value can be found in each column of pixels of the input image. A total of 1024 pixel points belonging to the grounding point of the floating object can be determined.

[0145] S2223: Obtain the coordinates of the grounding point of the floating object through threshold filtering.

[0146] Wherein, in order to reduce the false detection of the grounding point of the floating object, an artificial experience-based threshold is preset to filter the probability values of the maximum probability of belonging to the grounding point of the floating object in each column of pixels obtained, remove the pixel points with probability values below the threshold, and mark the coordinates of the final output grounding point of the floating object.

[0147] For example, based on the foregoing example, assuming that the artificial preset threshold is 80%, the probability values of the 1024 pixel points obtained are filtered, the pixel points with probability values below 80% belonging to the grounding point of the floating object are removed, and the remaining pixel points are the final obtained grounding point of the floating object.

[0148] In this embodiment, the probability value of the pixel point belonging to the grounding point of the floating object in each column of pixels of the input image is obtained based on the target feature vector of the floating object in the input image. Through the Argmax function, the pixel point coordinates with the maximum probability of belonging to the grounding point of the floating object and the probability value are obtained in each column of pixels. Through threshold filtering, the coordinates of the grounding point of the floating object are obtained. Thus, the position of the grounding point of the floating object is accurately obtained.

[0149] Figure 12 A flowchart of another floating object grounding point extraction method provided by the embodiment of the application is shown in Figure 12 Based on Figure 3 the foregoing, further, before S21, the method further includes obtaining a floating object grounding point data set and training a grounding point extraction model, as shown in Figure 12 The method of the embodiment of the application is as follows:

[0150] S201: Obtain the grounding point extraction model through training of the floating object grounding point data set.

[0151] The suspension object ground contact point data set includes a sample image, first annotation information of the sample image, and second annotation information of the sample image, the first annotation information is used for framing the suspension object in the sample image, and the second annotation information is used for annotating the ground contact point of the suspension object projected on the ground in the sample image.

[0152] For example, in a scene in which a vehicle avoids a suspension object, a surround view image of the vehicle around the vehicle can be collected by a surround view camera as a sample image, the sample image contains a suspension fire hydrant box and a suspension vehicle head and tail object, the objects and a suspension area between the objects and the ground are manually framed and labeled, and a position of a suspension object vertically projected on the ground is labeled as a ground contact point, so as to construct a suspension object ground contact point data set. The first annotation information is a suspension object detection result, and the second annotation information is a suspension object ground contact point. Then, the suspension object ground contact point data set is used for training to obtain the ground contact point extraction model.

[0153] In the embodiment, the ground contact point extraction model is obtained by training based on the suspension object ground contact point data set, so as to help improve the prediction accuracy of the ground contact point extraction model.

[0154] In the above embodiment, further, after the suspension object ground contact point is obtained, the vehicle can perform path planning or obstacle avoidance processing based on the suspension object ground contact point information.

[0155] It can be understood that, in a scene in which a vehicle avoids a suspension object, after the suspension object ground contact point information is obtained, that is, after a projection point of the suspension object vertically on the ground is obtained, the vehicle can know the area information that should be avoided on the ground and the drivable area information. Therefore, the vehicle can perform path planning according to the position of the suspension object ground contact point to bypass the area or to avoid far away.

[0156] For example, in a scene in which an autonomous vehicle parks in an underground parking lot, a wall in front of a parking area protrudes a suspension fire hydrant box, an image is collected by a surround view camera of the vehicle, then the suspension object detection result and the image feature information of the suspension fire hydrant box are obtained, and then the position information of the suspension object ground contact point of the suspension fire hydrant box is obtained by the ground contact point extraction model, so that the vehicle can perform path planning and distance calculation according to the ground position information of the area to realize automatic parking and avoid contacting the suspension fire hydrant box.

[0157] Figure 13 A structure diagram of a suspension object ground contact point extraction device provided in the embodiment is shown, the device includes an acquisition module 1301 and a processing module 1302, wherein,

[0158] The acquisition module 1301 is configured to determine a hovering object detection result and image feature information according to an input image.

[0159] The processing module 1302 is configured to input the hovering object detection result and the image feature information into a grounding point extraction model to obtain hovering object grounding point information in the input image.

[0160] Optionally, the processing module 1302 is specifically configured to acquire, by the grounding point extraction model, a target feature vector of the hovering object based on the hovering object detection result and the image feature information, and obtain the hovering object grounding point information based on the target feature vector of the hovering object, where the hovering object grounding point information includes position coordinates of the hovering object grounding point.

[0161] Optionally, the acquisition module 1301 is specifically configured to acquire image feature information of the input image, where the image feature information includes a full-image feature vector, and acquire a hovering object detection result in the input image, where the hovering object detection result includes an initial feature vector of the hovering object.

[0162] The processing module 1302 is specifically configured to perform attention calculation on the full-image feature vector of the input image and the initial feature vector of the hovering object by a cross-attention module to obtain the target feature vector of the hovering object.

[0163] Optionally, the processing module 1302 is specifically configured to obtain a value vector matrix and a key vector matrix for the full-image feature vector of the input image, obtain a hovering object query vector matrix for the initial feature vector of the hovering object, and obtain the target feature vector of the hovering object based on the value vector matrix, the key vector matrix and the hovering object query vector matrix.

[0164] Optionally, the processing module 1302 is specifically configured to obtain a hovering object target weight matrix based on the key vector matrix and the hovering object query vector matrix, and obtain the target feature vector of the hovering object based on the hovering object target weight matrix and the value vector matrix.

[0165] Optionally, the processing module 1302 is specifically configured to perform dot product calculation on the key vector matrix and the hovering object query vector matrix to obtain a hovering object initial weight matrix, and perform normalization processing on the hovering object initial weight matrix to obtain the hovering object target weight matrix.

[0166] Optionally, the processing module 1302 is specifically configured to wherein z i represents an i-th element in the hovering object initial weight matrix; and softmax() represents an activation function normalization operation. denotes taking the exponent of each element z i taking the exponent; C denotes the number of elements; denotes summing the exponent of all elements as a normalization factor.

[0167] Optionally, the processing module 1302 is specific to obtain a probability value of a pixel point in each column of pixels of the input image belonging to the landing point of the hovering object based on the target feature vector of the hovering object; obtain the pixel point coordinate and the probability value of the pixel point in each column of pixels belonging to the landing point of the hovering object with the maximum probability through the Argmax function; and obtain the coordinate of the landing point of the hovering object through threshold filtering.

[0168] Optionally, the acquisition module 1301 is further configured to train the landing point extraction model through a hovering object landing point data set, wherein the hovering object landing point data set includes a sample image, first annotation information of the sample image, and second annotation information of the sample image, the first annotation information is used for framing the hovering object in the sample image, and the second annotation information is used for annotating the landing point of the hovering object projected on the ground in the sample image.

[0169] The device of the embodiment can be used to execute the technical solutions of the above-mentioned method embodiments correspondingly, and has similar implementation principles and technical effects, which will not be described here.

[0170] The embodiment of the application further provides an electronic device, comprising: a processor, the processor is connected with a memory, the memory stores programs or instructions that can be run on the processor, and the programs or instructions are executed by the processor to realize the steps of any of the above-mentioned hovering object landing point extraction method embodiments.

[0171] The embodiment of the application further provides a vehicle, comprising: a processor and a memory, the memory stores programs or instructions that can be run on the processor, and the programs or instructions are executed by the processor to realize the steps of any of the above-mentioned hovering object landing point extraction method embodiments.

[0172] The embodiment of the application further provides a computer readable storage medium, the readable storage medium stores programs or instructions, and the programs or instructions are executed by the processor to realize the steps of any of the above-mentioned hovering object landing point extraction method embodiments.

[0173] The embodiment of the application further provides a computer program product, the computer program product is executed by the processor of the cloud server to realize the steps of any of the above-mentioned hovering object landing point extraction method embodiments.

[0174] It should be noted that, in the present document, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises a", "comprising", or the like does not, without more constraints, preclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element. Furthermore, it is to be understood that the methods and apparatuses of the present application can be carried out by specific hardware, software, or a combination thereof, and that the scope of the application is not limited to the specific order of execution of the steps described in the examples. In addition, features described in relation to certain examples can be combined in other examples.

[0175] From the above description of the embodiments, it is clear that the above-mentioned method can be realized by means of a computer software product and a general hardware platform, of course, it can also be realized by hardware. The computer software product is stored in a storage medium (such as ROM, RAM, magnetic disc, optical disc, etc.), and includes a plurality of instructions for making the terminal or network side device execute the method described in each embodiment of the present application.

[0176] The embodiments of the present application are described above in combination with the drawings, but the present application is not limited to the above-mentioned specific embodiments, the above-mentioned specific embodiments are only illustrative, not restrictive, and those skilled in the art can make many forms of embodiments under the inspiration of the present application without departing from the scope of the present application and the scope protected by the claims.

Claims

1. A method for extracting the grounding point of a suspended object, characterized in that, include: Determine the detection results and image feature information of suspended objects based on the input image; The detection results of the suspended object and the image feature information are input into the grounding point extraction model to obtain the grounding point information of the suspended object in the input image.

2. The method according to claim 1, characterized in that, The grounding point extraction model obtains the target feature vector of the suspended object based on the detection results of the suspended object and the image feature information; Based on the target feature vector of the suspended object, the grounding point information of the suspended object is obtained, wherein the grounding point information of the suspended object includes: the position coordinates of the grounding point of the suspended object.

3. The method according to claim 2, characterized in that, The step of obtaining the target feature vector of the suspended object based on the suspended object detection result and the image feature information includes: Obtain the image feature information of the input image, wherein the image feature information includes: full-image feature vector; Obtain the detection result of the suspended object in the input image, wherein the detection result of the suspended object includes: the initial feature vector of the suspended object; The cross-attention module performs attention calculations on the full-image feature vector of the input image and the initial feature vector of the suspended object to obtain the target feature vector of the suspended object.

4. The method according to claim 3, characterized in that, The step of performing attention calculations on the full-image feature vector of the input image and the initial feature vector of the suspended object through a cross-attention module to obtain the target feature vector of the suspended object in the input image includes: For the full-image feature vector of the input image, the value vector matrix and the key vector matrix are obtained; Based on the initial feature vector of the suspended object, the query vector matrix of the suspended object is obtained; Based on the value vector matrix, key vector matrix, and suspended object query vector matrix, the target feature vector of the suspended object is obtained.

5. The method according to claim 4, characterized in that, The process of obtaining the target feature vector of the suspended object based on the value vector matrix, key vector matrix, and suspended object query vector matrix includes: Based on the key vector matrix and the suspended object query vector matrix, the suspended object target weight matrix is ​​obtained; Based on the target weight matrix of the suspended object and the value vector matrix, the target feature vector of the suspended object is obtained.

6. The method according to claim 5, characterized in that, The process of obtaining the suspended object target weight matrix based on the key vector matrix and the suspended object query vector matrix includes: The initial weight matrix of the suspended object is obtained by performing a dot product between the key vector matrix and the query vector matrix of the suspended object. The initial weight matrix of the suspended object is normalized to obtain the target weight matrix of the suspended object.

7. The method according to claim 6, characterized in that, The normalization process for the initial weight matrix of the suspended object to obtain the target weight matrix of the suspended object includes: Among them, z i This represents the i-th element in the initial weight matrix of the suspended object; softmax() represents the activation function normalization operation; This means that for each element z i Take the exponent; C represents the number of elements; This indicates that the normalization factor is obtained by summing the exponents of all elements.

8. The method according to claim 2, characterized in that, The step of obtaining the grounding point information of the suspended object based on the target feature vector of the suspended object includes: Based on the target feature vector of the suspended object, the probability value of each pixel in each column of the input image belonging to the ground point of the suspended object is obtained. The Argmax function is used to obtain the coordinates and probability value of the pixel in each column of pixels that has the highest probability of being the ground point of the suspended object. The coordinates of the grounding point of the suspended object are obtained by threshold filtering.

9. The method according to claim 1, characterized in that, The grounding point extraction model is trained using a dataset of grounding points of suspended objects. The dataset includes sample images, first annotation information of the sample images, and second annotation information of the sample images. The first annotation information is used to select suspended objects in the sample images, and the second annotation information is used to annotate the grounding points of the suspended objects projected onto the ground in the sample images.

10. An electronic device, characterized in that, include: A processor for connection to a memory storing a computer program / instructions executable on the processor, the computer program / instructions being executed by the processor to implement the steps of the suspended object grounding point extraction method as described in any one of claims 1 to 9.

11. A vehicle, characterized in that, Including the electronic device as described in claim 10.

12. A computer-readable storage medium, characterized in that, The readable storage medium stores a computer program / instruction, which, when executed by a processor, implements the steps of the method for extracting the grounding point of a suspended object as described in any one of claims 1 to 9.

13. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method for extracting the grounding point of a suspended object as described in any one of claims 1 to 9.

Images