Boundary recognition model, recognition method, electronic equipment and storage medium

Abstract

The invention provides a boundary recognition model, a recognition method, electronic equipment and a storage medium. The boundary recognition model comprises a feature extraction module, a feature processing module and a prediction module, the feature processing module comprises a plurality of convolutional layers, and the feature extraction module is used for extracting image features of a target image; the feature processing module is used for acquiring contour features of the obstacle grounding area contained in the image features through operation processing of the plurality of convolutional layers; and the prediction module is used for predicting the position of the obstacle grounding area in the target image according to the contour features, so that a contour formed by the position of the obstacle grounding area is used as the boundary of the travelable area. Therefore, the boundary recognition model takes the plurality of convolutional layers as an algorithm core, so that the algorithm is relatively simple, the overhead is low, and the delay can be shortened.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence (AI) technology, and specifically relates to a boundary recognition model, a recognition method, an electronic device and a storage medium. Background Art

[0002] Identifying the boundaries of the free space is an important process in the field of automatic parking. The automatic parking system can identify the boundaries of the free space through a pre-deployed algorithm model.

[0003] A commonly used algorithm model, for example, is a multi-layer transformer module that performs core image processing. The multi-layer transformer module can extract image features of a panoramic image of a vehicle, obtain the grounding position of an obstacle based on the extracted image features, and then determine the boundary of the drivable area based on the grounding position of the obstacle.

[0004] However, the algorithm model with multi-layer transformer modules as the core not only has high computational complexity and high overhead, but also has large delay, resulting in poor recognition performance of the boundaries of the drivable area. Summary of the invention

[0005] The present application proposes a boundary recognition model, recognition method, electronic device and storage medium, which can solve the problems of high computational complexity, high overhead and large delay of existing algorithm models.

[0006] The first aspect of the present application proposes a boundary recognition model for identifying the boundary of a drivable area of ​​a vehicle. The boundary recognition model includes: a feature extraction module, a feature processing module and a prediction module. The feature processing module includes multiple convolutional layers, wherein:

[0007] The feature extraction module is used to extract image features of a target image, wherein the target image includes an image of an obstacle contact area in an area where the vehicle is located, and the contour of the obstacle contact area constitutes a boundary of the drivable area;

[0008] The feature processing module is used to obtain the contour feature of the obstacle ground contact area contained in the image feature through the operation processing of the multiple convolutional layers, and the contour feature includes the pixel feature and the position feature of the obstacle ground contact area;

[0009] The prediction module is used to predict the position of the obstacle contact area in the target image according to the contour feature, so as to use the contour formed by the position of the obstacle contact area as the boundary of the drivable area.

[0010] In some embodiments of the present application, the feature processing module includes a position feature processing module and an encoding module.

[0011] The position feature processing module is used to generate coordinate features of the target image, wherein the coordinate features are used to characterize the association between the image features and a first coordinate system and a second coordinate system, wherein the first coordinate system is used to characterize the position features from a first dimension, and the second coordinate system is used to characterize the position features from a second dimension;

[0012] The encoding module is used to encode the coordinate features through the operation of multiple convolutional coding layers, the encoded features include the contour features, and the multiple convolutional coding layers belong to the multiple convolutional layers.

[0013] In some embodiments of the present application, the position feature processing module is further used to construct a coordinate mapping relationship matrix, where the coordinate mapping relationship refers to a mapping relationship between the first coordinate system and the second coordinate system, and the coordinate mapping relationship matrix includes coordinate mapping relationships of multiple position points, and the multiple position points are associated with the first coordinate system;

[0014] The position feature processing module is also used to obtain pixel features corresponding to the multiple position points in the image feature to obtain pixel features of the multiple position points; according to the correspondence between the multiple position points and the eigenvalues ​​in the coordinate mapping relationship matrix, the sorting of each feature point in the pixel feature is deployed to associate each feature point with the corresponding eigenvalue in the coordinate mapping relationship matrix, and the sorted pixel feature is the coordinate feature.

[0015] In some embodiments of the present application, the position feature processing module is also used to construct the first coordinate system according to the size of the image feature, the first coordinate system including M directions, and the multiple position points are distributed in the M directions, wherein M is an integer greater than 3; corresponding to any position point among the multiple position points, the coordinates of the position point in the first coordinate system are calculated to obtain the first coordinate of the position point; according to the first coordinate and the mapping relationship between the first coordinate system and the second coordinate system, the coordinates of the position point in the second coordinate system are calculated to obtain the second coordinate of the position point; and the second coordinates of the multiple position points are used as eigenvalues ​​to generate the coordinate mapping relationship matrix.

[0016] In some embodiments of the present application, when the feature extraction module includes a convolution extraction layer, the convolution extraction layer is used to extract image features of the target image;

[0017] When the feature extraction module includes at least two convolution extraction layers, the weights of the at least two convolution extraction layers are different; the at least two convolution extraction layers are respectively used to extract image sub-features of the target image at different sizes, and the size of the image sub-features extracted by any convolution extraction layer is related to the weight of the convolution extraction layer.

[0018] In some embodiments of the present application, when the feature extraction module includes at least two convolution extraction layers, the boundary recognition model further includes a feature fusion module, wherein:

[0019] The feature fusion module is used to fuse the image sub-features of the target image at different sizes, so as to input the fused features into the feature processing module.

[0020] In some embodiments of the present application, the prediction module includes a fully connected layer,

[0021] The fully connected layer is used to predict the position of the obstacle ground contact area in the target image according to the contour feature, and the position of the obstacle ground contact area in the target image includes position information of the first dimension and position information of the second dimension.

[0022] In some embodiments of the present application, the prediction module further includes a classification module.

[0023] The classification module is used to predict the type of the object to be identified corresponding to each pixel feature in the contour feature, and the type of the object to be identified is the obstacle grounding area or the edge of the target image.

[0024] The embodiment of the second aspect of the present application provides a recognition method, which is implemented by calling a boundary recognition model, and the method includes:

[0025] Extracting image features of a target image, wherein the target image includes an image of an obstacle contact area in an area where the vehicle is located;

[0026] Acquire, by multi-layer convolution operation of the boundary recognition model, contour features of the obstacle ground contact area contained in the image features, wherein the contour features include pixel features and position features of the obstacle ground contact area;

[0027] The position of the obstacle contact area in the target image is predicted based on the contour features, and the contour formed by the position of the obstacle contact area is the boundary of the drivable area of ​​the vehicle.

[0028] In some embodiments of the present application, obtaining the contour features of the obstacle ground contact area contained in the image features includes:

[0029] Generate a coordinate feature of the target image, wherein the coordinate feature is used to characterize an association relationship between the image feature and a first coordinate system and a second coordinate system, wherein the first coordinate system is used to characterize a position feature from a first dimension, and the second coordinate system is used to characterize a position feature from a second dimension;

[0030] The coordinate features are encoded through multi-layer convolutional coding operations, and the encoded features include the contour features.

[0031] In some embodiments of the present application, generating the coordinate features of the target image includes:

[0032] Constructing a coordinate mapping relationship matrix, wherein the coordinate mapping relationship refers to a mapping relationship between the first coordinate system and the second coordinate system, and the coordinate mapping relationship matrix includes coordinate mapping relationships of multiple position points, and the multiple position points are associated with the first coordinate system;

[0033] Obtain pixel features corresponding to the multiple position points in the image features to obtain pixel features of the multiple position points;

[0034] According to the correspondence between the multiple position points and the eigenvalues ​​in the coordinate mapping relationship matrix, the sorting of each feature point in the pixel feature is deployed so that each feature point is associated with the corresponding eigenvalue in the coordinate mapping relationship matrix, and the sorted pixel feature is the coordinate feature.

[0035] In some embodiments of the present application, the step of constructing a coordinate mapping relationship matrix includes:

[0036] Constructing the first coordinate system according to the size of the image feature, the first coordinate system includes M directions, the multiple position points are distributed in the M directions, wherein M is an integer greater than 3;

[0037] Corresponding to any one of the multiple position points, calculating the coordinates of the position point in the first coordinate system to obtain a first coordinate of the position point;

[0038] Calculating the coordinates of the position point in the second coordinate system according to the first coordinates and a mapping relationship between the first coordinate system and the second coordinate system to obtain the second coordinates of the position point;

[0039] The coordinate mapping relationship matrix is ​​generated by taking the second coordinates of the plurality of position points as eigenvalues.

[0040] In some embodiments of the present application, the method further includes training the model to be trained to obtain the boundary recognition model.

[0041] The training method comprises:

[0042] Calling the model to be trained to identify the sample image to obtain a position prediction result and a type prediction result, wherein the boundary of the drivable area contained in the sample image has been marked;

[0043] Calculating a smoothing loss function and a cross entropy loss function, wherein the smoothing loss function is used to characterize the loss between the position prediction result and the labeled position information, and the cross entropy loss function is used to characterize the loss between the type prediction result and the labeled type information;

[0044] Determine whether the smoothing loss function and the cross entropy loss function converge;

[0045] If both the smoothing loss function and the cross entropy loss function converge, the model to be trained is used as the boundary recognition model;

[0046] If the smoothing loss function and / or the cross entropy loss function do not converge, adjust the parameters of the model to be trained, and use the convolution model with the adjusted parameters as the new model to be trained, and again call the model to be trained to identify the sample image.

[0047] An embodiment of a third aspect of the present application provides an electronic device, in which the boundary recognition model described in the first aspect is deployed, and the electronic device calls the boundary recognition model to implement the method described in the second aspect.

[0048] An embodiment of the fourth aspect of the present application provides a computer-readable storage medium on which a computer program is stored, and the program is executed by a processor to implement the method described in the second aspect above.

[0049] The technical solution provided in the embodiments of the present application has at least the following technical effects or advantages:

[0050] The boundary recognition model for identifying the boundary of the drivable area in the embodiment of the present application includes a feature extraction module, a feature processing module and a prediction module, wherein the feature processing module includes multiple convolution layers. The feature extraction module is an algorithm module for extracting the image features of the target image. The feature processing module is a module for obtaining the contour features of the obstacle grounding area contained in the image features through the operation processing of the multiple convolution layers, and the contour features processed by the feature processing module include the pixel features and position features of the obstacle grounding area. Afterwards, the prediction module predicts the position of the obstacle grounding area in the target image according to the contour features, so as to use the contour formed by the position of the obstacle grounding area as the boundary of the drivable area. That is to say, in the boundary recognition model of the embodiment of the present application, the core algorithm module is composed of multiple convolution layers, that is, the boundary recognition model uses multi-layer convolution operations to process image features. Compared with the operation mechanism of the existing multi-layer transformer module, not only is the algorithm relatively simple and the overhead is small, but also the delay can be shortened and the recognition performance of the boundary of the drivable area can be optimized.

[0051] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] By reading the detailed description of the preferred embodiment below, various other advantages and benefits will become clear to those of ordinary skill in the art. The accompanying drawings are only used for the purpose of illustrating the preferred embodiment and are not considered to be limitations of the present application. In addition, the same reference symbols are used to represent the same components throughout the accompanying drawings.

[0053] In the attached picture:

[0054] Figure 1A A schematic diagram showing the composition of a boundary recognition model provided by an embodiment of the present application is shown;

[0055] Figure 1B A schematic diagram showing the composition of a boundary recognition model provided by another embodiment of the present application is shown;

[0056] Figure 1C A schematic diagram showing the composition of an exemplary boundary recognition model provided in an embodiment of the present application is shown;

[0057] Figure 2 A schematic diagram of a method flow of an identification method provided by an embodiment of the present application is shown;

[0058] Figure 3 A schematic diagram of the structure of an electronic device provided by an embodiment of the present application is shown;

[0059] Figure 4 A schematic diagram of a storage medium provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0060] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0061] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by technicians in the field to which this application belongs.

[0062] The embodiments of the present application relate to the field of AI technology and are applied to automatic parking scenarios. In the automatic parking scenario, after the vehicle enters the area to be parked, the boundary of the vehicle's drivable area (free space) in the area to be parked can be identified, and the parking control scheme can be planned based on the free space boundary. In some embodiments, the automatic parking system can identify the obstacle grounding position on the image of the panoramic monitoring imaging system (Around View Monitor, AVM), and the boundary composed of the contour line of the obstacle grounding position can be considered as the free space boundary.

[0063] Among them, the automatic parking system can call the pre-deployed algorithm model to process the AVM image to obtain the characteristics of the obstacle grounding image contained in the AVM image, and then identify the location of the obstacle grounding area according to the characteristics of the obstacle grounding image.

[0064] Some conventional algorithm models for identifying free space boundaries include multi-layer transformer modules. After extracting the image features of the AVM image, the multi-layer transformer module is called to calculate and process the image features to obtain predictable features, and then the free space boundary is identified from the predictable features. The multi-layer transformer module uses an attention mechanism for calculation, which is relatively computationally complex, expensive, and has a large delay, resulting in poor recognition performance of the drivable area boundary and a poor user experience.

[0065] In view of this, in the technical solution provided in the embodiment of the present application, the boundary recognition model includes a feature processing module, and the feature processing module includes multiple convolution layers. After obtaining the image features, the feature processing module obtains the contour features of the obstacle grounding area contained in the image features through the operation processing of the multiple convolution layers. Afterwards, the boundary recognition model identifies the free space boundary based on the contour features. That is, the boundary recognition model uses multi-layer convolution operations to process the image features, the algorithm is relatively simple, the overhead is low, and it can shorten the delay and optimize the recognition performance of the boundary of the drivable area.

[0066] The execution subject of this technical solution can be any device that supports the obstacle grounding position recognition function, including vehicles, ships, aircraft or robots. Such devices can support the obstacle grounding position recognition of the embodiment of this application through the automatic parking system. The automatic parking system can deploy a boundary recognition model for obstacle grounding position recognition.

[0067] The following describes a boundary recognition model, a recognition method, an electronic device, and a storage medium according to an embodiment of the present application in conjunction with the accompanying drawings.

[0068] See also Figure 1A The boundary recognition model provided by the embodiment of the present application may include: a feature extraction module, a feature processing module and a prediction module. Among them, the feature processing module includes multiple convolution layers.

[0069] It should be pointed out that, according to the data flow, the feature extraction module can be the backbone of the boundary recognition model, the feature processing module can be the neck module of the boundary recognition model, and the prediction module can be the head module of the boundary recognition model.

[0070] The feature extraction module can be used to extract image features of a target image. Exemplarily, the target image is, for example, an AVM image. Then, the target image includes an image of an obstacle contact area in the area where the vehicle is located.

[0071] In some embodiments, the feature extraction module can be implemented as any algorithm module that supports extracting image features from AVM images, such as but not limited to convolutional neural network (CNN), EfficientNetLite network, etc.

[0072] In addition, when the feature extraction module is an EfficientNetLite network, the weight of the EfficientNetLite can be pre-trained on the imagenet dataset. The weight size of the EfficientNetLite is related to the size of the image features extracted by the EfficientNetLite network.

[0073] Furthermore, the feature processing module can be used to obtain the contour features of the obstacle ground contact area contained in the image features through calculation processing of multiple convolutional layers, and the contour features include pixel features and position features of the obstacle ground contact area.

[0074] The prediction module is used to predict the position of the obstacle contact area in the target image according to the contour feature, so as to use the contour formed by the position of the obstacle contact area as the boundary of the drivable area.

[0075] It should be noted that the prediction module can be implemented as different algorithm modules according to the attributes of the recognition result corresponding to the boundary recognition model. For example, if the recognition result corresponding to the boundary recognition model is the type of the obstacle grounding area, then the prediction module can be a classification module; for another example, if the recognition result corresponding to the boundary recognition model includes the position of the obstacle grounding area in the target image, including the position information of the first dimension and the position information of the second dimension, the prediction module can be a fully connected layer.

[0076] It can be seen that in the boundary recognition model using the present implementation, both the feature extraction module and the feature processing module can be convolution modules, that is, the boundary recognition model uses multi-layer convolution operations to process image features. Not only is the algorithm simpler and less expensive than the existing multi-layer transformer module, but it can also shorten the delay.

[0077] It should be pointed out that, in the boundary recognition model provided in the embodiment of the present application, the feature extraction module may include at least one convolution extraction layer.

[0078] In some embodiments, when the feature extraction module includes a convolution extraction layer, the convolution extraction layer is used to extract the image features of the target image, and then the extracted image features of the target image are transmitted to the feature extraction module. In this embodiment, the boundary recognition model can be as follows: Figure 1A shown.

[0079] In some other embodiments, when the feature extraction module includes at least two convolution extraction layers, the weights of the at least two convolution extraction layers are different. The at least two convolution extraction layers can be used to extract image sub-features of the target image, and the sizes of the image features extracted by each two convolution extraction layers are different. In this embodiment, Figure 1B As shown, the boundary recognition model may also include a feature fusion module.

[0080] exist Figure 1B In the boundary recognition model shown, the feature fusion module can be used to fuse image sub-features of the target image at different sizes, so as to input the fused features into the feature processing module. The fused features can represent the image features of target images of different sizes, which is conducive to obtaining relatively complete image features.

[0081] Exemplarily, the feature fusion module can be implemented as a feature pyramid network (FPN).

[0082] It should be noted that the feature processing module can perform multiple processes, the functions of different processes can be different, and the features obtained are also different. According to the process to be performed and the function corresponding to the process to be performed, the feature processing module may include a position feature processing module and an encoding module, accordingly. The multiple convolutional layers may include convolutional layers that perform different functions, and the convolutional layers that perform each function are multiple convolutional layers. For example, the encoding module includes multiple convolutional encoding layers.

[0083] In some embodiments, the position feature processing module can be used to generate coordinate features of a target image, wherein the coordinate features are used to characterize the association between the image features and a first coordinate system and a second coordinate system, wherein the first coordinate system is used to characterize the position features from a first dimension, and the second coordinate system is used to characterize the position features from a second dimension.

[0084] The encoding module can be used to encode the coordinate features through the operation of multiple convolutional coding layers, and the encoded features include the contour features.

[0085] In this way, the prediction module may include a fully connected layer. The fully connected layer may predict the position of the obstacle contact area in the target image according to the contour feature, and the position of the obstacle contact area in the target image includes position information of the first dimension and position information of the second dimension.

[0086] In this way, the location information extracted from the feature data containing location features of two dimensions can contain location information of two dimensions, so that the obtained location representation has finer granularity and relatively higher accuracy.

[0087] Exemplarily, the position feature processing module can be used to construct the first coordinate system according to the size of the image feature, the first coordinate system includes M directions, and the multiple position points are distributed in the M directions, wherein M is an integer greater than 3; corresponding to any position point among the multiple position points, the coordinates of the position point in the first coordinate system are calculated to obtain the first coordinate of the position point; according to the first coordinate and the mapping relationship between the first coordinate system and the second coordinate system, the coordinates of the position point in the second coordinate system are calculated to obtain the second coordinate of the position point; the second coordinates of the multiple position points are used as eigenvalues ​​to generate the coordinate mapping relationship matrix. The coordinate mapping relationship refers to the mapping relationship between the first coordinate system and the second coordinate system.

[0088] In some embodiments, the first coordinate system may be, for example, a polar coordinate system, M may be, for example, 360, and 20 position points may be deployed in each of the 360 ​​directions, for example. The 20 position points in each direction may be sampling points in that direction. The first coordinate system may be, for example, a Cartesian coordinate system. In this way, the first coordinate system may represent the angle in the position information, and the second coordinate system may represent the distance in the position information. Furthermore, for any position point X, the coordinate value of the position point X in the polar coordinate system may be calculated to obtain the first coordinate of the position point X. Afterwards, combining the mapping relationship between the polar coordinate system and the Cartesian coordinate system, the first coordinate is converted into the coordinate value of the position point X in the Cartesian coordinate system to obtain the second coordinate of the position point X. After obtaining the second coordinates of 20*360=7200 position points, the second coordinates of the 7200 position points may be used as a coordinate mapping relationship matrix between the polar coordinate system and the Cartesian coordinate system.

[0089] Furthermore, the position feature processing module can also be used to obtain pixel features corresponding to the multiple position points in the image feature, and obtain pixel features of the multiple position points; according to the correspondence between the multiple position points and the eigenvalues ​​in the coordinate mapping relationship matrix, the sorting of each feature point in the pixel feature is deployed, so that each feature point is associated with the corresponding eigenvalue in the coordinate mapping relationship matrix, and the sorted pixel feature is the coordinate feature.

[0090] It can be seen that by adopting this implementation method, by using multi-layer convolution operations, the coordinate system of the image features is converted and mapped, and the image features are mapped to different position representation dimensions. Under the condition of simplifying the algorithm, the position representation of multiple dimensions of the obstacle grounding area can also be obtained, so that the obtained position representation has a finer granularity and relatively higher accuracy.

[0091] It should be pointed out that, affected by the field of view of the image acquisition device, the AVM image may present the scenery within a certain range around the vehicle. In some implementation scenarios, the grounding area of ​​the obstacle may intersect with the boundary of the AVM image, or the grounding area of ​​the obstacle may overlap with the edge of the AVM image. Based on this, the prediction module may also include a classification module, which is used to predict the type of object to be identified corresponding to each pixel feature in the contour feature, and the type of object to be identified is the grounding area of ​​the obstacle or the edge of the target image. Furthermore, if the type of object to be identified is the grounding area of ​​the obstacle, the fully connected layer can be called to predict the position of the grounding area of ​​the obstacle in the target image, including the position information of the first dimension and the position information of the second dimension.

[0092] In this way, when it is determined that it is the obstacle touchdown area, the position information of the obstacle touchdown area can be predicted, thereby further improving the accuracy of the position information of the obstacle touchdown area.

[0093] The following uses examples to introduce the composition of the boundary recognition model and the functions of each module in the boundary recognition model.

[0094] See also Figure 1C , Figure 1C An exemplary boundary recognition model based on a convolutional layer is shown, which may include: an EfficientNetLite network, an FPN, a position feature processing module, an encoding module, and a prediction module. Figure 1C The EfficientNetLite network in may include multiple convolutional layers, and the weights loaded by the multiple convolutional layers are different. The prediction module may be a fully connected layer and a classification module.

[0095] The following examples are combined with Figure 1C It is introduced from the perspective of the functions performed by each module.

[0096] For example, the AVM image of the vehicle is an image with a resolution of 896*896P. After the boundary recognition model receives the AVM image, the preprocessing module ( Figure 1C (not shown) 896*896P pixels can be processed according to the RGB three channels, for example, image data with RGB mean values ​​of [123.675, 116.28, 103.53] and RGB variances of [58.395, 57.12, 57.375] are obtained. Then, the RGB mean value [123.675, 116.28, 103.53] and the RGB variance [58.395, 57.12, 57.375] are input into the EfficientNetLite network.

[0097] The multiple convolutional layers in the EfficientNetLite network can process the image data separately, and each convolutional layer can extract an image sub-feature of the AVM image. The image sub-features extracted by the multiple convolutional layers correspond to AVM images of different resolutions. In this way, features of various sizes of the AVM image can be obtained. Furthermore, the EfficientNetLite network inputs the features of various sizes of the AVM image into the FPN.

[0098] FPN can fuse features of various sizes of AVM images, and the fused features are, for example, image features of AVM images. Afterwards, FPN transmits the image features to the position feature processing module. In this example, the image features include features of AVM images at various resolutions, thereby supporting the recognition of grounded areas of obstacles of various sizes. The size of the image features is, for example, 64 channels, and each channel has a resolution of 224*224: [64,224,224].

[0099] The position feature processing module can, for example, deploy a polar coordinate system and a Cartesian coordinate system according to the image features, establish a coordinate mapping relationship matrix between the polar coordinate system and the Cartesian coordinate system, and then obtain coordinate features according to the coordinate mapping relationship matrix and the image features.

[0100] For example, the position feature processing module can construct the features of the polar coordinate system according to the size of the image feature, for example, the features of the polar coordinate system are: [64, 20, 360], that is, 64 channels are also deployed in 360 radial directions, and 20 position points (also called sampling points) are sampled in each radial direction. The center point of the image feature can be set as the pole, and the length of the radius r can be set to w / 2, for example, where w refers to half of the polar coordinate angle.

[0101] Then, the Cartesian coordinates of the corresponding position point are calculated based on the polar coordinates of each position point. In the Cartesian coordinates of any position point, X=r.cos(θ)+W / 2, Y=r.sin(θ)+H / 2, where θ is the polar angle of the position point, that is, the angle w between the direction of the position point and the polar axis. W refers to the width of the image feature, H is the height of the image feature, r.cos(θ) refers to the coordinate value of the polar coordinate of the position point mapped to the x-axis, and r.sin(θ) refers to the coordinate value of the polar coordinate of the position point mapped to the y-axis. Then, the Cartesian coordinates of each position point are normalized, and the normalized coordinates of each position point form a matrix M. Exemplarily, the normalized coordinates of any position point can be expressed as x=X / W / 2, y=Y / H / 2.

[0102] Among them, the coordinate values ​​of each position point in the matrix M and the position feature processing module can be arranged according to the direction of the polar coordinate system and the sampling order in each direction.

[0103] Further, according to 360 directions and 20 position points in each direction, the image features are sampled to obtain 7200 pixel feature points, and the 7200 pixel feature points are arranged accordingly according to the distribution of the position points in the coordinate mapping relationship matrix M to obtain the coordinate features. Afterwards, the position feature processing module can transmit the coordinate features to the encoding module.

[0104] The encoding module may include multiple convolutional layers, so that multiple rounds of convolution operations can be performed on the coordinate features to extract richer features of the AVM image and obtain contour features. A contour feature is implemented as [32, 20, 360], where 32 refers to the number of channels, 360 refers to the direction, and 20 refers to the number of sampling points in each direction, which is also a height feature.

[0105] Furthermore, the encoding module transmits the contour features to the prediction module. After receiving the contour features [32, 20, 360], the prediction module can merge the channel features and the height features to retain the angle features corresponding to 360 directions. Afterwards, the angle features corresponding to the 360 ​​directions can be input into the classification module to predict whether the boundary pixel in each direction is an obstacle grounding area or the edge of the panoramic image. If any boundary pixel is an obstacle grounding area, the angle features corresponding to the 360 ​​directions are input into the fully connected layer to predict the distance between each obstacle grounding area and the center point and the angle in polar coordinates.

[0106] It should be understood that Figure 1C The illustrated boundary recognition model and the processing of each module in the boundary recognition model are all schematic descriptions and do not limit the boundary recognition model involved in the embodiments of the present application. In other embodiments, the operation process of each module and the output feature data may be different from the presentation of the above embodiment. This is not limited here.

[0107] In addition, it should be pointed out that Figure 1A , Figure 1B and Figure 1C They are different implementations of the boundary recognition model provided in the embodiments of the present application. Figure 1A , Figure 1B and Figure 1C The same or similar parts in the illustrated embodiments can be used as reference for each other and will not be described in detail here.

[0108] By calling Figures 1A to 1C For any boundary recognition model, the embodiment of the present application can also provide a recognition method.

[0109] See also Figure 2 , an identification method provided in an embodiment of the present application includes steps S101 to S103.

[0110] Step S101, extracting image features of a target image.

[0111] The target image includes an image of an obstacle contact area in the area where the vehicle is located.

[0112] In some embodiments, step S101 may be performed by calling a feature extraction module. In this embodiment, the feature extraction module may include a convolution extraction layer. In other embodiments, step S101 may be performed by calling a feature extraction module and a feature fusion module. In this embodiment, the feature extraction module may include at least two convolution extraction layers. Figure 1C The description of the function of the feature extraction module in is not repeated here.

[0113] Step S102: acquiring contour features of the obstacle ground contact area contained in the image features through multi-layer convolution operations of the boundary recognition model.

[0114] The contour features include pixel features and position features of the obstacle ground contact area.

[0115] In an embodiment of the present application, after acquiring the image features, the boundary recognition model can be called to construct a first coordinate system according to the size of the image features. The first coordinate system includes M directions, and the multiple position points are distributed in the M directions, wherein M is an integer greater than 3. Then, corresponding to any position point among the multiple position points, the coordinates of the position point in the first coordinate system are calculated to obtain the first coordinates of the position point. Afterwards, according to the first coordinates and the mapping relationship between the first coordinate system and the second coordinate system, the coordinates of the position point in the second coordinate system are calculated to obtain the second coordinates of the position point. The second coordinates of the multiple position points are used as eigenvalues ​​to generate the coordinate mapping relationship matrix.

[0116] The first coordinate system is used to characterize the position feature from a first dimension, and the second coordinate system is used to characterize the position feature from a second dimension. Exemplarily, the first coordinate system is, for example, a polar coordinate system, and the second coordinate system is, for example, a Cartesian coordinate system.

[0117] Among them, each eigenvalue in the coordinate mapping relationship matrix can represent the mapping relationship between the coordinates of a position point in the first coordinate system and the coordinates in the second coordinate system. Therefore, it can be considered that the coordinate mapping relationship matrix can represent the mapping relationship between the first coordinate system and the second coordinate system.

[0118] Furthermore, the pixel features corresponding to the multiple position points in the image features can be obtained to obtain the pixel features of the multiple position points. Then, according to the correspondence between the multiple position points and the eigenvalues ​​in the coordinate mapping relationship matrix, the sorting of each feature point in the pixel features is deployed so that each feature point is associated with the corresponding eigenvalue in the coordinate mapping relationship matrix, and the sorted pixel features are the coordinate features. Afterwards, the coordinate features can be encoded through a multi-layer convolution encoding operation, and the encoded features include the contour features, which may include features with higher dimensions than the coordinate features.

[0119] By adopting this implementation method, the image features are converted and mapped in the coordinate system, and the image features can be mapped to different position representation dimensions, so as to obtain the position representation of multiple dimensions of the obstacle grounding area, so that the obtained position representation has finer granularity and relatively higher accuracy.

[0120] In combination with the description of the boundary recognition model in the above embodiment, it can be known that the feature processing module in the boundary recognition model can be called to execute step S102. The feature processing module may include a position feature processing module and an encoding module. The role and operation performed by the position feature processing module and the encoding module in implementing the function of step S102 are detailed in 1A to Figure 1C The relevant description in will not be repeated here.

[0121] Step S103: predicting the position of the obstacle contact area in the target image according to the contour feature.

[0122] The contour formed by the position of the obstacle contact area is the boundary of the vehicle's drivable area.

[0123] In some embodiments, the position of the obstacle touchdown area may include position information of a first dimension and position information of a second dimension of the obstacle touchdown area. For example, when the first coordinate system is a polar coordinate system and the second coordinate system is a polar coordinate system, the position information of the first dimension may be an angle, and the position information of the second dimension may be a distance.

[0124] The prediction module in the boundary recognition model may be called to execute step S103. For the implementation of the prediction module executing the operation, please refer to Figures 1A to 1C The relevant description in will not be repeated here.

[0125] It should be noted that the boundary recognition model involved in step S101 to step S103 is an algorithm model that has been trained to be usable. Prior to this, the embodiment of the present application may also include an implementation process of training the model to be trained to obtain the boundary recognition model.

[0126] In some embodiments, the composition of the model to be trained can be Figures 1A to 1C Any one of the following. The parameters of each functional module in the model to be trained need to be adjusted so that the model after the adjustment of parameters can be used.

[0127] Exemplarily, the embodiment of the present application can use labeled sample images to train the to-be-trained model. The boundaries of the drivable area contained in the labeled sample images are labeled.

[0128] The training method may include:

[0129] Call the model to be trained to identify the sample image and obtain the position prediction result and the type prediction result. Calculate the smoothing loss function (smoothl1loss) and the cross entropy loss function (crossentropy loss), and determine whether the smoothing loss function and the cross entropy loss function converge. If both the smoothing loss function and the cross entropy loss function converge, use the model to be trained as the boundary recognition model; if the smoothing loss function and / or the cross entropy loss function do not converge, adjust the parameters of the model to be trained, and use the convolution model with adjusted parameters as the new model to be trained, and call the model to be trained again to identify the sample image.

[0130] Among them, the smooth loss function (smoothl1loss) is used to characterize the loss between the position prediction result and the labeled position information; the cross entropy loss function (crossentropy loss) is used to characterize the loss between the type prediction result and the labeled type information.

[0131] In summary, this implementation method uses multi-layer convolution operations, which is relatively simple in algorithm, low in overhead, and can shorten the delay compared to the existing multi-layer transformer module operation mechanism. By converting and mapping the image features to different position representation dimensions, the position representation of the obstacle grounding area in multiple dimensions can be obtained while simplifying the algorithm, so that the obtained position representation has finer granularity and relatively higher accuracy, thereby optimizing the recognition performance of the drivable area boundary.

[0132] The present application embodiment provides an electronic device, which can be used as the above-mentioned execution subject (such as a vehicle-mounted device). Figures 1A to 1C The electronic device can call the boundary recognition model to perform any boundary recognition model shown in Figure 2 identification method.

[0133] In some implementations, the functions, algorithms, etc. of the boundary recognition model may be implemented in the form of a computer program, and the electronic device may call the boundary recognition model to execute the corresponding computer program by a processor. Figure 2 For example, a schematic diagram of an electronic device such as Figure 3 As shown, the electronic device 3 includes: a processor 300, a memory 301, a bus 302 and a communication interface 303, wherein the processor 300, the communication interface 303 and the memory 301 are connected via the bus 302; the memory 301 stores a computer program that can be run on the processor 300, and when the processor 300 runs the computer program, the identification method provided in any of the aforementioned embodiments of the present application is executed.

[0134] The memory 301 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the device network element and at least one other network element is realized through at least one communication interface 303 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0135] The bus 302 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 301 is used to store programs, and the processor 300 executes the programs after receiving the execution instruction. The identification method disclosed in any implementation of the embodiment of the present application may be applied to the processor 300, or implemented by the processor 300.

[0136] The processor 300 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 300. The above processor 300 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a readily available programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 301, and the processor 300 reads the information in the memory 301 and completes the steps of the above method in combination with its hardware.

[0137] The electronic device provided in the embodiment of the present application and the identification method provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, operated or implemented therein.

[0138] The present application also provides a computer-readable storage medium corresponding to the identification method provided in the above embodiment. Figure 4 The computer-readable storage medium shown is a CD 30 on which a computer program (ie, a program product) is stored. When the computer program is run by a processor, the identification method provided by any of the aforementioned embodiments will be executed.

[0139] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0140] The computer-readable storage medium provided in the above-mentioned embodiments of the present application and the identification method provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0141] It should be noted that:

[0142] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known structures and technologies are not shown in detail so as not to obscure the understanding of this description.

[0143] Similarly, it should be understood that in order to streamline the present application and help understand one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be interpreted as reflecting the following schematic diagram: the claimed application requires more features than the features clearly stated in each claim. More specifically, as reflected in the claims below, the inventive aspects are less than all the features of the single embodiment disclosed above. Therefore, the claims following the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present application.

[0144] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present application and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

[0145] The above is only a preferred specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A boundary recognition model, characterized in that: Used to identify the boundary of the drivable area of ​​the vehicle, the boundary recognition model includes: a feature extraction module, a feature processing module and a prediction module, the feature processing module includes multiple convolution layers, wherein: The feature extraction module is used to extract image features of a target image, wherein the target image includes an image of an obstacle contact area in an area where the vehicle is located, and the contour of the obstacle contact area constitutes a boundary of the drivable area; The feature processing module is used to obtain the contour feature of the obstacle ground contact area contained in the image feature through the operation processing of the multiple convolutional layers, and the contour feature includes the pixel feature and the position feature of the obstacle ground contact area; The prediction module is used to predict the position of the obstacle contact area in the target image according to the contour feature, so as to use the contour formed by the position of the obstacle contact area as the boundary of the drivable area.

2. The boundary recognition model according to claim 1, characterized in that: The feature processing module includes a position feature processing module and a coding module. The position feature processing module is used to generate coordinate features of the target image, wherein the coordinate features are used to characterize the association between the image features and a first coordinate system and a second coordinate system, wherein the first coordinate system is used to characterize the position features from a first dimension, and the second coordinate system is used to characterize the position features from a second dimension; The encoding module is used to encode the coordinate features through the operation of multiple convolutional coding layers, the encoded features include the contour features, and the multiple convolutional coding layers belong to the multiple convolutional layers.

3. The boundary recognition model according to claim 2, characterized in that: The position feature processing module is further used to construct a coordinate mapping relationship matrix, wherein the coordinate mapping relationship refers to a mapping relationship between the first coordinate system and the second coordinate system, and the coordinate mapping relationship matrix includes coordinate mapping relationships of multiple position points, and the multiple position points are associated with the first coordinate system; The position feature processing module is also used to obtain pixel features corresponding to the multiple position points in the image feature to obtain pixel features of the multiple position points; according to the correspondence between the multiple position points and the eigenvalues ​​in the coordinate mapping relationship matrix, the sorting of each feature point in the pixel feature is deployed so that each feature point is associated with the corresponding eigenvalue in the coordinate mapping relationship matrix, and the sorted pixel feature is the coordinate feature.

4. The boundary recognition model according to claim 2 or 3, characterized in that: The position feature processing module is also used to construct the first coordinate system according to the size of the image feature, the first coordinate system including M directions, the multiple position points are distributed in the M directions, wherein M is an integer greater than 3; corresponding to any position point among the multiple position points, the coordinates of the position point in the first coordinate system are calculated to obtain the first coordinate of the position point; according to the first coordinate and the mapping relationship between the first coordinate system and the second coordinate system, the coordinates of the position point in the second coordinate system are calculated to obtain the second coordinate of the position point; and the second coordinates of the multiple position points are used as eigenvalues ​​to generate the coordinate mapping relationship matrix.

5. The boundary recognition model according to claim 1, characterized in that: In the case where the feature extraction module includes a convolution extraction layer, the convolution extraction layer is used to extract image features of the target image; When the feature extraction module includes at least two convolution extraction layers, the weights of the at least two convolution extraction layers are different; the at least two convolution extraction layers are respectively used to extract image sub-features of the target image, the sizes of the image features extracted by every two convolution extraction layers are different, and the size of the image sub-features extracted by any convolution extraction layer is related to the weight of the convolution extraction layer.

6. The boundary recognition model according to claim 5, characterized in that: In the case where the feature extraction module includes at least two convolution extraction layers, the boundary recognition model further includes a feature fusion module, wherein: The feature fusion module is used to fuse the image sub-features of the target image at different sizes, so as to input the fused features into the feature processing module.

7. The boundary recognition model according to claim 1, characterized in that: The prediction module includes a fully connected layer, The fully connected layer is used to predict the position of the obstacle ground contact area in the target image according to the contour feature, and the position of the obstacle ground contact area in the target image includes position information of a first dimension and position information of a second dimension.

8. The boundary recognition model according to claim 7, characterized in that: The prediction module also includes a classification module. The classification module is used to predict the type of the object to be identified corresponding to each pixel feature in the contour feature, and the type of the object to be identified is the obstacle grounding area or the edge of the target image.

9. A recognition method, characterized in that: The method is implemented by calling a boundary recognition model, and the method includes: Extracting image features of a target image, wherein the target image includes an image of an obstacle contact area in an area where the vehicle is located; Acquire, by multi-layer convolution operation of the boundary recognition model, contour features of the obstacle ground contact area contained in the image features, wherein the contour features include pixel features and position features of the obstacle ground contact area; The position of the obstacle contact area in the target image is predicted based on the contour features, and the contour formed by the position of the obstacle contact area is the boundary of the drivable area of ​​the vehicle.

10. The method according to claim 9, characterized in that Acquiring the contour features of the obstacle ground contact area contained in the image features, including: Generate a coordinate feature of the target image, wherein the coordinate feature is used to characterize an association relationship between the image feature and a first coordinate system and a second coordinate system, wherein the first coordinate system is used to characterize a position feature from a first dimension, and the second coordinate system is used to characterize a position feature from a second dimension; The coordinate features are encoded through multi-layer convolutional coding operations, and the encoded features include the contour features.

11. The method according to claim 10, characterized in that The generating of the coordinate features of the target image comprises: Constructing a coordinate mapping relationship matrix, wherein the coordinate mapping relationship refers to a mapping relationship between the first coordinate system and the second coordinate system, and the coordinate mapping relationship matrix includes coordinate mapping relationships of multiple position points, and the multiple position points are associated with the first coordinate system; Obtain pixel features corresponding to the multiple position points in the image features to obtain pixel features of the multiple position points; According to the correspondence between the multiple position points and the eigenvalues ​​in the coordinate mapping relationship matrix, the sorting of each feature point in the pixel feature is deployed so that each feature point is associated with the corresponding eigenvalue in the coordinate mapping relationship matrix, and the sorted pixel feature is the coordinate feature.

12. The method according to claim 11, characterized in that The step of constructing a coordinate mapping relationship matrix includes: Constructing the first coordinate system according to the size of the image feature, the first coordinate system includes M directions, the multiple position points are distributed in the M directions, wherein M is an integer greater than 3; Corresponding to any one of the multiple position points, calculating the coordinates of the position point in the first coordinate system to obtain a first coordinate of the position point; Calculating the coordinates of the position point in the second coordinate system according to the first coordinates and a mapping relationship between the first coordinate system and the second coordinate system to obtain the second coordinates of the position point; The coordinate mapping relationship matrix is ​​generated by taking the second coordinates of the plurality of position points as eigenvalues.

13. The method according to claim 9, characterized in that It also includes training the model to be trained to obtain the boundary recognition model, The training method comprises: Calling the model to be trained to identify the sample image to obtain a position prediction result and a type prediction result, wherein the boundary of the drivable area contained in the sample image has been marked; Calculating a smoothing loss function and a cross entropy loss function, wherein the smoothing loss function is used to characterize the loss between the position prediction result and the labeled position information, and the cross entropy loss function is used to characterize the loss between the type prediction result and the labeled type information; Determine whether the smoothing loss function and the cross entropy loss function converge; If both the smoothing loss function and the cross entropy loss function converge, the model to be trained is used as the boundary recognition model; If the smoothing loss function and / or the cross entropy loss function do not converge, adjust the parameters of the model to be trained, and use the convolution model with the adjusted parameters as the new model to be trained, and again call the model to be trained to identify the sample image.

14. An electronic device, characterized in that: The electronic device deploys the boundary recognition model described in any one of claims 1-8, and the electronic device calls the boundary recognition model to implement the method described in any one of claims 9-13.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the method according to any one of claims 9 to 13.

Images