Region boundary identification method and device, electronic equipment and storage medium

Abstract

The invention provides a region boundary recognition method and device, electronic equipment and a storage medium, the method is realized by calling a region boundary recognition model comprising a plurality of convolution layers, and the method comprises the steps: extracting image features from a panoramic image; performing first convolution operation through the plurality of convolution layers to construct a coordinate mapping relation matrix; performing second convolution operation on the image features and the coordinate mapping relation matrix through a plurality of convolution layers to generate coordinate features of the panoramic image; third convolution operation is carried out on the coordinate features through a plurality of convolution layers so as to identify the position information of the region boundary, and the position information comprises the position information of the first dimension and the position information of the second dimension of the region boundary. On one hand, recognition is carried out through the convolutional layer, the algorithm is simple, and the overhead is low; and on the other hand, the position identified by the method comprises the position information of two dimensions, and the identified granularity is relatively fine, so that the accuracy of the boundary information is improved.

Description

Technical Field

[0001] The present application belongs to the technical field of automatic parking, and specifically relates to a method, device, electronic device and storage medium for identifying area boundaries. Background Art

[0002] Automatic parking is an important research project in the field of autonomous driving. During the automatic parking process, the automatic parking system can identify the boundaries of the vehicle's drivable area (free space) based on the vehicle's panoramic image.

[0003] The automatic parking system can identify the boundary of the drivable area through a pre-deployed area boundary recognition model. The area boundary recognition model can include multiple neural network algorithm modules, and the area boundary recognition model can call the multiple neural network algorithm modules to process the panoramic image of the vehicle to identify the boundary of the drivable area based on the panoramic image of the vehicle.

[0004] However, the region boundary recognition model currently used not only has complex algorithms and high overhead, but also the accuracy of the recognized boundary information is poor. Summary of the invention

[0005] The present application proposes a region boundary recognition method, device, electronic device and storage medium, which can perform region boundary recognition with a relatively simple algorithm, and the granularity of the recognition result is fine, which is conducive to improving the accuracy of the recognized boundary information.

[0006] The first aspect of the present application provides a method for identifying a region boundary, which is implemented by calling a region boundary identification model, wherein the region boundary identification model includes multiple convolutional layers, and the method includes:

[0007] Extracting image features from the panoramic image, the image features including image features of the region boundary;

[0008] Performing a first convolution operation through the multiple convolution layers to construct a coordinate mapping relationship matrix, where the coordinate mapping relationship refers to a mapping relationship between the first coordinate system and the second coordinate system, and the coordinate mapping relationship matrix includes a coordinate mapping relationship of multiple position points, where the multiple position points are associated with the first coordinate system, the first coordinate system is used to represent position features from a first dimension, and the second coordinate system is used to represent position features from a second dimension;

[0009] Performing a second convolution operation on the image features and the coordinate mapping relationship matrix through the multiple convolution layers to generate coordinate features of the panoramic image, the coordinate features including pixel features of the multiple location points and coordinate mapping relationship features of the multiple location points;

[0010] A third convolution operation is performed on the coordinate features through the multiple convolution layers to identify position information of the region boundary, where the position information includes the position information of the first dimension and the position information of the second dimension of the region boundary.

[0011] In some embodiments of the present application, the multiple convolutional layers include a position feature processing convolutional layer, and the position feature processing convolutional layer includes multiple convolution operation modules; wherein,

[0012] Performing a first convolution operation through the multiple convolution layers to construct a coordinate mapping relationship matrix includes:

[0013] The position feature processing convolution layer is called to perform the following calculations to construct a coordinate mapping relationship matrix:

[0014] Constructing the first coordinate system according to the size of the image feature, the first coordinate system includes M directions, the multiple position points are distributed in the M directions, wherein M is an integer greater than 3;

[0015] Corresponding to any one of the multiple position points, calculating the coordinates of the position point in the first coordinate system to obtain a first coordinate of the position point;

[0016] Calculating the coordinates of the position point in the second coordinate system according to the first coordinates and a mapping relationship between the first coordinate system and the second coordinate system to obtain the second coordinates of the position point;

[0017] The coordinate mapping relationship matrix is ​​generated by taking the second coordinates of the plurality of position points as eigenvalues.

[0018] In some embodiments of the present application, the multiple convolutional layers further include a sampling convolutional layer, and the sampling convolutional layer includes multiple convolutional operation modules; wherein,

[0019] The multiple convolution layers perform a second convolution operation on the image features and the coordinate mapping relationship matrix to generate the coordinate features of the panoramic image, including:

[0020] Calling the sampling convolution layer to obtain pixel features corresponding to the multiple position points in the image features, so as to obtain pixel features of the multiple position points;

[0021] And according to the correspondence between the multiple position points and the eigenvalues ​​in the coordinate mapping relationship matrix, the sorting of each feature point in the pixel feature is deployed, so that each feature point is associated with the corresponding eigenvalue in the coordinate mapping relationship matrix, and the sorted pixel feature is the coordinate feature.

[0022] In some embodiments of the present application, the multiple convolutional layers further include a coding convolutional layer and a fully connected layer, and the coding convolutional layer includes multiple convolutional operation modules;

[0023] The performing a third convolution operation on the coordinate features through the multiple convolution layers to identify the position information of the region boundary includes:

[0024] Calling the encoding convolution layer to encode the coordinate features, where the encoded features include position features of the region boundary, and the position features include position features of the first dimension and position features of the second dimension;

[0025] The fully connected layer is called to identify the position information of the first dimension and the position information of the second dimension of the region boundary according to the position feature of the region boundary.

[0026] In some embodiments of the present application, the region boundary recognition model further includes a convolution extraction layer, and the step of extracting image features from the panoramic image includes:

[0027] The convolution extraction layer is called to extract image features of the panoramic image.

[0028] In some embodiments of the present application, the region boundary recognition model further includes at least two convolution extraction layers and a feature fusion layer, and the extracting of image features from the panoramic image includes:

[0029] Calling the at least two convolution extraction layers to respectively extract image sub-features of the panoramic image, wherein the sizes of the image features extracted by every two convolution extraction layers are different, and the size of the image sub-features extracted by any convolution extraction layer is related to the weight of the convolution extraction layer;

[0030] The feature fusion layer is called to fuse the image sub-features of the panoramic image at different sizes to obtain the image feature.

[0031] In some embodiments of the present application, the region boundary recognition model further includes a classification module, and before identifying the location information of the region boundary, further includes:

[0032] Calling the classification module to identify the type of object to be identified corresponding to each pixel feature in the position feature of the region boundary, the type of object to be identified being the region boundary or the edge of the panoramic image;

[0033] If the type of the object to be identified is the region boundary, the fully connected layer is called to identify the position information of the first dimension and the position information of the second dimension of the region boundary.

[0034] In some embodiments of the present application, the method further includes training the model to be trained to obtain the region boundary recognition model, and the training method includes:

[0035] Calling the model to be trained to identify the sample image to obtain a position prediction result and a type prediction result, wherein the boundary of the drivable area contained in the sample image has been marked;

[0036] Calculating a smoothing loss function and a cross entropy loss function, wherein the smoothing loss function is used to characterize the loss between the position prediction result and the labeled position information, and the cross entropy loss function is used to characterize the loss between the type prediction result and the labeled type information;

[0037] Determine whether the smoothing loss function and the cross entropy loss function converge;

[0038] If both the smoothing loss function and the cross entropy loss function converge, the model to be trained is used as the region boundary recognition model;

[0039] If the smoothing loss function and / or the cross entropy loss function do not converge, adjust the parameters of the model to be trained, and use the convolution model with the adjusted parameters as the new model to be trained, and again call the model to be trained to identify the sample image.

[0040] An embodiment of a second aspect of the present application provides a region boundary recognition device, deploying a region boundary recognition model, the region boundary recognition model including a plurality of convolutional layers, and the device including:

[0041] A feature extraction module, used to extract image features from the panoramic image, wherein the image features include image features of the region boundary;

[0042] a first convolution operation module, configured to perform a first convolution operation through the plurality of convolution layers to construct a coordinate mapping relationship matrix, wherein the coordinate mapping relationship refers to a mapping relationship between a first coordinate system and a second coordinate system, and the coordinate mapping relationship matrix includes a coordinate mapping relationship of a plurality of position points, and the plurality of position points are associated with the first coordinate system, the first coordinate system is used to represent position features from a first dimension, and the second coordinate system is used to represent position features from a second dimension;

[0043] a second convolution operation module, configured to perform a second convolution operation on the image features and the coordinate mapping relationship matrix through the multiple convolution layers to generate coordinate features of the panoramic image, wherein the coordinate features include pixel features of the multiple location points and coordinate mapping relationship features of the multiple location points;

[0044] An identification module is used to perform a third convolution operation on the coordinate features through the multiple convolution layers to identify the position information of the region boundary, where the position information includes the position information of the first dimension and the position information of the second dimension of the region boundary.

[0045] An embodiment of the third aspect of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect above.

[0046] An embodiment of the fourth aspect of the present application provides a computer-readable storage medium on which a computer program is stored, and the program is executed by a processor to implement the method described in the first aspect above.

[0047] The technical solution provided in the embodiments of the present application has at least the following technical effects or advantages:

[0048] In an embodiment of the present application, the regional boundary recognition is performed by calling the regional boundary recognition model, and the regional boundary recognition model includes multiple convolution layers. Based on this, in an embodiment of the present application, after extracting image features from the panoramic image, a first convolution operation is performed through multiple convolution layers to construct a coordinate mapping relationship matrix; a second convolution operation is performed on the image features and the coordinate mapping relationship matrix through multiple convolution layers to generate the coordinate features of the panoramic image; and a third convolution operation is performed on the coordinate features through multiple convolution layers to identify the position information of the regional boundary. That is to say, in the embodiment of the present application, multi-layer convolution operations are used to process image features. Compared with the operation mechanism of the existing multi-layer transformer module, not only is the algorithm relatively simple, the overhead is small, but also the delay can be shortened. Further, the coordinate mapping relationship matrix in the embodiment of the present application represents the mapping relationship between the first coordinate system and the second coordinate system. The first coordinate system is used to represent the position feature from the first dimension, and the second coordinate system is used to represent the position feature from the second dimension. Correspondingly, the position information of the regional boundary includes the position information of the first dimension and the position information of the second dimension of the regional boundary. It can be seen that in the embodiment of the present application, the image features are mapped to different position representation dimensions, so that the position representation of multiple dimensions of the region boundary can be obtained, so that the obtained position representation has a finer granularity and relatively higher accuracy.

[0049] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] By reading the detailed description of the preferred embodiment below, various other advantages and benefits will become clear to those of ordinary skill in the art. The accompanying drawings are only used for the purpose of illustrating the preferred embodiment and are not considered to be limitations of the present application. In addition, the same reference symbols are used to represent the same components throughout the accompanying drawings.

[0051] In the attached picture:

[0052] Figure 1 A schematic diagram showing the composition of a region boundary recognition model provided by an embodiment of the present application is shown;

[0053] Figure 2 A method flow chart of a method for identifying a region boundary provided by an embodiment of the present application is shown;

[0054] Figure 3 A schematic diagram showing the data flow of a region boundary recognition model provided by an embodiment of the present application is shown;

[0055] Figure 4 A schematic diagram of the structure of a region boundary recognition device provided by an embodiment of the present application is shown;

[0056] Figure 5 A schematic diagram of the structure of an electronic device provided by an embodiment of the present application is shown;

[0057] Figure 6 A schematic diagram of a storage medium provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0058] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0059] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by technicians in the field to which this application belongs.

[0060] The embodiments of the present application relate to automatic parking scenarios. In the automatic parking scenarios, after a vehicle enters an area to be parked, the automatic parking system can identify the boundary of the vehicle's drivable area (free space) in the area to be parked, and a parking control scheme can be planned based on the boundary of the free space. In some embodiments, the automatic parking system can identify the grounding position of an obstacle on an image of an Around View Monitor (AVM), and the boundary formed by the contour line of the grounding position of the obstacle can be considered as the free space boundary.

[0061] The automatic parking system can call the pre-deployed algorithm model to process the AVM image to obtain the features of the obstacle ground image contained in the AVM image, and then identify the location of the area boundary according to the features of the obstacle ground image. In the embodiment of the present application, the pre-deployed algorithm model is referred to as the area boundary recognition model.

[0062] Some conventional regional boundary recognition models for identifying free space boundaries, after obtaining an AVM image, for example, map the AVM image to a polar coordinate system, so that the pixels in the AVM image have polar coordinate position features. After extracting the image features that can represent the free space boundary, the included multi-layer transformer module is called to calculate and process the image features to obtain predictable features, and then identify the free space boundary from the predictable features. In this way, on the one hand, the AVM image is mapped to a coordinate system, so that the dimension represented by the features of the obstacle grounding position is relatively single, resulting in the identified obstacle grounding position features with relatively large granularity and relatively low accuracy. On the other hand, the multi-layer transformer module uses an attention mechanism for calculation, which is relatively computationally complex, has high overhead, and has a large delay, resulting in poor recognition performance of the drivable area boundary and poor user experience.

[0063] In view of this, the technical solution provided by the embodiment of the present application, the region boundary recognition model used includes multiple convolution layers. After extracting image features from the panoramic image, different convolution operations of multiple convolution layers are performed: constructing a coordinate mapping relationship matrix, generating coordinate features of the panoramic image, identifying the location information of the region boundary, and other operations. That is, the image features are processed by multi-layer convolution operations, which not only has a relatively simple algorithm and low overhead, but also can shorten the delay. In addition, the coordinate mapping relationship matrix represents the mapping relationship between the first coordinate system and the second coordinate system. The first coordinate system is used to represent the position feature from the first dimension, and the second coordinate system is used to represent the position feature from the second dimension. In this way, the coordinate feature can include the position feature of the first dimension and the position feature of the second dimension, so that the position information of the region boundary identified from the coordinate feature can include the position information of the first dimension and the position information of the second dimension. It can be seen that by adopting this implementation method, the image features are converted and mapped by the coordinate system, and the image features are mapped to different position representation dimensions, so that the position representation of multiple dimensions of the region boundary can be obtained, so that the obtained position representation has a finer granularity and a relatively higher accuracy.

[0064] The execution subject of this technical solution can be any device that supports regional boundary recognition, including vehicles, ships, aircraft or robots. Such devices can support regional boundary recognition of the embodiment of this application through an automatic parking system. The automatic parking system can be provided with relevant functional mechanisms such as image acquisition and image processing.

[0065] The following describes a method, device, electronic device and storage medium for identifying region boundaries according to an embodiment of the present application in conjunction with the accompanying drawings.

[0066] See also Figure 1 , Figure 1 A region boundary recognition model provided by an embodiment of the present application is shown. According to the data flow direction, the region boundary recognition model may include: a model backbone, a model neck, and a model head.

[0067] The backbone model can be used to extract image features. The neck model can be used to process image features. The head model can be used to identify and predict processed image features. In this embodiment, the neck model can include multiple convolutional layers.

[0068] It should be understood that the model neck can perform multiple processing. Accordingly, the multiple convolutional layers included in the model neck can support multiple convolutional operations. There can also be multiple convolutional layers supporting each convolutional operation, and the functions corresponding to each convolutional operation can be different. For example, the multiple convolutional layers can include a sampling convolutional layer, an encoding convolutional layer, etc.

[0069] The following combination Figure 1 The region boundary recognition model shown introduces the region boundary recognition method of the embodiment of the present application.

[0070] The region boundary recognition method of the embodiment of the present application can be performed by calling Figure 1 This is achieved by using the illustrated regional boundary recognition model. Figure 2 As shown, the region boundary recognition method may include steps S101-S104.

[0071] Step S101: extracting image features from the panoramic image.

[0072] The panoramic image is, for example, an AVM image. Then, the image feature includes the image feature of the region boundary.

[0073] In some embodiments, you can call Figure 1 The model backbone in step S101 performs the operation. The model backbone can be implemented as any algorithm module that supports extracting image features from AVM images, such as but not limited to convolutional neural network (CNN), EfficientNetLite network, etc.

[0074] In addition, when the model backbone is an EfficientNetLite network, the weights of the EfficientNetLite can be pre-trained on the imagenet dataset. The weight size of EfficientNetLite is related to the size of the image features extracted by the EfficientNetLite network.

[0075] In some embodiments, the model backbone may include at least one convolutional extraction layer.

[0076] Exemplarily, if the model backbone includes a convolutional extraction layer, the image features of the panoramic image can be extracted by calling the convolutional extraction layer.

[0077] Exemplarily, if the model backbone includes at least two convolution extraction layers, the weights of the at least two convolution extraction layers are different. In this example, the region boundary recognition model may also include a feature fusion layer. In this example, the at least two convolution extraction layers may be called to respectively extract image sub-features of the panoramic image, and the sizes of the image features extracted by each two convolution extraction layers are different. The size of the image sub-features extracted by any convolution extraction layer is related to the weight of the convolution extraction layer. Furthermore, the feature fusion layer is called to fuse the image sub-features of the panoramic image at different sizes to obtain the image features.

[0078] Exemplarily, the feature fusion layer may be implemented as a feature pyramid network (FPN).

[0079] Step S102: performing a first convolution operation through the multiple convolution layers to construct a coordinate mapping relationship matrix.

[0080] Among them, the coordinate mapping relationship refers to the mapping relationship between the first coordinate system and the second coordinate system, the coordinate mapping relationship matrix includes the coordinate mapping relationship of multiple position points, the multiple position points are associated with the first coordinate system, the first coordinate system is used to represent the position characteristics from the first dimension, and the second coordinate system is used to represent the position characteristics from the second dimension.

[0081] Exemplarily, the multiple position points may be sampling points pre-set in the process of constructing the first coordinate system, and these sampling points may be used to sample and arrange pixels corresponding to the image features in a certain order.

[0082] In some embodiments, the plurality of convolutional layers may include a position feature processing convolutional layer, which includes a plurality of convolutional operation modules, and the convolutional layers may perform a first convolution operation to construct a coordinate mapping relationship matrix.

[0083] Exemplarily, during the execution of the first convolution operation, a first coordinate system can be constructed according to the size of the image feature, and the first coordinate system includes M directions, and the multiple position points are distributed in the M directions, wherein M is an integer greater than 3. For any position point among the multiple position points, the coordinates of the position point in the first coordinate system can be calculated to obtain the first coordinate of the position point. Furthermore, according to the first coordinate and the mapping relationship between the first coordinate system and the second coordinate system, the coordinates of the position point in the second coordinate system can be calculated to obtain the second coordinate of the position point. In some embodiments, the second coordinate of the position point is obtained by converting the first coordinate and the mapping relationship, and the second coordinate can characterize the mapping relationship between the first coordinate system and the second coordinate system of the position point. Based on this, after obtaining the first coordinates and the second coordinates of all position points, the second coordinates of the multiple position points can be used as eigenvalues ​​to generate the coordinate mapping relationship matrix.

[0084] In some embodiments, the first coordinate system may be, for example, a polar coordinate system, M may be, for example, 360, and 20 position points may be deployed in each of the 360 ​​directions, for example. The 20 position points in each direction may be sampling points in that direction. The first coordinate system may be, for example, a Cartesian coordinate system. In this way, the first coordinate system may represent the angle in the position information, and the second coordinate system may represent the distance in the position information. Furthermore, for any position point X, the coordinate value of the position point X in the polar coordinate system may be calculated to obtain the first coordinate of the position point X. Afterwards, combining the mapping relationship between the polar coordinate system and the Cartesian coordinate system, the first coordinate is converted into the coordinate value of the position point X in the Cartesian coordinate system to obtain the second coordinate of the position point X. After obtaining the second coordinates of 20*360=7200 position points, the second coordinates of the 7200 position points may be used as a coordinate mapping relationship matrix between the polar coordinate system and the Cartesian coordinate system.

[0085] Exemplarily, a feature column in the coordinate mapping relationship matrix may correspond to one of the 360 ​​directions, and any two adjacent columns are two adjacent directions in the polar coordinate system. A feature row is a position point in the polar coordinate system from the pole to the radiation direction. That is, the matrix may be a feature matrix of 360 columns*20 rows.

[0086] It should be understood that the above introduction to the polar coordinate system is a schematic description and does not constitute a limitation on the first coordinate system involved in the embodiment of the present application. In other embodiments, the polar coordinate system can be deployed in 270 or 180 directions, and the position points in each direction can be deployed more or less. This is not limited here.

[0087] Step S103: performing a second convolution operation on the image features and the coordinate mapping relationship matrix through the multiple convolution layers to generate coordinate features of the panoramic image.

[0088] The coordinate features include pixel features of the multiple location points and coordinate mapping relationship features of the multiple location points.

[0089] Furthermore, the multiple convolution layers also include a sampling convolution layer, and the sampling convolution layer includes multiple convolution operation modules. After the position feature processing convolution layer processes to obtain a coordinate mapping relationship matrix, the coordinate mapping relationship matrix and the image features obtained in step S101 can be input into the sampling convolution layer, and the sampling convolution layer can perform a second convolution operation to generate the coordinate features of the panoramic image.

[0090] During the second convolution operation, the pixel features corresponding to the multiple position points in the image features can be triggered, that is, the image features are sampled according to the pre-deployed multiple position points to obtain the pixel features of the multiple position points. Combined with the aforementioned relationship between the multiple position points and the polar coordinates, it can be known that the multiple position points are in a one-to-one correspondence with the positions indicated by the eigenvalues ​​in the coordinate mapping relationship matrix. Then, the image features of each position point correspond to the corresponding eigenvalues ​​in the coordinate mapping relationship matrix. To this end, the sorting of each feature point in the pixel feature can be deployed according to the correspondence between the multiple position points and the eigenvalues ​​in the coordinate mapping relationship matrix, so that each feature point is associated with the corresponding eigenvalue in the coordinate mapping relationship matrix. The sorted pixel features can include each feature point, as well as the position features of each feature point in the first coordinate system and the second coordinate system. The sorted pixel features can be considered as the coordinate features.

[0091] Still taking the example that the first coordinate system is a polar coordinate system, 20 positions (i.e., sampling points) are deployed in each of the 360 ​​directions, and the second coordinate system is a Cartesian coordinate system. In this example, the sampling convolution layer can sample the image features according to the 20 positions in each direction, sample the pixel feature points corresponding to the 20 positions in each direction, and obtain the pixel features composed of 7200 pixel feature points. Afterwards, the 7200 pixel feature points can be arranged accordingly according to the distribution of each direction and the 20 positions in each direction to form the sampled pixel features.

[0092] Exemplarily, corresponding to the aforementioned coordinate mapping relationship matrix, the pixel feature can be a feature matrix of 360 columns * 20 rows, wherein each pixel feature point corresponds to a coordinate mapping relationship of a position point. In view of this, the position feature of each feature point of the pixel feature can be characterized by the corresponding coordinate mapping relationship.

[0093] It can be seen that by adopting this implementation method, two coordinate systems can be constructed for image features, and the two coordinate systems represent position features from different dimensions respectively, and the two coordinate systems can be mapped to each other. After establishing the correspondence between the sampling points of the image features and the two coordinate systems, the sampled pixel features can have the position features corresponding to the two coordinate systems, so that the position representation of the region boundary in multiple dimensions can be obtained.

[0094] Step S104: performing a third convolution operation on the coordinate features through the multiple convolution layers to identify position information of the region boundary.

[0095] The location information includes location information of the first dimension and location information of the second dimension of the region boundary.

[0096] Exemplarily, if the first coordinate system is a polar coordinate system, the position information of the first dimension may be the angle between the region boundary and the center position of the panoramic image (also referred to as angle information). If the second coordinate system is a Cartesian coordinate system, the position information of the second dimension may be the distance between the region boundary and the center of the panoramic image.

[0097] In this way, the location information extracted from the feature data containing location features of two dimensions can contain location information of two dimensions, so that the obtained location representation has finer granularity and relatively higher accuracy.

[0098] In some implementations, the multiple convolutional layers further include a coding convolutional layer. The region boundary recognition model further includes a fully connected layer, which belongs to the model head.

[0099] Exemplarily, after obtaining the coordinate features, the encoding convolution layer may be called to perform a third convolution operation on the coordinate features, and the third convolution operation may be an encoding operation, and the encoded features include the position features of the region boundary. The encoded features may include higher-dimensional features of the region boundary, and the position features include the position features of the first dimension and the position features of the second dimension. Furthermore, the fully connected layer may be called to identify the position information of the first dimension and the position information of the second dimension of the region boundary based on the position features of the region boundary.

[0100] It should be noted that, affected by the field of view of the image acquisition device, in some implementation scenarios, the region boundary may intersect with the AVM image boundary, or the region boundary may coincide with the AVM image edge. Based on this, in other embodiments, the region boundary recognition model may also include a classification module, which belongs to the model head.

[0101] After obtaining the coordinate features, the classification module can be called for any direction in the M directions to determine the type of object to be identified corresponding to the direction according to the encoded features. The type of object to be identified here is the region boundary or the edge of the panoramic image. If the type of object to be identified corresponding to the direction is the region boundary, the fully connected layer can be further called to identify the location information of the region boundary. In this way, the accuracy of the location information of the region boundary can be further improved.

[0102] It should be pointed out that Figure 1 and Figure 2 The region boundary recognition model involved in is an algorithm model that has been trained to be usable. Prior to this, the embodiment of the present application may also include an implementation process of training the model to be trained to obtain the region boundary recognition model.

[0103] In some embodiments, the composition of the model to be trained can be Figure 1 The parameters of each functional module in the model to be trained need to be adjusted so that the model after the adjustment of parameters can be used.

[0104] Exemplarily, the embodiment of the present application can use labeled sample images to train the to-be-trained model. The boundaries of the drivable area contained in the labeled sample images are labeled.

[0105] The training method may include:

[0106] Call the model to be trained to identify the sample image and obtain the position prediction result and the type prediction result. Calculate the smoothing loss function (smoothl1loss) and the cross entropy loss function (crossentropy loss), and determine whether the smoothing loss function and the cross entropy loss function converge. If both the smoothing loss function and the cross entropy loss function converge, use the model to be trained as the boundary recognition model; if the smoothing loss function and / or the cross entropy loss function do not converge, adjust the parameters of the model to be trained, and use the convolution model with adjusted parameters as the new model to be trained, and call the model to be trained again to identify the sample image.

[0107] Among them, the smooth loss function (smoothl1loss) is used to characterize the loss between the position prediction result and the labeled position information; the cross entropy loss function (crossentropy loss) is used to characterize the loss between the type prediction result and the labeled type information.

[0108] In summary, this implementation method uses multi-layer convolution operations, which is relatively simple in algorithm, low in overhead, and can shorten the delay compared to the existing multi-layer transformer module operation mechanism. By converting and mapping the image features to different position representation dimensions, the position representation of the region boundary in multiple dimensions can be obtained while simplifying the algorithm, so that the obtained position representation has finer granularity and relatively higher accuracy, thereby optimizing the recognition performance of the drivable region boundary.

[0109] The following introduces the region boundary recognition method of the embodiment of the present application through an exemplary region boundary recognition model and examples.

[0110] See also Figure 3 , Figure 3 An exemplary data flow diagram of a region boundary identification model is shown. Figure 3The model used to perform the region boundary recognition method includes, for example, an EfficientNetLite network, FPN, a position feature processing module, an encoding module, and a prediction module. Among them, the EfficientNetLite network, FPN, the position feature processing module, and the encoding module may each include multiple convolutional layers. The prediction module may include a fully connected layer and a classification module.

[0111] Exemplarily, the EfficientNetLite network may include multiple convolutional layers, each of which may be loaded with pre-trained weights, so that the corresponding convolutional layer has the ability to extract image features. In this example, the weights loaded by the multiple convolutional layers are different, so that different convolutional layers can be used to extract features of images of different sizes (e.g., resolutions). The pre-trained weights can be obtained by pre-training on the imagenet dataset.

[0112] The EfficientNetLite network can be Figure 1 The backbone of the region boundary recognition model shown in the figure; FPN, position feature processing module and encoding module can be composed Figure 1 The neck of the region boundary recognition model shown; the prediction module can be Figure 1 The region boundaries shown identify the head of the model.

[0113] The following examples are combined with Figure 3 From the perspective of the actions performed by each network in the present application, the region boundary recognition method of the embodiment of the present application is introduced.

[0114] For example, the AVM image of the vehicle is an image with a resolution of 896*896P. After the region boundary recognition model receives the AVM image, the preprocessing module ( Figure 3 (not shown) 896*896P pixels can be processed according to the RGB three channels, for example, image data with RGB mean values ​​of [123.675, 116.28, 103.53] and RGB variances of [58.395, 57.12, 57.375] are obtained. Then, the RGB mean value [123.675, 116.28, 103.53] and the RGB variance [58.395, 57.12, 57.375] are input into the EfficientNetLite network.

[0115] The multiple convolutional layers in the EfficientNetLite network can process the image data separately, and each convolutional layer can extract an image sub-feature of the AVM image. The image sub-features extracted by the multiple convolutional layers correspond to AVM images of different resolutions. In this way, features of various sizes of the AVM image can be obtained. Furthermore, the EfficientNetLite network inputs the features of various sizes of the AVM image into the FPN.

[0116] FPN can fuse features of various sizes of AVM images, and the fused features are, for example, image features of AVM images. Afterwards, FPN transmits the image features to the position feature processing module. In this example, the image features include features of AVM images at various resolutions, thereby supporting the recognition of boundary areas of various sizes. The size of the image features is, for example, 64 channels, and each channel has a resolution of 224*224: [64,224,224].

[0117] It should be noted that in some other embodiments, if the EfficientNetLite network includes a convolution layer, then the image features output by the EfficientNetLite network are features of an AVM image of one size, and feature fusion is not required. The model corresponding to this embodiment may not include FPN, and the EfficientNetLite network may input the extracted image features into the position feature processing module.

[0118] The position feature processing module can, for example, deploy a polar coordinate system and a Cartesian coordinate system according to the image features, establish a coordinate mapping relationship matrix between the polar coordinate system and the Cartesian coordinate system, and then obtain coordinate features according to the coordinate mapping relationship matrix and the image features.

[0119] For example, the position feature processing module can construct the features of the polar coordinate system according to the size of the image feature, for example, the features of the polar coordinate system are: [64, 20, 360], that is, 64 channels are also deployed in 360 radial directions, and 20 position points (also called sampling points) are sampled in each radial direction. The center point of the image feature can be set as the pole, and the length of the radius r can be set to w / 2, for example, where w refers to half of the polar coordinate angle.

[0120] Then, the Cartesian coordinates of the corresponding position point are calculated based on the polar coordinates of each position point. In the Cartesian coordinates of any position point, X=r.cos(θ)+W / 2, Y=r.sin(θ)+H / 2, where θ is the polar angle of the position point, that is, the angle w between the direction of the position point and the polar axis. W refers to the width of the image feature, H is the height of the image feature, r.cos(θ) refers to the coordinate value of the polar coordinate of the position point mapped to the x-axis, and r.sin(θ) refers to the coordinate value of the polar coordinate of the position point mapped to the y-axis. Then, the Cartesian coordinates of each position point are normalized, and the normalized coordinates of each position point form a matrix M. Exemplarily, the normalized coordinates of any position point can be expressed as x=X / W / 2, y=Y / H / 2.

[0121] Among them, the coordinate values ​​of each position point in the matrix M and the position feature processing module can be arranged according to the direction of the polar coordinate system and the sampling order in each direction.

[0122] Further, according to 360 directions and 20 position points in each direction, the image features are sampled to obtain 7200 pixel feature points, and the 7200 pixel feature points are arranged accordingly according to the distribution of the position points in the coordinate mapping relationship matrix M to obtain the coordinate features. Afterwards, the position feature processing module can transmit the coordinate features to the encoding module.

[0123] The encoding module may include multiple convolutional layers, so that multiple rounds of convolution operations can be performed on the coordinate features to extract richer features of the AVM image and obtain contour features. A contour feature is implemented as [32, 20, 360], where 32 refers to the number of channels, 360 refers to the direction, and 20 refers to the number of sampling points in each direction, which is also a height feature.

[0124] Furthermore, the encoding module transmits the contour features to the prediction module. After receiving the contour features [32, 20, 360], the prediction module can merge the channel features and the height features to retain the angle features corresponding to 360 directions. Afterwards, the angle features corresponding to the 360 ​​directions can be input into the classification module to predict whether the boundary pixels in each direction are the region boundaries or the edges of the panoramic image. If any boundary pixel is a region boundary, the angle features corresponding to the 360 ​​directions are input into the fully connected layer to predict the distance between each region boundary and the center point and the angle in polar coordinates.

[0125] It should be understood that Figure 3 The illustrated region boundary recognition model and the processing of each module in the region boundary recognition model are all schematic descriptions and do not constitute a limitation on the region boundary recognition model involved in the embodiments of the present application. In other embodiments, the operation process of each module and the output feature data may be different from the presentation of the above embodiment. This is not limited here.

[0126] In an embodiment of the present application, the regional boundary recognition is performed by calling the regional boundary recognition model, and the regional boundary recognition model includes multiple convolution layers. Based on this, in an embodiment of the present application, after extracting image features from the panoramic image, a first convolution operation is performed through multiple convolution layers to construct a coordinate mapping relationship matrix; a second convolution operation is performed on the image features and the coordinate mapping relationship matrix through multiple convolution layers to generate the coordinate features of the panoramic image; and a third convolution operation is performed on the coordinate features through multiple convolution layers to identify the position information of the regional boundary. That is to say, in the embodiment of the present application, multi-layer convolution operations are used to process image features. Compared with the operation mechanism of the existing multi-layer transformer module, not only is the algorithm relatively simple, the overhead is small, but also the delay can be shortened. Further, the coordinate mapping relationship matrix in the embodiment of the present application represents the mapping relationship between the first coordinate system and the second coordinate system. The first coordinate system is used to represent the position feature from the first dimension, and the second coordinate system is used to represent the position feature from the second dimension. Correspondingly, the position information of the regional boundary includes the position information of the first dimension and the position information of the second dimension of the regional boundary. It can be seen that in the embodiment of the present application, the image features are mapped to different position representation dimensions, so that the position representation of multiple dimensions of the region boundary can be obtained, so that the obtained position representation has a finer granularity and relatively higher accuracy.

[0127] The present application also provides a region boundary recognition device, in which a region boundary recognition model is deployed, and the region boundary recognition model includes multiple convolutional layers. The region boundary recognition device is used to execute the region boundary recognition method provided in any of the above embodiments. Figure 4 As shown, the device comprises:

[0128] A feature extraction module 401 is used to extract image features from a panoramic image, wherein the image features include image features of the region boundary; a first convolution operation module 402 is used to perform a first convolution operation through the multiple convolution layers to construct a coordinate mapping relationship matrix, wherein the coordinate mapping relationship refers to a mapping relationship between a first coordinate system and a second coordinate system, wherein the coordinate mapping relationship matrix includes a coordinate mapping relationship of multiple position points, wherein the multiple position points are associated with the first coordinate system, wherein the first coordinate system is used to represent position features from a first dimension, and wherein the second coordinate system is used to represent position features from a second dimension; a second convolution operation module 403 is used to perform a second convolution operation on the image features and the coordinate mapping relationship matrix through the multiple convolution layers to generate coordinate features of the panoramic image, wherein the coordinate features include pixel features of the multiple position points and coordinate mapping relationship features of the multiple position points; an identification module 404 is used to perform a third convolution operation on the coordinate features through the multiple convolution layers to identify position information of the region boundary, wherein the position information includes position information of the first dimension and position information of the second dimension of the region boundary.

[0129] The region boundary identification device provided in the embodiment of the present application and the region boundary identification method provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, operated or implemented therein.

[0130] The present application also provides an electronic device to perform the above-mentioned area boundary recognition method. Figure 5 It shows a schematic diagram of an electronic device provided by some embodiments of the present application. Figure 5 As shown, the electronic device 5 includes: a processor 500, a memory 501, a bus 502 and a communication interface 503, wherein the processor 500, the communication interface 503 and the memory 501 are connected via the bus 502; the memory 501 stores a computer program that can be run on the processor 500, and when the processor 500 runs the computer program, the area boundary recognition method provided in any of the aforementioned embodiments of the present application is executed.

[0131] The memory 501 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the device network element and at least one other network element is realized through at least one communication interface 503 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0132] The bus 502 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 501 is used to store programs, and the processor 500 executes the programs after receiving execution instructions. The region boundary recognition method disclosed in any implementation of the aforementioned embodiment of the present application may be applied to the processor 500, or implemented by the processor 500.

[0133] The processor 500 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 500. The above processor 500 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 501, and the processor 500 reads the information in the memory 501 and completes the steps of the above method in combination with its hardware.

[0134] The electronic device provided in the embodiment of the present application and the area boundary recognition method provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, operated or implemented therein.

[0135] The present application also provides a computer-readable storage medium corresponding to the region boundary recognition method provided in the above embodiment. Figure 6 The computer-readable storage medium shown is a CD 30 on which a computer program (ie, a program product) is stored. When the computer program is run by a processor, the region boundary recognition method provided by any of the aforementioned embodiments will be executed.

[0136] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0137] The computer-readable storage medium provided in the above-mentioned embodiments of the present application and the area boundary identification method provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the method adopted, run or implemented by the application program stored therein.

[0138] It should be noted that:

[0139] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known structures and technologies are not shown in detail so as not to obscure the understanding of this description.

[0140] Similarly, it should be understood that in order to streamline the present application and help understand one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be interpreted as reflecting the following schematic diagram: the claimed application requires more features than the features clearly stated in each claim. More specifically, as reflected in the claims below, the inventive aspects are less than all the features of the single embodiment disclosed above. Therefore, the claims following the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present application.

[0141] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present application and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

[0142] The above is only a preferred specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for identifying region boundaries, characterized in that: This is achieved by calling a region boundary recognition model, wherein the region boundary recognition model includes a plurality of convolutional layers, and the method includes: Extracting image features from the panoramic image, the image features including image features of the region boundary; Performing a first convolution operation through the multiple convolution layers to construct a coordinate mapping relationship matrix, where the coordinate mapping relationship refers to a mapping relationship between the first coordinate system and the second coordinate system, and the coordinate mapping relationship matrix includes a coordinate mapping relationship of multiple position points, where the multiple position points are associated with the first coordinate system, the first coordinate system is used to represent position features from a first dimension, and the second coordinate system is used to represent position features from a second dimension; Performing a second convolution operation on the image features and the coordinate mapping relationship matrix through the multiple convolution layers to generate coordinate features of the panoramic image, the coordinate features including pixel features of the multiple location points and coordinate mapping relationship features of the multiple location points; A third convolution operation is performed on the coordinate features through the multiple convolution layers to identify position information of the region boundary, where the position information includes the position information of the first dimension and the position information of the second dimension of the region boundary.

2. The method according to claim 1, characterized in that The multiple convolutional layers include a position feature processing convolutional layer, and the position feature processing convolutional layer includes multiple convolution operation modules; wherein, Performing a first convolution operation through the multiple convolution layers to construct a coordinate mapping relationship matrix includes: The position feature processing convolution layer is called to perform the following calculations to construct a coordinate mapping relationship matrix: Constructing the first coordinate system according to the size of the image feature, the first coordinate system includes M directions, the multiple position points are distributed in the M directions, wherein M is an integer greater than 3; Corresponding to any one of the multiple position points, calculating the coordinates of the position point in the first coordinate system to obtain a first coordinate of the position point; Calculating the coordinates of the position point in the second coordinate system according to the first coordinates and a mapping relationship between the first coordinate system and the second coordinate system to obtain the second coordinates of the position point; The coordinate mapping relationship matrix is ​​generated by taking the second coordinates of the plurality of position points as eigenvalues.

3. The method according to claim 2, characterized in that The multiple convolutional layers further include a sampling convolutional layer, and the sampling convolutional layer includes multiple convolutional operation modules; wherein, The multiple convolution layers perform a second convolution operation on the image features and the coordinate mapping relationship matrix to generate the coordinate features of the panoramic image, including: Calling the sampling convolution layer to obtain pixel features corresponding to the multiple position points in the image features, so as to obtain pixel features of the multiple position points; And according to the correspondence between the multiple position points and the eigenvalues ​​in the coordinate mapping relationship matrix, the sorting of each feature point in the pixel feature is deployed, so that each feature point is associated with the corresponding eigenvalue in the coordinate mapping relationship matrix, and the sorted pixel feature is the coordinate feature.

4. The method according to claim 2, characterized in that: The region boundary recognition model further includes a coding convolution layer and a fully connected layer, the coding convolution layer belongs to the multiple convolution layers, and the coding convolution layer includes multiple convolution operation modules; The performing a third convolution operation on the coordinate features through the multiple convolution layers to identify the position information of the region boundary includes: Calling the encoding convolution layer to encode the coordinate features, where the encoded features include position features of the region boundary, and the position features include position features of the first dimension and position features of the second dimension; The fully connected layer is called to identify the position information of the first dimension and the position information of the second dimension of the region boundary according to the position feature of the region boundary.

5. The method according to any one of claims 1 to 4, characterized in that The region boundary recognition model further includes a convolution extraction layer, and the step of extracting image features from the panoramic image includes: The convolution extraction layer is called to extract image features of the panoramic image.

6. The method according to any one of claims 1 to 4, characterized in that The region boundary recognition model further includes at least two convolution extraction layers and a feature fusion layer, and the step of extracting image features from the panoramic image includes: Calling the at least two convolution extraction layers to respectively extract image sub-features of the panoramic image, wherein the sizes of the image features extracted by every two convolution extraction layers are different, and the size of the image sub-features extracted by any convolution extraction layer is related to the weight of the convolution extraction layer; The feature fusion layer is called to fuse the image sub-features of the panoramic image at different sizes to obtain the image feature.

7. The method according to claim 4, characterized in that The region boundary recognition model further includes a classification module, and before identifying the location information of the region boundary, further includes: Calling the classification module to identify the type of object to be identified corresponding to each pixel feature in the position feature of the region boundary, the type of object to be identified being the region boundary or the edge of the panoramic image; If the type of the object to be identified is the region boundary, the fully connected layer is called to identify the position information of the first dimension and the position information of the second dimension of the region boundary.

8. The method according to claim 1, characterized in that It also includes training the model to be trained to obtain the region boundary recognition model, and the training method includes: Calling the model to be trained to identify the sample image to obtain a position prediction result and a type prediction result, wherein the boundary of the drivable area contained in the sample image has been marked; Calculating a smoothing loss function and a cross entropy loss function, wherein the smoothing loss function is used to characterize the loss between the position prediction result and the labeled position information, and the cross entropy loss function is used to characterize the loss between the type prediction result and the labeled type information; Determine whether the smoothing loss function and the cross entropy loss function converge; If both the smoothing loss function and the cross entropy loss function converge, the model to be trained is used as the region boundary recognition model; If the smoothing loss function and / or the cross entropy loss function do not converge, adjust the parameters of the model to be trained, and use the convolution model with the adjusted parameters as the new model to be trained, and again call the model to be trained to identify the sample image.

9. A region boundary recognition device, characterized in that: A region boundary recognition model is deployed, wherein the region boundary recognition model includes a plurality of convolutional layers, and the apparatus includes: A feature extraction module, used to extract image features from the panoramic image, wherein the image features include image features of the region boundary; a first convolution operation module, configured to perform a first convolution operation through the plurality of convolution layers to construct a coordinate mapping relationship matrix, wherein the coordinate mapping relationship refers to a mapping relationship between a first coordinate system and a second coordinate system, and the coordinate mapping relationship matrix includes a coordinate mapping relationship of a plurality of position points, wherein the plurality of position points are associated with the first coordinate system, the first coordinate system is used to represent position features from a first dimension, and the second coordinate system is used to represent position features from a second dimension; a second convolution operation module, configured to perform a second convolution operation on the image features and the coordinate mapping relationship matrix through the multiple convolution layers to generate coordinate features of the panoramic image, wherein the coordinate features include pixel features of the multiple location points and coordinate mapping relationship features of the multiple location points; An identification module is used to perform a third convolution operation on the coordinate features through the multiple convolution layers to identify the position information of the region boundary, where the position information includes the position information of the first dimension and the position information of the second dimension of the region boundary.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: The processor runs the computer program to implement the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the method according to any one of claims 1 to 8.

Images