Vehicle control method and device based on forward image and electronic equipment

Abstract

The invention discloses a vehicle control method and device based on a forward image and electronic equipment, and the method comprises the steps: carrying out the image feature extraction of an obtained forward image of a vehicle through a first neural network model, obtaining a corresponding first feature map, and enabling the first feature map to comprise the original features of a target object in a vehicle driving scene; based on the target objects, the forward image is segmented, masks corresponding to the target objects are determined, and the masks represent first position information of the corresponding target objects in the first feature map; based on the first feature map and the mask, performing pooling operation on the first feature map to obtain a corresponding target feature vector; based on the target feature vector and the category information of the target object, determining an embedded vector corresponding to each target object; and generating a control signal for controlling the vehicle by using a second neural network model based on the embedded vector and the obtained navigation information of the vehicle. According to the method, the intelligence of the automatic driving model is improved.

Description

Technical Field

[0001] The present application relates to the field of vehicle autonomous driving, and in particular to a vehicle control method, device and electronic equipment based on a forward image. Background Art

[0002] In the field of vehicle autonomous driving, large-scale natural driving data is easy to obtain and does not require any subsequent manual annotation and is easy to use. Based on natural driving data, the use of imitation learning may have better adaptability and robustness to new environments. Therefore, the use of imitation learning to build autonomous driving models has emerged in recent years. Most of these solutions directly process the output signals of sensors to achieve vehicle control. However, due to the extreme complexity of real-world scenarios, the above-mentioned autonomous driving solutions may be limited by the scenarios covered by the training data, resulting in low intelligence of the related autonomous driving models. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a vehicle control method, device and electronic device based on a forward image, which can effectively and accurately perceive relevant information of target objects outside the vehicle, thereby improving the intelligence of the vehicle's automatic driving model.

[0004] In order to achieve the above object, the present application provides a vehicle control method based on a forward image, comprising:

[0005] Using the first neural network model, extracting image features from the acquired forward image of the vehicle to obtain a corresponding first feature map, wherein the first feature map includes original features of a target object in a vehicle driving scene;

[0006] Based on the target object, a segmentation operation is performed on the forward image to determine a mask corresponding to each of the target objects, wherein the mask represents first position information of the corresponding target object in the first feature map;

[0007] Based on the first feature map and the mask, performing a pooling operation on the first feature map to obtain a corresponding target feature vector;

[0008] Determine an embedding vector corresponding to each of the target objects based on the target feature vector and the category information of the target object;

[0009] Based on the embedded vector and the acquired navigation information of the vehicle, a control signal for controlling the vehicle is generated using a second neural network model.

[0010] Optionally, the first neural network model includes at least one of the following: a position component, a sampling component, and a convolution component. The first neural network model is used to extract image features from the acquired forward image of the vehicle to obtain a corresponding first feature map, including:

[0011] Using the position component, adding a plurality of channels representing position encoding to the forward image to supplement the position information of the forward image;

[0012] Using the sampling component, down-sampling the forward image at a predetermined ratio, so that the training speed and / or inference speed of the first neural network model when processing the forward image both meet corresponding preset conditions;

[0013] The convolution component is used to perform a convolution operation on the forward image to sort out the feature dimensions of the forward image so that the image features can be effectively read.

[0014] Optionally, performing a segmentation operation on the forward image based on the target object to determine a mask corresponding to each of the target objects includes:

[0015] Based on a panoramic segmentation model, segment the front image to obtain semantic information of the front image;

[0016] The mask is determined as a binary image having the same resolution as the first feature map.

[0017] Optionally, performing a pooling operation on the first feature map based on the first feature map and the mask to obtain a corresponding target feature vector includes:

[0018] Multiplying the mask by the first feature map pixel by pixel to determine the area of ​​each target object in the first feature map;

[0019] A pooling operation is performed on the first feature map in a region corresponding to each of the target objects to obtain the target feature vector.

[0020] Optionally, the method further comprises:

[0021] Determine a destination of the vehicle and a plurality of guide waypoints for pointing to the destination, wherein the distance between adjacent guide waypoints is associated with the driving speed of the vehicle;

[0022] Encoding the guidance waypoints;

[0023] The guidance waypoint is determined as the navigation information.

[0024] Optionally, determining a destination of the vehicle and a plurality of guide waypoints pointing to the destination includes:

[0025] The guide waypoint is determined by removing passive intention information for passively operating the vehicle from the original waypoint and retaining only active intention information for actively operating the vehicle.

[0026] Optionally, the generating a control signal for controlling the vehicle based on the embedded vector and the acquired navigation information of the vehicle using the second neural network model includes:

[0027] Using the navigation information as prior information of the second neural network model and the embedding vector as input of the second neural network model to generate a planned waypoint of the vehicle;

[0028] Comparing the planned waypoint with preset expert data to determine a loss function corresponding to the planned waypoint, wherein the loss function is used for training the second neural network model;

[0029] The control signal is generated based on the planned waypoint.

[0030] Optionally, the generating a control signal for controlling the vehicle based on the embedded vector and the acquired navigation information of the vehicle using the second neural network model includes:

[0031] Using the second neural network model, a weight of the target object is determined, wherein the weight is associated with the attention allocation of the second neural network model.

[0032] The embodiment of the present application further provides a vehicle control device based on a forward image, comprising:

[0033] An extraction module is configured to use a first neural network model to perform image feature extraction on the acquired forward image of the vehicle to obtain a corresponding first feature map, wherein the first feature map includes original features of a target object in a vehicle driving scene;

[0034] a segmentation module, configured to perform a segmentation operation on the forward image based on the target object, and determine a mask corresponding to each of the target objects, wherein the mask represents first position information of the corresponding target object in the first feature map;

[0035] A pooling module, configured to perform a pooling operation on the first feature map based on the first feature map and the mask to obtain a corresponding target feature vector;

[0036] A processing module is configured to determine an embedding vector corresponding to each of the target objects based on the target feature vector and the category information of the target object; and generate a control signal for controlling the vehicle using a second neural network model based on the embedding vector and the acquired navigation information of the vehicle.

[0037] An embodiment of the present application further provides an electronic device, including a processor and a memory, wherein an executable program is stored in the memory, and the processor executes the executable program to perform the steps of the method described above.

[0038] The vehicle control method of the embodiment of the present application can cope with complex vehicle driving scenarios, effectively and accurately perceive relevant information of target objects outside the vehicle, and improve the intelligence of the vehicle's automatic driving model. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A flow chart of a vehicle control method based on a forward image according to an embodiment of the present application;

[0040] Figure 2 For the embodiments of this application Figure 1 A flowchart of an embodiment of step S100;

[0041] Figure 3 For the embodiments of this application Figure 1 A flowchart of an embodiment of step S200;

[0042] Figure 4 For the embodiments of this application Figure 1 A flowchart of an embodiment of step S300;

[0043] Figure 5 A flow chart of an embodiment of a vehicle control method according to an embodiment of the present application;

[0044] Figure 6 For the embodiments of this application Figure 1 A flowchart of an embodiment of step S500;

[0045] Figure 7 A flowchart of another specific embodiment of the vehicle control method of the embodiment of the present application;

[0046] Figure 8 This is a structural block diagram of a vehicle control device according to an embodiment of the present application. DETAILED DESCRIPTION

[0047] Various aspects and features of the present application are described herein with reference to the accompanying drawings.

[0048] It should be understood that various modifications may be made to the embodiments of the present application. Therefore, the above description should not be considered as limiting, but only as an example of an embodiment. Other modifications within the scope and spirit of the present application will occur to those skilled in the art.

[0049] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the present application and, together with the general description of the present application given above and the detailed description of the embodiments given below, serve to explain the principles of the present application.

[0050] These and other characteristics of the present application will become apparent from the following description of a preferred form of embodiment given as a non-limiting example with reference to the accompanying drawings.

[0051] It should also be understood that although the present application has been described with reference to some specific examples, those skilled in the art will be able to readily implement many other equivalent forms of the present application.

[0052] The above and other aspects, features and advantages of the present application will become more apparent in view of the following detailed description when taken in conjunction with the accompanying drawings.

[0053] Specific embodiments of the present application are described hereinafter with reference to the accompanying drawings; however, it should be understood that the embodiments applied for are merely examples of the present application, which may be implemented in a variety of ways. Well-known and / or repeated functions and structures are not described in detail to avoid unnecessary or redundant details that obscure the present application. Therefore, the specific structural and functional details applied for herein are not intended to be limiting, but merely serve as a basis and representative basis for the claims to teach those skilled in the art to use the present application in a variety of ways with substantially any suitable detailed structure.

[0054] This specification may use the phrases "in one embodiment," "in another embodiment," "in yet another embodiment," or "in other embodiments," all of which may refer to one or more of the same or different embodiments according to the present application.

[0055] A vehicle control method based on a forward image in an embodiment of the present application can be applied to an imitation learning model. The method includes obtaining a forward image of a vehicle, such as a forward image of a vehicle taken by a forward camera. Using a preset first neural network model, image features are extracted from the forward image to obtain a first feature map that can represent the original information, so as to avoid missing the original pixel-level information in the forward image when extracting image features, such as color features. In addition, the method also includes performing a segmentation operation on the forward image to obtain a segmentation result to obtain a target object in the driving scene. The first feature map is integrated with the segmentation result to generate a target feature vector. Based on the target feature vector and the category information of the target object, the embedding vector corresponding to each target is determined. The preset navigation information is used as the prior information of the preset second neural network model (such as a Transformer model), and the embedding vector is input into the second neural network model to obtain a planned waypoint. By comparing the planned waypoint with expert data (such as excellent driving data of professional users), it is determined whether the planned waypoint meets the requirements. If it meets the requirements, a control signal for controlling the vehicle can be generated based on the planned waypoint to realize automatic driving of the vehicle.

[0056] The vehicle control method based on the forward image according to the embodiment of the present application is described in detail below with reference to the accompanying drawings. Figure 1 Flow chart of the vehicle control method based on the forward image according to the embodiment of the present application, as shown in FIG. Figure 1 Shown and combined Figure 7 , the method comprises the following steps:

[0057] S100, using a first neural network model, extracting image features from an acquired forward image of the vehicle to obtain a corresponding first feature map, wherein the first feature map includes original features of a target object in a vehicle driving scene.

[0058] Exemplarily, the first neural network model is a preset network model, including a multi-layer convolutional network, for extracting features from an input image. The forward image of the vehicle may be an image taken by a forward camera of a vehicle capable of autonomous driving, which is an image in the forward direction of the vehicle. The target object may be various objects in the vehicle driving scene, such as roadside objects, traffic equipment, moving vehicles, pedestrians, and other objects in the driving scene.

[0059] The first neural network model is used to extract image features from the forward image to obtain a first feature map storing original image information, wherein the original image information includes original features of the target object in the vehicle driving scene.

[0060] For example, the forward image collected from the forward camera is an RGB image, which has three channels of red, green and blue. After adding CoordConv displayed in grayscale, the forward image is feature extracted through a 4*4 convolution kernel convolution layer and a 1*1 convolution kernel convolution layer, generating a first feature map including the original features of the target object in the vehicle driving scene.

[0061] S200, based on the target object, performing a segmentation operation on the forward image to determine a mask corresponding to each of the target objects, wherein the mask represents first position information of the corresponding target object in the first feature map.

[0062] For example, the target object may be various objects in the vehicle driving scene, and the segmentation operation on the forward image may be performed based on the target object, such as performing the segmentation of the forward image for each target object, or performing the segmentation of the forward image for each group of target objects.

[0063] During segmentation, a panoramic segmentation model can be used to process the video signal input by the forward camera to obtain a segmentation result. In one embodiment, the panoramic segmentation model can be trained using a segmentation data set shot in a traffic scene, and such a panoramic segmentation model can be built based on a convolutional neural network or a Transformer model. The trained panoramic segmentation model is then used to perform segmentation operations on the forward image.

[0064] In the process of segmenting the forward image, the mask corresponding to each target object can be determined by the panoramic segmentation model, wherein the mask can represent the first position information of the target object in the first feature map. The mask can be used to completely or partially block the processing object to control the processing area or process.

[0065] S300: Based on the first feature map and the mask, perform a pooling operation on the first feature map to obtain a corresponding target feature vector.

[0066] For example, the pooling operation is used to reduce the dimension of the feature map in a convolutional neural network or a recurrent neural network. In a convolutional neural network, the pooling operation usually follows the convolution operation to reduce the spatial size of the feature map. The pooling operation divides the feature map into several sub-regions and performs statistical aggregation on each sub-region. Reducing the dimension of the feature map reduces the number of parameters in the network, avoids overfitting, and improves the calculation speed and operation efficiency of the relevant model.

[0067] In the implementation of this city, by pixel-by-pixel multiplication of the mask and the first feature map, only the area where each target object exists on the feature map can be retained. The first feature map is pooled in the area corresponding to each target object to obtain a target feature vector.

[0068] S400, determining an embedding vector corresponding to each of the target objects based on the target feature vector and the category information of the target object.

[0069] Exemplarily, the target object has multiple different categories, which can be represented based on category information. The category information can be represented by a corresponding category vector, such as obtaining a category vector by vectorizing the content of the category information.

[0070] In one embodiment, the target feature vector and the category vector corresponding to the category information may be added to calculate the embedding vector corresponding to each target object, and the embedding vector may be used as the input of the second neural network model.

[0071] S500, based on the embedded vector and the acquired navigation information of the vehicle, using a second neural network model, generating a control signal for controlling the vehicle.

[0072] Exemplarily, the navigation information is pre-constructed information for navigating the vehicle, and can determine the basic direction of the vehicle. In one embodiment, the navigation information is pre-constructed initial information for navigation, and the second neural network model can be used to adjust it or regenerate corresponding planning information to instruct the vehicle to travel according to the planning information during actual driving.

[0073] The second neural network model can be a Transformer model, and the second neural network model is used to determine the understanding of the relationship between each target object. Specifically, the information contained in each input element is updated through the attention mechanism. Through this structural principle, the relevant information corresponding to each target object in the forward image captured by the forward camera can be updated using the navigation information. In addition, the embedded vector is used as the input of the second neural network model, and the above-mentioned planning information is output through the multi-layer perceptron of the second neural network model at the output position of the navigation information. A control signal for controlling the vehicle is generated based on the planning information, so that the vehicle can travel automatically under the control of the control signal.

[0074] The vehicle control method of the embodiment of the present application can cope with complex vehicle driving scenarios, effectively and accurately perceive relevant information of target objects outside the vehicle, and improve the intelligence of the vehicle's automatic driving model.

[0075] In one embodiment of the present application, the first neural network model includes at least one of the following: a position component, a sampling component, and a convolution component. The first neural network model is used to extract image features from the acquired forward image of the vehicle to obtain a corresponding first feature map, such as Figure 2 As shown, including:

[0076] S110, using the position component, adding a plurality of channels representing position encoding to the forward image to supplement the position information of the forward image.

[0077] Exemplarily, the first neural network model includes a position component, which may be CoordConv, which is a neural network layer for converting input coordinate information into a feature map of the neural network layer. It generates a new feature map by adding coordinate information and convolving it with the feature map. This new feature map not only contains the features of the original data, but also contains the features of the coordinate information, so that the first neural network model can better understand the input data. In this embodiment, CoordConv is used to add multiple channels representing position encoding to the forward image. For example, I coordinates and J coordinates are added to the forward image respectively, and both coordinates carry coordinate information. The position information of the forward image is supplemented so that CoordConv allows the first neural network model to be completely translation invariant and change translation correlation.

[0078] S120, using the sampling component to downsample the forward image by a predetermined ratio, so that the training speed and / or inference speed of the first neural network model when processing the forward image meets corresponding preset conditions.

[0079] For example, downsampling the forward image by a predetermined ratio can prevent the forward image resolution from being too high and affecting the training and reasoning speed of the first neural network model. Figure 7 For example, the stem convolution layer with a convolution kernel size of 4*4 of the first neural network model performs a convolution operation (including the corresponding activation function) on the input image with CoordConv added, and can downsample the forward image by 4 times. For example, the original resolution of the forward image is 512*512, and every other row and every other column in the forward image are deleted to obtain a 256*256 image, thereby achieving 2*2=4 times downsampling. When the first neural network model processes the forward image, its training speed and / or inference speed meet the corresponding preset conditions.

[0080] S130, using the convolution component to perform a convolution operation on the forward image, and to arrange the feature dimensions of the forward image so that the image features can be effectively read.

[0081] Exemplary, combined Figure 7 , the convolution layer of the convolution component with a kernel size of 1*1 can be used to perform a convolution operation on the output of the sampling component (including the corresponding activation function), sort out the dimensions of the image features of the forward image, and align them with the feature dimensions of the back-end network, so that the image features can be effectively read.

[0082] In one embodiment of the present application, based on the target object, the forward image is segmented to determine the mask corresponding to each target object, such as Figure 3 As shown, including:

[0083] S210, segmenting the front image based on a panoramic segmentation model to obtain semantic information of the front image;

[0084] S220: Determine the mask as a binary image having the same resolution as the first feature image.

[0085] Exemplarily, the panoramic segmentation model can be a model built based on a convolutional neural network or a Transformer network. The panoramic segmentation model is used to determine the mask corresponding to each target object, wherein the mask can represent the first position information of the target object in the first feature map. Specifically, the panoramic segmentation model determines the mask as a 0-1 binary image with the same resolution as the first feature map, which represents the inference of the panoramic segmentation model on the position information corresponding to each target object.

[0086] In one embodiment of the present application, based on the first feature map and the mask, a pooling operation is performed on the first feature map to obtain a corresponding target feature vector, such as Figure 4 As shown, including:

[0087] S310: Multiply the mask by the first feature map pixel by pixel to determine a region of each target object in the first feature map.

[0088] S320: Perform a pooling operation on the first feature map in the area corresponding to each target object to obtain the target feature vector.

[0089] Exemplarily, the mask is a 0-1 binary image with the same resolution as the first feature map. The mask is multiplied pixel by pixel by the first feature map, and only the area where each target object exists on the first feature map can be retained, thereby determining the area of ​​each target object in the first feature map. The first feature map is pooled in the area corresponding to each target object to obtain a target feature vector. In the convolutional neural network, the pooling operation usually follows the convolution operation, which is a dimensionality reduction operation on the first feature map. The pooling operation reduces the feature dimension of the convolution layer output, reduces network parameters and computational costs, and reduces overfitting.

[0090] In one embodiment of the present application, Figure 5 As shown, the method further comprises the following steps:

[0091] S600, determining a destination of the vehicle and a plurality of guide waypoints pointing to the destination, wherein the distance between adjacent guide waypoints is associated with the driving speed of the vehicle.

[0092] Exemplarily, the conditional imitation learning model needs to add some form of navigation information as input to inform the imitation learning model of the basic direction of travel. The present application uses several guide waypoints pointing to the destination as navigation information, and the distance between these adjacent guide waypoints is proportional to the current speed of the vehicle, which provides a basic prior for the decision result of the second neural network model.

[0093] S700: Encode the guidance waypoint.

[0094] Exemplarily, a multi-layer perceptron may be used to encode the set of guidance waypoints.

[0095] S800: Determine the guidance waypoint as the navigation information.

[0096] Among them, the guidance waypoints are determined as navigation information, and the navigation information indicates the expected driving path of the vehicle, so that the navigation information can be used as the original reference data of the second neural network model. After the embedded vectors are processed by the second neural network, the planned waypoints for indicating the vehicle's driving are obtained.

[0097] In one embodiment of the present application, determining the destination of the vehicle and a plurality of guidance waypoints pointing to the destination comprises the following steps:

[0098] The guide waypoint is determined by removing passive intention information for passively operating the vehicle from the original waypoint and retaining only active intention information for actively operating the vehicle.

[0099] Exemplarily, on the one hand, when a user is driving a vehicle, the user can actively operate the vehicle based on subjective intentions, thereby forming active intention information, which represents the user's active driving intention. Active intention information can be formed based on the user's active operation. On the other hand, when a user is driving a vehicle, the vehicle may undergo passive operations to continuously adapt to new scenarios due to the constant changes in driving scenarios, such as passively operating the current vehicle to avoid other vehicles, thereby forming passive intention information. In this implementation, original waypoints (such as waypoints determined based on comprehensive factors) can be obtained, and waypoint information corresponding to passive intention information for passively operating the vehicle can be removed from the original waypoints, while only waypoint information corresponding to active intention information for actively operating the vehicle is retained, thereby determining guidance waypoints based on active intention information, thereby determining navigation information.

[0100] For example, by uniformly interpolating and fitting the expert trajectory in space, navigation information that can reflect the user's subjective intention can be extracted, and then the guidance waypoints can be determined based on the navigation information of the subjective intention.

[0101] In one embodiment of the present application, based on the embedded vector and the acquired navigation information of the vehicle, the second neural network model is used to generate a control signal for controlling the vehicle, such as Figure 6 Shown and combined Figure 7 ,include:

[0102] S510, using the navigation information as prior information of the second neural network model, using the embedded vector as input of the second neural network model, and generating planned waypoints for the vehicle.

[0103] In some embodiments, the second neural network model can be a model built based on a neural network, such as a Transformer model, which is a basic structure in the field of artificial intelligence. Its core principle is to update the information contained in each input element through an attention mechanism. Through this basic structure, the input element corresponding to the navigation information will perceive the relevant information corresponding to each target object in the forward image captured by the camera.

[0104] In this embodiment, the navigation information is used as the prior information of the second neural network model, so as to inform the second neural network model of the vehicle's planned route, so that the second neural network model has a reference in the process of generating the planned waypoints.

[0105] In this embodiment, the above embedding vector is used as the input of a group of Transformer encoders, and the planned waypoint is output through a multi-layer perceptron at the output position of the navigation information. The planned waypoint is a waypoint used to control the driving of the vehicle, indicating the actual driving direction of the vehicle.

[0106] S520, comparing the planned waypoint with preset expert data to determine a loss function corresponding to the planned waypoint, wherein the loss function is used for training the second neural network model.

[0107] Exemplarily, the expert data may be pre-selected standard driving data, such as a real trajectory of an excellent driver driving a vehicle may be selected as the expert data, or the expert data may be determined based on historical data containing a plurality of historical trajectories.

[0108] Compare the planned waypoints with the expert data, calculate the loss function and optimize the second neural network model accordingly. The loss function is a function that maps the value of a random event or its related random variables to a non-negative real number to represent the "risk" or "loss" of the random event. The loss function is usually associated with the optimization problem as a learning criterion, and the second neural network model can be trained by minimizing the loss function.

[0109] S530: Generate the control signal based on the planned waypoint.

[0110] Exemplarily, a controller such as PID is used to generate a control signal for controlling the vehicle based on the planned waypoints, so that the vehicle travels under the control of the control signal.

[0111] In one embodiment of the present application, the step of generating a control signal for controlling the vehicle based on the embedded vector and the acquired navigation information of the vehicle using the second neural network model includes:

[0112] Using the second neural network model, a weight of the target object is determined, wherein the weight is associated with the attention allocation of the second neural network model.

[0113] Exemplarily, the second neural network model can be a Transformer model. Since the architecture of the Transformer model is an attention architecture, the attention of each target object can be weighted. The higher the attention, the higher the weight of the target object, and vice versa. In order to distinguish target objects with different weights, the corresponding target objects can be labeled based on the weights. For example, the target objects are labeled with different colors. The areas with colors close to red indicate that the model has assigned higher attention weights, and the areas with colors close to blue indicate that the model has assigned lower attention weights. The second neural network model allocates different attention weights in different driving scenarios.

[0114] Based on the same concept, the embodiment of the present application also provides a vehicle control device based on a forward image, which can be applied to an imitation learning model, such as Figure 8 As shown, the device comprises:

[0115] The extraction module is configured to use a first neural network model to perform image feature extraction on the acquired forward image of the vehicle to obtain a corresponding first feature map, wherein the first feature map includes original features of a target object in a vehicle driving scene.

[0116] Exemplarily, the first neural network model is a preset network model, including a multi-layer convolutional network, for extracting features from an input image. The forward image of the vehicle may be an image taken by a forward camera of a vehicle capable of autonomous driving, which is an image in the forward direction of the vehicle. The target object may be various objects in the vehicle driving scene, such as roadside objects, traffic equipment, moving vehicles, pedestrians, and other objects in the driving scene.

[0117] The extraction module uses the first neural network model to extract image features from the forward image to obtain a first feature map that stores original image information. The original image information includes original features of the target object in the vehicle driving scene.

[0118] For example, the forward image collected from the forward camera is an RGB image, which has three channels of red, green and blue. After adding CoordConv displayed in grayscale, the forward image is feature extracted through a 4*4 convolution kernel convolution layer and a 1*1 convolution kernel convolution layer, generating a first feature map including the original features of the target object in the vehicle driving scene.

[0119] A segmentation module is configured to perform a segmentation operation on the forward image based on the target object, and determine a mask corresponding to each of the target objects, wherein the mask represents first position information of the corresponding target object in the first feature map.

[0120] For example, the target object may be various objects in the vehicle driving scene, and the segmentation module may perform segmentation operations on the forward image based on the target object, such as segmenting the forward image for each target object or for each group of target objects.

[0121] The segmentation module can use a panoramic segmentation model to process the video signal input by the forward camera during segmentation to obtain a segmentation result. In one embodiment, the panoramic segmentation model can be trained using a segmentation data set shot in a traffic scene, and such a panoramic segmentation model can be built based on a convolutional neural network or a Transformer model. The trained panoramic segmentation model is then used to perform segmentation operations on the forward image.

[0122] In the process of segmenting the forward image, the segmentation module can determine the mask corresponding to each target object through the panoramic segmentation model, wherein the mask can represent the first position information of the target object in the first feature map. The mask can be used to completely or partially block the processing object to control the processing area or process.

[0123] A pooling module is configured to perform a pooling operation on the first feature map based on the first feature map and the mask to obtain a corresponding target feature vector.

[0124] Exemplarily, the pooling module is used to reduce the dimension of the feature map in the convolutional neural network or the recurrent neural network through the pooling operation. In the convolutional neural network, the pooling operation usually follows the convolution operation to reduce the spatial size of the feature map. The pooling operation divides the feature map into several sub-regions and performs statistical aggregation on each sub-region. Reducing the dimension of the feature map reduces the number of parameters in the network, avoids the occurrence of overfitting, and improves the calculation speed and operation efficiency of the relevant model.

[0125] In this embodiment, by pixel-by-pixel multiplication of the mask and the first feature map, only the region where each target object exists on the feature map can be retained. The pooling module performs a pooling operation on the first feature map in the region corresponding to each target object to obtain a target feature vector.

[0126] A processing module is configured to determine an embedding vector corresponding to each of the target objects based on the target feature vector and the category information of the target object; and generate a control signal for controlling the vehicle using a second neural network model based on the embedding vector and the acquired navigation information of the vehicle.

[0127] Exemplarily, the target object has multiple different categories, which can be represented based on category information. The category information can be represented by a corresponding category vector, such as obtaining a category vector by vectorizing the content of the category information.

[0128] In one embodiment, the processing module may add the target feature vector to the category vector corresponding to the category information to calculate the embedding vector corresponding to each target object. The embedding vector may be used as the input of the second neural network model.

[0129] The navigation information is pre-constructed information for navigating the vehicle, and can determine the basic direction of the vehicle. In one embodiment, the navigation information is pre-constructed initial information for navigation, and the processing module can use the second neural network model to adjust it or regenerate corresponding planning information to instruct the vehicle to travel according to the planning information during the actual formation process.

[0130] The second neural network model can be a Transformer model, and the second neural network model is used to determine the understanding of the relationship between each target object. Specifically, the information contained in each input element is updated through the attention mechanism. Through this structural principle, the processing module can use the navigation information to update the relevant information corresponding to each target object in the forward image captured by the forward camera. In addition, the processing module uses the embedded vector as the input of the second neural network model, and outputs the above-mentioned planning information through the multi-layer perceptron of the second neural network model at the output position of the navigation information. A control signal for controlling the vehicle is generated based on the planning information, and the vehicle can automatically drive under the control of the control signal.

[0131] In one embodiment of the present application, the first neural network model includes at least one of the following: a position component, a sampling component, and a convolution component, and the extraction module is further configured as follows:

[0132] Using the position component, adding a plurality of channels representing position encoding to the forward image to supplement the position information of the forward image;

[0133] Using the sampling component, down-sampling the forward image at a predetermined ratio, so that the training speed and / or inference speed of the first neural network model when processing the forward image both meet corresponding preset conditions;

[0134] The convolution component is used to perform a convolution operation on the forward image to sort out the feature dimensions of the forward image so that the image features can be effectively read.

[0135] In one embodiment of the present application, the segmentation module is further configured as follows:

[0136] Based on a panoramic segmentation model, segment the front image to obtain semantic information of the front image;

[0137] The mask is determined as a binary image having the same resolution as the first feature map.

[0138] In one embodiment of the present application, the pooling module is further configured as follows:

[0139] Multiplying the mask by the first feature map pixel by pixel to determine the area of ​​each target object in the first feature map;

[0140] A pooling operation is performed on the first feature map in a region corresponding to each of the target objects to obtain the target feature vector.

[0141] In one embodiment of the present application, the device further includes a navigation module, and the navigation module is configured to:

[0142] Determine a destination of the vehicle and a plurality of guide waypoints for pointing to the destination, wherein the distance between adjacent guide waypoints is associated with the driving speed of the vehicle;

[0143] Encoding the guidance waypoints;

[0144] The guidance waypoint is determined as the navigation information.

[0145] In one embodiment of the present application, the navigation module is further configured as follows:

[0146] The guide waypoint is determined by removing passive intention information for passively operating the vehicle from the original waypoint and retaining only active intention information for actively operating the vehicle.

[0147] In one embodiment of the present application, the processing module is further configured to:

[0148] Using the navigation information as prior information of the second neural network model and the embedding vector as input of the second neural network model to generate a planned waypoint of the vehicle;

[0149] Comparing the planned waypoint with preset expert data to determine a loss function corresponding to the planned waypoint, wherein the loss function is used for training the second neural network model;

[0150] The control signal is generated based on the planned waypoint.

[0151] In one embodiment of the present application, the processing module is further configured to:

[0152] Using the second neural network model, a weight of the target object is determined, wherein the weight is associated with the attention allocation of the second neural network model.

[0153] An embodiment of the present application further provides an electronic device, including a processor and a memory, wherein an executable program is stored in the memory, and the processor executes the executable program to perform the steps of the method described above.

[0154] An embodiment of the present application further provides a computer-readable storage medium, which carries one or more computer programs. When the one or more computer programs are executed by a processor, the steps of the method described above are implemented.

[0155] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, electronic equipment, computer-readable storage media or computer program products. Therefore, the present application may adopt the form of complete hardware embodiments, complete software embodiments or embodiments in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media that include computer-usable program code. When implemented by software, these functions may be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0156] The processor may be a general purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof. The general purpose processor may be a microprocessor or any conventional processor.

[0157] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0158] The above-mentioned readable storage medium can be a disk, an optical disk, a DVD, a USB, a read-only memory (ROM) or a random access memory (RAM), etc. This application does not limit the specific form of the storage medium.

[0159] The above embodiments are only exemplary embodiments of the present application and are not intended to limit the present application. The protection scope of the present application is defined by the claims. Those skilled in the art may make various modifications or equivalent substitutions to the present application within the essence and protection scope of the present application, and such modifications or equivalent substitutions shall also be deemed to fall within the protection scope of the present application.

Claims

1. A vehicle control method based on a forward image, characterized in that: include: Using the first neural network model, extracting image features from the acquired forward image of the vehicle to obtain a corresponding first feature map, wherein the first feature map includes original features of a target object in a vehicle driving scene; Based on the target object, a segmentation operation is performed on the forward image to determine a mask corresponding to each of the target objects, wherein the mask represents first position information of the corresponding target object in the first feature map; Based on the first feature map and the mask, performing a pooling operation on the first feature map to obtain a corresponding target feature vector; Determine an embedding vector corresponding to each of the target objects based on the target feature vector and the category information of the target object; Based on the embedded vector and the acquired navigation information of the vehicle, a control signal for controlling the vehicle is generated using a second neural network model.

2. The method according to claim 1, characterized in that The first neural network model includes at least one of the following: a position component, a sampling component, and a convolution component. The first neural network model is used to extract image features from the acquired forward image of the vehicle to obtain a corresponding first feature map, including: Using the position component, adding a plurality of channels representing position encoding to the forward image to supplement the position information of the forward image; Using the sampling component, down-sampling the forward image at a predetermined ratio, so that the training speed and / or inference speed of the first neural network model when processing the forward image both meet corresponding preset conditions; The convolution component is used to perform a convolution operation on the forward image to sort out the feature dimensions of the forward image so that the image features can be effectively read.

3. The method according to claim 1, characterized in that The step of performing a segmentation operation on the forward image based on the target object to determine a mask corresponding to each target object includes: Based on a panoramic segmentation model, segment the front image to obtain semantic information of the front image; The mask is determined as a binary image having the same resolution as the first feature map.

4. The method according to claim 1, characterized in that: The performing a pooling operation on the first feature map based on the first feature map and the mask to obtain a corresponding target feature vector includes: Multiplying the mask by the first feature map pixel by pixel to determine the area of ​​each target object in the first feature map; A pooling operation is performed on the first feature map in a region corresponding to each of the target objects to obtain the target feature vector.

5. The method according to claim 1, characterized in that The method further comprises: Determine a destination of the vehicle and a plurality of guide waypoints for pointing to the destination, wherein the distance between adjacent guide waypoints is associated with the driving speed of the vehicle; Encoding the guidance waypoints; The guidance waypoint is determined as the navigation information.

6. The method according to claim 5, characterized in that The determining of the destination of the vehicle and a plurality of guide waypoints pointing to the destination comprises: The guide waypoint is determined by removing passive intention information for passively operating the vehicle from the original waypoint and retaining only active intention information for actively operating the vehicle.

7. The method according to claim 1, characterized in that The method of generating a control signal for controlling the vehicle based on the embedded vector and the acquired navigation information of the vehicle and using the second neural network model comprises: Using the navigation information as prior information of the second neural network model and the embedding vector as input of the second neural network model to generate a planned waypoint of the vehicle; Comparing the planned waypoint with preset expert data to determine a loss function corresponding to the planned waypoint, wherein the loss function is used for training the second neural network model; The control signal is generated based on the planned waypoint.

8. The method according to claim 1, characterized in that The method of generating a control signal for controlling the vehicle based on the embedded vector and the acquired navigation information of the vehicle and using the second neural network model comprises: Using the second neural network model, a weight of the target object is determined, wherein the weight is associated with the attention allocation of the second neural network model.

9. A vehicle control device based on a forward image, characterized in that: include: An extraction module configured to facilitate the first neural network model to perform image feature extraction on the acquired forward image of the vehicle to obtain a corresponding first feature map, wherein the first feature map includes original features of a target object in a vehicle driving scene; a segmentation module, configured to perform a segmentation operation on the forward image based on the target object, and determine a mask corresponding to each of the target objects, wherein the mask represents first position information of the corresponding target object in the first feature map; A pooling module, configured to perform a pooling operation on the first feature map based on the first feature map and the mask to obtain a corresponding target feature vector; A processing module, configured to determine an embedding vector corresponding to each of the target objects based on the target feature vector and the category information of the target object; Based on the embedded vector and the acquired navigation information of the vehicle, a control signal for controlling the vehicle is generated using a second neural network model.

10. An electronic device, comprising a processor and a memory, wherein an executable program is stored in the memory, and the processor executes the executable program to perform the steps of the method according to any one of claims 1 to 8.

Images