Vehicle control method and device based on front image and electronic equipment

Abstract

This application discloses a vehicle control method, device, and electronic device based on a forward-looking image. The method includes: extracting image features from an acquired forward-looking image of the vehicle using a first neural network model to obtain a corresponding first feature map, the first feature map including the original features of target objects in the vehicle driving scene; segmenting the forward-looking image based on the target objects to determine a mask corresponding to each target object, the mask representing the first position information of the corresponding target object in the first feature map; performing pooling on the first feature map based on the first feature map and the mask to obtain a corresponding target feature vector; determining an embedding vector corresponding to each target object based on the target feature vector and the category information of the target object; and generating control signals for controlling the vehicle using a second neural network model based on the embedding vector and the acquired vehicle navigation information. This method improves the intelligence of autonomous driving models.

Description

Technical Field

[0001] This application relates to the field of autonomous driving, and in particular to a vehicle control method, device, and electronic device based on a forward-looking image. Background Technology

[0002] In the field of autonomous driving, large-scale natural driving data is readily available and easy to use without any subsequent manual annotation. Based on natural driving data, imitation learning may exhibit good adaptability and robustness to new environments. Therefore, the use of imitation learning to build autonomous driving models has been emerging in recent years, with most of these solutions directly processing sensor output signals to achieve vehicle control. However, due to the extreme complexity of real-world scenarios, the aforementioned autonomous driving solutions may be limited by the scenarios covered by the training data, resulting in lower intelligence of the related autonomous driving models. Summary of the Invention

[0003] The purpose of this application is to provide a vehicle control method, device, and electronic device based on forward-looking images. This method can effectively and accurately perceive relevant information of target objects outside the vehicle, thereby improving the intelligence of the vehicle's autonomous driving model.

[0004] To achieve the above objectives, this application provides a vehicle control method based on a forward-looking image, comprising:

[0005] Using a first neural network model, image features are extracted from the acquired forward-facing image of the vehicle to obtain a corresponding first feature map, wherein the first feature map includes the original features of the target object in the vehicle driving scene.

[0006] Based on the target object, the forward image is segmented to determine the mask corresponding to each target object. The mask represents the first position information of the corresponding target object in the first feature map.

[0007] Based on the first feature map and the mask, a pooling operation is performed on the first feature map to obtain the corresponding target feature vector;

[0008] Based on the target feature vector and the category information of the target object, determine the embedding vector corresponding to each target object;

[0009] Based on the embedded vector and the obtained navigation information of the vehicle, a control signal for controlling the vehicle is generated using a second neural network model.

[0010] Optionally, the first neural network model includes at least one of the following: a position component, a sampling component, and a convolutional component. The step of using the first neural network model to extract image features from the acquired forward-facing image of the vehicle to obtain a corresponding first feature map includes:

[0011] Using the aforementioned position component, multiple channels representing position codes are added to the forward image to supplement the position information of the forward image;

[0012] Using the sampling component, the forward image is downsampled by a predetermined ratio so that the training speed and / or inference speed of the first neural network model when processing the forward image meet the corresponding preset conditions.

[0013] The convolution component is used to perform a convolution operation on the forward image, thereby organizing the feature dimensions of the forward image so that the image features can be effectively read.

[0014] Optionally, the step of segmenting the forward image based on the target object to determine the mask corresponding to each target object includes:

[0015] Based on the panoramic segmentation model, the forward image is segmented to obtain the semantic information of the forward image;

[0016] The mask is determined to be a binary image with the same resolution as the first feature map.

[0017] Optionally, the step of performing a pooling operation on the first feature map based on the first feature map and the mask to obtain a corresponding target feature vector includes:

[0018] The mask is multiplied pixel by pixel with the first feature map to determine the region of each target object in the first feature map;

[0019] Pooling is performed on the first feature map within the region corresponding to each target object to obtain the target feature vector.

[0020] Alternatively, the method may further include:

[0021] The destination of the vehicle is determined, as well as a plurality of waypoints for pointing to the destination, wherein the distance between adjacent waypoints is related to the vehicle’s speed.

[0022] The waypoints are encoded;

[0023] The waypoints are determined as the navigation information.

[0024] Optionally, determining the destination of the vehicle and the multiple waypoints for pointing to the destination includes:

[0025] The guide waypoints are determined by removing passive intent information for passively operating the vehicle from the original waypoints and retaining only active intent information for actively operating the vehicle.

[0026] Optionally, the step of generating control signals for controlling the vehicle based on the embedded vector and the acquired navigation information of the vehicle, using the second neural network model, includes:

[0027] The navigation information is used as the prior information of the second neural network model, and the embedding vector is used as the input of the second neural network model to generate the planned waypoints of the vehicle.

[0028] The planned waypoints are compared with preset expert data to determine the loss function corresponding to the planned waypoints, wherein the loss function is used for training the second neural network model;

[0029] The control signal is generated based on the planned waypoints.

[0030] Optionally, the step of generating control signals for controlling the vehicle based on the embedded vector and the acquired navigation information of the vehicle, using the second neural network model, includes:

[0031] Using the second neural network model, the weights of the target object are determined, wherein the weights are associated with the attention allocation of the second neural network model.

[0032] This application also provides a vehicle control device based on a forward-looking image, comprising:

[0033] The extraction module is configured to use a first neural network model to extract image features from the acquired forward image of the vehicle to obtain a corresponding first feature map, wherein the first feature map includes the original features of the target object in the vehicle driving scene.

[0034] The segmentation module is configured to perform segmentation operations on the forward image based on the target object, and determine the mask corresponding to each target object, wherein the mask represents the first position information of the corresponding target object in the first feature map;

[0035] The pooling module is configured to perform a pooling operation on the first feature map based on the first feature map and the mask to obtain the corresponding target feature vector;

[0036] The processing module is configured to determine the embedding vector corresponding to each target object based on the target feature vector and the category information of the target object; and to generate control signals for controlling the vehicle using a second neural network model based on the embedding vector and the acquired navigation information of the vehicle.

[0037] This application also provides an electronic device, including a processor and a memory, wherein an executable program is stored in the memory, and the processor executes the executable program to perform the steps of the method described above.

[0038] The vehicle control method described in this application can handle complex vehicle driving scenarios, effectively and accurately perceive relevant information of target objects outside the vehicle, and improve the intelligence of the vehicle's autonomous driving model. Attached Figure Description

[0039] Figure 1 This is a flowchart of a vehicle control method based on a forward-looking image according to an embodiment of this application;

[0040] Figure 2 Examples of embodiments of this application Figure 1 A flowchart of one embodiment of step S100;

[0041] Figure 3 Examples of embodiments of this application Figure 1 A flowchart of one embodiment of step S200;

[0042] Figure 4 Examples of embodiments of this application Figure 1 A flowchart of one embodiment of step S300;

[0043] Figure 5 This is a flowchart of one embodiment of the vehicle control method according to this application;

[0044] Figure 6 Examples of embodiments of this application Figure 1 A flowchart of one embodiment of step S500;

[0045] Figure 7 This is a flowchart of another specific embodiment of the vehicle control method according to the present application;

[0046] Figure 8 This is a structural block diagram of a vehicle control device according to an embodiment of this application. Detailed Implementation

[0047] Various embodiments and features of this application are described herein with reference to the accompanying drawings.

[0048] It should be understood that various modifications can be made to the embodiments described herein. Therefore, the above description should not be considered as limiting, but merely as an example of embodiments. Other modifications within the scope and spirit of this application will be apparent to those skilled in the art.

[0049] The accompanying drawings, which are included in and form part of this specification, illustrate embodiments of the present application and, together with the general description of the present application given above and the detailed description of the embodiments given below, serve to explain the principles of the present application.

[0050] These and other features of this application will become apparent from the following description of preferred forms of embodiments given as non-limiting examples, with reference to the accompanying drawings.

[0051] It should also be understood that although this application has been described with reference to some specific examples, those skilled in the art can certainly implement many other equivalent forms of this application.

[0052] The above and other aspects, features and advantages of this application will become more apparent when taken in conjunction with the accompanying drawings and in view of the following detailed description.

[0053] Specific embodiments of this application are described thereafter with reference to the accompanying drawings; however, it should be understood that the claimed embodiments are merely examples of this application, which can be implemented in various ways. Well-known and / or repeated functions and structures are not described in detail to avoid unnecessary or redundant details that could obscure the application. Therefore, the specific structural and functional details claimed herein are not intended to be limiting, but merely serve as the basis and representative basis for the claims to teach those skilled in the art to use this application in a variety of substantially any suitable detailed structures.

[0054] This specification may use the phrases “in one embodiment,” “in another embodiment,” “in yet another embodiment,” or “in other embodiments,” all of which may refer to one or more of the same or different embodiments according to this application.

[0055] This application discloses a vehicle control method based on a forward-facing image, which can be applied to imitation learning models. The method includes acquiring a forward-facing image of the vehicle, such as an image captured by a forward-facing camera. Using a pre-defined first neural network model, image features are extracted from the forward-facing image to obtain a first feature map that represents the original information, avoiding the omission of original pixel-level information in the forward-facing image, such as color features, during feature extraction. Furthermore, the method includes segmenting the forward-facing image to obtain segmentation results for target objects in the driving scene. The first feature map and the segmentation results are integrated to generate a target feature vector. Based on the target feature vector and the category information of the target object, the embedding vector corresponding to each target is determined. Pre-defined navigation information is used as prior information for a pre-defined second neural network model (such as a Transformer model), and the embedding vector is then input into the second neural network model to obtain planned waypoints. By comparing the planned waypoints with expert data (such as excellent driving data from professional users), it is determined whether the planned waypoints meet the requirements. If they do, control signals for controlling the vehicle can be generated based on the planned waypoints to achieve autonomous driving.

[0056] The vehicle control method based on forward image according to embodiments of this application will be described in detail below with reference to the accompanying drawings. Figure 1 This is a flowchart of a vehicle control method based on a forward-looking image according to an embodiment of this application, such as... Figure 1 As shown and combined Figure 7 The method includes the following steps:

[0057] S100, using the first neural network model, image features are extracted from the acquired forward image of the vehicle to obtain a corresponding first feature map, wherein the first feature map includes the original features of the target object in the vehicle driving scene.

[0058] For example, the first neural network model is a pre-defined network model, including a multi-layer convolutional network, used for feature extraction from the input image. The forward-facing image of the vehicle can be an image captured by the forward-facing camera of a vehicle capable of autonomous driving, and is an image in the vehicle's forward driving direction. The target object can be various objects in the vehicle's driving scene, such as roadside objects, traffic equipment, moving vehicles, pedestrians, etc.

[0059] Using the first neural network model, image features are extracted from the forward image to obtain a first feature map that preserves the original image information. The original image information includes the original features of the target object in the vehicle driving scene.

[0060] For example, the forward-facing image captured by the forward-facing camera is an RGB image with three channels: red, green, and blue. After adding a CoordConv layer that displays the image as a grayscale image, features are extracted from the forward-facing image through 4x4 convolutional layers and 1x1 convolutional layers, generating a first feature map that includes the original features of the target object in the vehicle driving scene.

[0061] S200, based on the target object, perform a segmentation operation on the forward image to determine the mask corresponding to each target object, the mask representing the first position information of the corresponding target object in the first feature map.

[0062] For example, the target objects can be various objects in a vehicle driving scene, and the segmentation operation on the forward image can be performed based on the target objects. This could involve segmenting the forward image for each target object, or segmenting the forward image for each group of target objects.

[0063] During segmentation, a panoramic segmentation model can be used to process the video signal input from the forward-facing camera to obtain the segmentation result. In one embodiment, the panoramic segmentation model can be trained using a segmentation dataset captured in a traffic scene, and such a panoramic segmentation model can be built based on a convolutional neural network or a Transformer model. The trained panoramic segmentation model is then used to segment the forward-facing image.

[0064] During the segmentation of the forward image, a mask corresponding to each target object can be determined using a panoramic segmentation model. This mask represents the first position information of the target object in the first feature map. This mask can be used to occlude all or part of the object being processed, thereby controlling the processing area or process.

[0065] S300, based on the first feature map and the mask, perform a pooling operation on the first feature map to obtain the corresponding target feature vector.

[0066] For example, pooling operations are used to reduce the dimensionality of feature maps in convolutional neural networks or recurrent neural networks. In convolutional neural networks, pooling operations typically follow convolution operations to reduce the spatial size of feature maps. Pooling divides the feature map into several sub-regions and performs statistical summation on each sub-region. Reducing the dimensionality of the feature map decreases the number of parameters in the network, avoids overfitting, and improves the computational speed and efficiency of the model.

[0067] In this implementation, pixel-by-pixel multiplication of the mask and the first feature map allows us to retain only the region on the feature map where each target object exists. Pooling is then performed on the first feature map within the region corresponding to each target object to obtain the target feature vector.

[0068] S400, based on the target feature vector and the category information of the target object, determine the embedding vector corresponding to each target object.

[0069] For example, the target object may have multiple different categories, which can be represented based on category information. This category information can then be represented by corresponding category vectors, such as by vectorizing the content of the category information to obtain the category vectors.

[0070] In one embodiment, the target feature vector can be added to the category vector corresponding to the category information to calculate the embedding vector for each target object. This embedding vector can then be used as input to a second neural network model.

[0071] S500, based on the embedded vector and the acquired navigation information of the vehicle, a control signal for controlling the vehicle is generated using a second neural network model.

[0072] For example, navigation information is pre-built information used for navigating a vehicle, capable of determining the basic direction of the vehicle's movement. In one embodiment, the navigation information is pre-built initial information for navigation, which can be adjusted or regenerated using a second neural network model to instruct the vehicle to travel according to the planned information during actual driving.

[0073] The second neural network model can be a Transformer model, which is used to determine the relationships between various target objects. Specifically, it updates the information contained in each input element through an attention mechanism. Based on this structural principle, navigation information can be used to update the relevant information corresponding to each target object in the forward-facing image captured by the forward-facing camera. Furthermore, the embedding vector is used as input to the second neural network model, and the aforementioned planning information is output through a multilayer perceptron of the second neural network model at the output position of the navigation information. Based on this planning information, control signals are generated to control the vehicle, enabling the vehicle to drive automatically under the control of these signals.

[0074] The vehicle control method described in this application can handle complex vehicle driving scenarios, effectively and accurately perceive relevant information of target objects outside the vehicle, and improve the intelligence of the vehicle's autonomous driving model.

[0075] In one embodiment of this application, the first neural network model includes at least one of the following: a position component, a sampling component, and a convolution component. The first neural network model is used to extract image features from the acquired forward-facing image of the vehicle to obtain a corresponding first feature map, such as... Figure 2 As shown, it includes:

[0076] S110, using the position component, add multiple channels representing position codes to the forward image to supplement the position information of the forward image.

[0077] For example, the first neural network model includes a position component, which can be CoordConv, a neural network layer used to convert the input coordinate information into a feature map of the neural network layer. It generates a new feature map by adding coordinate information and convolving it with the feature map. This new feature map contains not only the features of the original data but also the features of the coordinate information, enabling the first neural network model to better understand the input data. In this embodiment, CoordConv is used to add multiple channels representing position encoding to the forward image. For example, I and J coordinates are added to the forward image, both of which carry coordinate information. This supplements the position information of the forward image, allowing CoordConv to ensure complete translation invariance and variable translation relevance of the first neural network model.

[0078] S120, using the sampling component, the forward image is downsampled by a predetermined ratio so that the training speed and / or inference speed of the first neural network model when processing the forward image both meet the corresponding preset conditions.

[0079] For example, downsampling the feedforward image by a predetermined ratio can prevent the feedforward image resolution from being too high, which would affect the training and inference speed of the first neural network model. Combined with... Figure 7 For example, the first neural network model uses a 4x4 stem convolutional layer to perform convolution operations (including the corresponding activation function) on the input image with CoordConv applied. This can downsample the feed image by a factor of 4. For instance, if the original resolution of the feed image is 512x512, deleting every other row and every other column in the feed image yields a 256x256 image, thus achieving a 2x2 = 4x downsampling. This ensures that the training speed and / or inference speed of the first neural network model meet the corresponding preset conditions during the processing of the feed image.

[0080] S130, using the convolution component, perform a convolution operation on the forward image to organize the feature dimensions of the forward image so that the image features can be effectively read.

[0081] For example, in combination Figure 7 The output of the sampling component can be convolved using a convolutional layer with a kernel size of 1*1 (including the corresponding activation function) to organize the dimensions of the image features of the forward image and align them with the dimensions of the backend network features, thereby enabling the image features to be effectively read.

[0082] In one embodiment of this application, the step of performing a segmentation operation on the forward image based on the target object to determine the mask corresponding to each target object is as follows: Figure 3 As shown, it includes:

[0083] S210, Based on the panoramic segmentation model, the forward image is segmented to obtain the semantic information of the forward image;

[0084] S220, the mask is determined to be a binary image with the same resolution as the first feature image.

[0085] For example, the panoptic segmentation model can be a model built based on a convolutional neural network or a Transformer network. The panoptic segmentation model determines a mask corresponding to each target object, where the mask represents the first position information of the target object in the first feature map. Specifically, the panoptic segmentation model determines the mask as a 0-1 binary map with the same resolution as the first feature map, representing the panoptic segmentation model's inference of the position information corresponding to each target object.

[0086] In one embodiment of this application, the first feature map is pooled based on the first feature map and the mask to obtain a corresponding target feature vector, such as... Figure 4 As shown, it includes:

[0087] S310, the mask is multiplied pixel by pixel with the first feature map to determine the region of each target object in the first feature map.

[0088] S320, perform pooling operation on the first feature map within the region corresponding to each target object to obtain the target feature vector.

[0089] For example, the mask is a 0-1 binary image with the same resolution as the first feature map. Multiplying the mask pixel-by-pixel with the first feature map retains only the region where each target object exists in the first feature map, thus determining the region of each target object in the first feature map. Pooling is then performed on the first feature map within the region corresponding to each target object to obtain the target feature vector. In convolutional neural networks, pooling operations typically follow convolution operations to reduce the dimensionality of the first feature map. By reducing the feature dimension of the convolutional layer output through pooling, network parameters and computational costs are reduced, while overfitting is also mitigated.

[0090] In one embodiment of this application, such as Figure 5 As shown, the method further includes the following steps:

[0091] S600, determine the destination of the vehicle and a plurality of waypoints for pointing to the destination, wherein the distance between adjacent waypoints is related to the vehicle's travel speed.

[0092] For example, conditional imitation learning models require some form of navigation information as input to inform the model of the basic direction of travel. This application uses several waypoints pointing to the destination as navigation information; the distance between these adjacent waypoints is proportional to the vehicle's current speed, providing a basic prior for the decision-making results of the second neural network model.

[0093] S700, the guide point is encoded.

[0094] For example, a multilayer perceptron can be used to encode these sets of waypoints.

[0095] S800, the waypoints are determined as the navigation information.

[0096] In this process, waypoints are defined as navigation information, which indicates the vehicle's expected travel path. This navigation information can be used as the original reference data for the second neural network model. After the second neural network processes the embedded vectors, it obtains the planned waypoints used to indicate the vehicle's travel path.

[0097] In one embodiment of this application, determining the destination of the vehicle and a plurality of waypoints for pointing to the destination includes the following steps:

[0098] The guide waypoints are determined by removing passive intent information for passively operating the vehicle from the original waypoints and retaining only active intent information for actively operating the vehicle.

[0099] For example, on the one hand, while driving, a user can actively operate the vehicle based on subjective intent, forming active intent information that represents the user's active driving intention. Active intent information can be formed based on the user's active operation. On the other hand, during driving, the constantly changing driving scenario causes the vehicle to passively operate in order to adapt to the new scenario, such as passively operating the current vehicle to avoid other vehicles, forming passive intent information. In this implementation, original waypoints (such as waypoints determined based on comprehensive factors) can be obtained. The waypoint information corresponding to passive intent information used for passive vehicle operation is removed from the original waypoints, and only the waypoint information corresponding to active intent information used for active vehicle operation is retained. Thus, guide waypoints are determined based on the active intent information, thereby determining navigation information.

[0100] For example, by uniformly interpolating and fitting polyline segments to the expert trajectory in space, navigation information that can reflect the user's subjective intent can be extracted, and waypoints can be determined based on the navigation information of subjective intent.

[0101] In one embodiment of this application, based on the embedding vector and the acquired vehicle navigation information, a control signal for controlling the vehicle is generated using the second neural network model, such as... Figure 6 As shown and combined Figure 7 ,include:

[0102] S510, the navigation information is used as prior information for the second neural network model, and the embedding vector is used as input to the second neural network model to generate the planned waypoints for the vehicle.

[0103] In some embodiments, the second neural network model can be a model built on a neural network, such as the Transformer model. The Transformer model is a fundamental structure in the field of artificial intelligence, and its core principle is to update the information contained in each input element through an attention mechanism. Through this basic structure, the input element corresponding to the navigation information will perceive the relevant information corresponding to each target object in the forward image captured by the camera.

[0104] In this embodiment, navigation information is used as prior information for the second neural network model, thereby informing the model of the vehicle's intended route. This provides a reference for the second neural network model in generating waypoints.

[0105] In this embodiment, the aforementioned embedding vector is used as input to a set of Transformer encoders, and a planned waypoint is output through a multilayer perceptron at the output position of the navigation information. This planned waypoint is used to control the vehicle's driving and indicates the actual direction the vehicle is traveling.

[0106] S520, compare the planned waypoints with preset expert data to determine the loss function corresponding to the planned waypoints, wherein the loss function is used for training the second neural network model.

[0107] For example, expert data can be pre-selected standard driving data, such as selecting the actual trajectory of a vehicle driven by an excellent driver as expert data, or expert data can be determined based on historical data containing multiple historical trajectories.

[0108] The planned waypoints are compared with expert data, a loss function is calculated, and the second neural network model is optimized accordingly. The loss function maps the values ​​of a random event or its related random variables to non-negative real numbers to represent the "risk" or "loss" of that random event. The loss function is often used as a learning criterion in relation to the optimization problem; the second neural network model can be trained by minimizing the loss function.

[0109] S530, the control signal is generated based on the planned waypoints.

[0110] For example, a controller such as a PID controller is used to generate control signals for controlling the vehicle based on the planned waypoints, so that the vehicle can drive under the control of the control signals.

[0111] In one embodiment of this application, the step of generating control signals for controlling the vehicle using the second neural network model based on the embedding vector and the acquired navigation information of the vehicle includes:

[0112] Using the second neural network model, the weights of the target object are determined, wherein the weights are associated with the attention allocation of the second neural network model.

[0113] For example, the second neural network model can be a Transformer model. Since the Transformer model's architecture is an attention-based architecture, it can assign weights to the attention of each target object; higher attention corresponds to a higher weight for the target object, and vice versa. To distinguish target objects with different weights, they can be labeled based on their weights. For example, different colors can be used to label target objects; areas closer to red indicate that the model has assigned a higher attention weight, while areas closer to blue indicate that the model has assigned a lower attention weight. The second neural network model assigns different attention weights to different driving scenarios.

[0114] Based on the same concept, embodiments of this application also provide a vehicle control device based on a forward-looking image, which can be applied to imitation learning models, such as... Figure 8 As shown, the device includes:

[0115] The extraction module is configured to use a first neural network model to extract image features from the acquired forward-facing image of the vehicle to obtain a corresponding first feature map, wherein the first feature map includes the original features of the target object in the vehicle driving scene.

[0116] For example, the first neural network model is a pre-defined network model, including a multi-layer convolutional network, used for feature extraction from the input image. The forward-facing image of the vehicle can be an image captured by the forward-facing camera of a vehicle capable of autonomous driving, and is an image in the vehicle's forward driving direction. The target object can be various objects in the vehicle's driving scene, such as roadside objects, traffic equipment, moving vehicles, pedestrians, etc.

[0117] The extraction module uses a first neural network model to extract image features from the forward image, obtaining a first feature map that stores the original image information. The original image information includes the original features of the target object in the vehicle driving scene.

[0118] For example, the forward-facing image captured by the forward-facing camera is an RGB image with three channels: red, green, and blue. After adding a CoordConv layer that displays the image as a grayscale image, features are extracted from the forward-facing image through 4x4 convolutional layers and 1x1 convolutional layers, generating a first feature map that includes the original features of the target object in the vehicle driving scene.

[0119] The segmentation module is configured to perform segmentation operations on the forward image based on the target object, and determine the mask corresponding to each target object. The mask represents the first position information of the corresponding target object in the first feature map.

[0120] For example, the target objects can be various objects in a vehicle driving scene, and the segmentation module can perform segmentation operations on the forward image based on the target objects. This could involve segmenting the forward image for each target object, or segmenting the forward image for each group of target objects.

[0121] The segmentation module can process the video signal input from the forward-facing camera using a panoramic segmentation model to obtain the segmentation result. In one embodiment, the panoramic segmentation model can be trained using a segmentation dataset captured in a traffic scene, and such a panoramic segmentation model can be built based on a convolutional neural network or a Transformer model. The trained panoramic segmentation model is then used to segment the forward-facing image.

[0122] During the segmentation process of the forward image, the segmentation module can determine the mask corresponding to each target object using a panoramic segmentation model. This mask represents the first position information of the target object in the first feature map. This mask can be used to completely or partially occlude the object being processed, thereby controlling the processing area or process.

[0123] The pooling module is configured to perform a pooling operation on the first feature map based on the first feature map and the mask to obtain the corresponding target feature vector.

[0124] For example, pooling modules use pooling operations to reduce the dimensionality of feature maps in convolutional neural networks or recurrent neural networks. In convolutional neural networks, pooling operations typically follow convolution operations to reduce the spatial size of feature maps. Pooling divides the feature map into several sub-regions and performs statistical summation on each sub-region. Reducing the dimensionality of the feature map decreases the number of parameters in the network, avoids overfitting, and improves the computational speed and operational efficiency of the model.

[0125] In this embodiment, by performing pixel-by-pixel multiplication between the mask and the first feature map, only the region where each target object exists on the feature map can be retained. The pooling module performs a pooling operation on the first feature map within the region corresponding to each target object to obtain the target feature vector.

[0126] The processing module is configured to determine the embedding vector corresponding to each target object based on the target feature vector and the category information of the target object; and to generate control signals for controlling the vehicle using a second neural network model based on the embedding vector and the acquired navigation information of the vehicle.

[0127] For example, the target object may have multiple different categories, which can be represented based on category information. This category information can then be represented by corresponding category vectors, such as by vectorizing the content of the category information to obtain the category vectors.

[0128] In one embodiment, the processing module can add the target feature vector to the category vector corresponding to the category information to calculate the embedding vector for each target object. This embedding vector can be used as input to the second neural network model.

[0129] Navigation information is pre-built information used for vehicle navigation, which determines the basic direction of the vehicle's movement. In one embodiment, this navigation information is pre-built initial information for navigation, and the processing module can use a second neural network model to adjust or regenerate corresponding planning information to instruct the vehicle to travel according to the planning information during actual formation.

[0130] The second neural network model can be a Transformer model, which is used to determine the relationships between various target objects. Specifically, it updates the information contained in each input element through an attention mechanism. Based on this structural principle, the processing module uses navigation information to update the relevant information corresponding to each target object in the forward-facing image captured by the forward-facing camera. Furthermore, the processing module uses the embedding vector as input to the second neural network model, and at the output position of the navigation information, it outputs the aforementioned planning information through the multilayer perceptron of the second neural network model. Based on this planning information, control signals are generated to control the vehicle, enabling the vehicle to drive automatically under the control of these signals.

[0131] In one embodiment of this application, the first neural network model includes at least one of the following: a location component, a sampling component, and a convolutional component, and the extraction module is further configured to:

[0132] Using the location component, multiple channels representing location codes are added to the forward image to supplement the location information of the forward image;

[0133] Using the sampling component, the forward image is downsampled by a predetermined ratio so that the training speed and / or inference speed of the first neural network model when processing the forward image meet the corresponding preset conditions.

[0134] The convolution component is used to perform a convolution operation on the forward image, thereby organizing the feature dimensions of the forward image so that the image features can be effectively read.

[0135] In one embodiment of this application, the segmentation module is further configured as follows:

[0136] Based on the panoramic segmentation model, the forward image is segmented to obtain the semantic information of the forward image;

[0137] The mask is determined to be a binary image with the same resolution as the first feature map.

[0138] In one embodiment of this application, the pooling module is further configured as follows:

[0139] The mask is multiplied pixel by pixel with the first feature map to determine the region of each target object in the first feature map;

[0140] Pooling is performed on the first feature map within the region corresponding to each target object to obtain the target feature vector.

[0141] In one embodiment of this application, the device further includes a navigation module configured to:

[0142] The destination of the vehicle is determined, as well as a plurality of waypoints for pointing to the destination, wherein the distance between adjacent waypoints is related to the vehicle’s speed.

[0143] The waypoints are encoded;

[0144] The waypoints are determined as the navigation information.

[0145] In one embodiment of this application, the navigation module is further configured as follows:

[0146] The guide waypoints are determined by removing passive intent information for passively operating the vehicle from the original waypoints and retaining only active intent information for actively operating the vehicle.

[0147] In one embodiment of this application, the processing module is further configured as follows:

[0148] The navigation information is used as the prior information of the second neural network model, and the embedding vector is used as the input of the second neural network model to generate the planned waypoints of the vehicle.

[0149] The planned waypoints are compared with preset expert data to determine the loss function corresponding to the planned waypoints, wherein the loss function is used for training the second neural network model;

[0150] The control signal is generated based on the planned waypoints.

[0151] In one embodiment of this application, the processing module is further configured as follows:

[0152] Using the second neural network model, the weights of the target object are determined, wherein the weights are associated with the attention allocation of the second neural network model.

[0153] This application also provides an electronic device, including a processor and a memory, wherein an executable program is stored in the memory, and the processor executes the executable program to perform the steps of the method described above.

[0154] This application also provides a computer-readable storage medium carrying one or more computer programs that, when executed by a processor, implement the steps of the method described above.

[0155] Those skilled in the art will understand that embodiments of this application can be provided as methods, electronic devices, computer-readable storage media, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware. Furthermore, this application can take the form of a computer program product implemented on one or more computer-readable storage media containing computer-readable program code. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0156] The aforementioned processor can be a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The aforementioned PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0157] The aforementioned memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0158] The aforementioned readable storage medium may be a magnetic disk, optical disk, DVD, USB, read-only memory (ROM) or random access memory (RAM), etc. This application does not limit the specific form of storage medium.

[0159] The above embodiments are merely exemplary embodiments of this application and are not intended to limit this application. The scope of protection of this application is defined by the claims. Those skilled in the art can make various modifications or equivalent substitutions to this application within its substance and scope of protection, and such modifications or equivalent substitutions should also be considered to fall within the scope of protection of this application.

Claims

1. A vehicle control method based on a forward-looking image, characterized in that, include: Using a first neural network model, image features are extracted from the acquired forward-facing image of the vehicle to obtain a corresponding first feature map, wherein the first feature map includes the original features of the target object in the vehicle driving scene. Based on the target object, the forward image is segmented to determine the mask corresponding to each target object. The mask represents the first position information of the corresponding target object in the first feature map. Based on the first feature map and the mask, a pooling operation is performed on the first feature map to obtain the corresponding target feature vector; Based on the target feature vector and the category information of the target object, determine the embedding vector corresponding to each target object; Based on the embedded vector and the obtained navigation information of the vehicle, a control signal for controlling the vehicle is generated using a second neural network model.

2. The method according to claim 1, characterized in that, The first neural network model includes at least one of the following: a position component, a sampling component, and a convolution component. The step of using the first neural network model to extract image features from the acquired forward-facing image of the vehicle to obtain a corresponding first feature map includes: Using the aforementioned position component, multiple channels representing position codes are added to the forward image to supplement the position information of the forward image; Using the sampling component, the forward image is downsampled by a predetermined ratio so that the training speed and / or inference speed of the first neural network model when processing the forward image meet the corresponding preset conditions. The convolution component is used to perform a convolution operation on the forward image, thereby organizing the feature dimensions of the forward image so that the image features can be effectively read.

3. The method according to claim 1, characterized in that, The step of segmenting the forward image based on the target object to determine the mask corresponding to each target object includes: Based on the panoramic segmentation model, the forward image is segmented to obtain the semantic information of the forward image; The mask is determined to be a binary image with the same resolution as the first feature map.

4. The method according to claim 1, characterized in that, The step of performing a pooling operation on the first feature map based on the first feature map and the mask to obtain the corresponding target feature vector includes: The mask is multiplied pixel by pixel with the first feature map to determine the region of each target object in the first feature map; Pooling is performed on the first feature map within the region corresponding to each target object to obtain the target feature vector.

5. The method according to claim 1, characterized in that, The method further includes: The destination of the vehicle is determined, as well as a plurality of waypoints for pointing to the destination, wherein the distance between adjacent waypoints is related to the vehicle’s speed. The waypoints are encoded; The waypoints are determined as the navigation information.

6. The method according to claim 5, characterized in that, The determination of the vehicle's destination and the plurality of waypoints for pointing to the destination includes: The guide waypoints are determined by removing passive intent information for passively operating the vehicle from the original waypoints and retaining only active intent information for actively operating the vehicle.

7. The method according to claim 1, characterized in that, Based on the embedded vector and the acquired vehicle navigation information, the second neural network model is used to generate control signals for controlling the vehicle, including: The navigation information is used as the prior information of the second neural network model, and the embedding vector is used as the input of the second neural network model to generate the planned waypoints of the vehicle. The planned waypoints are compared with preset expert data to determine the loss function corresponding to the planned waypoints, wherein the loss function is used for training the second neural network model; The control signal is generated based on the planned waypoints.

8. The method according to claim 1, characterized in that, Based on the embedded vector and the acquired vehicle navigation information, the second neural network model is used to generate control signals for controlling the vehicle, including: Using the second neural network model, the weights of the target object are determined, wherein the weights are associated with the attention allocation of the second neural network model.

9. A vehicle control device based on a forward-looking image, characterized in that, include: The extraction module is configured to use a first neural network model to extract image features from the acquired forward image of the vehicle to obtain a corresponding first feature map, wherein the first feature map includes the original features of the target object in the vehicle driving scene. The segmentation module is configured to perform segmentation operations on the forward image based on the target object, and determine the mask corresponding to each target object, wherein the mask represents the first position information of the corresponding target object in the first feature map; The pooling module is configured to perform a pooling operation on the first feature map based on the first feature map and the mask to obtain the corresponding target feature vector; The processing module is configured to determine the embedding vector corresponding to each target object based on the target feature vector and the category information of the target object; Based on the embedded vector and the obtained navigation information of the vehicle, a control signal for controlling the vehicle is generated using a second neural network model.

10. An electronic device comprising a processor and a memory, the memory storing an executable program, the processor executing the executable program to perform the steps of the method as claimed in any one of claims 1 to 8.

Images