Model training method, vehicle control method, device, vehicle, medium and product

Abstract

This application discloses a model training method, a vehicle control method, an electronic device, a control device, a vehicle, a computer-readable storage medium, and a computer program product. The model training method includes: acquiring image samples of road navigation arrows and sample labels for the image samples; training an initial model based on the image samples and sample labels; and determining a recognition model. Thus, this application can train the model using the guiding direction corresponding to the road navigation arrow during the training process. The model can then recognize the road navigation arrow within the image sample and its corresponding guiding direction, avoiding the situation where two road navigation arrows with different or even opposite guiding directions are identified as the same road navigation arrow. This reduces the recognition error of road navigation arrows and the difficulty of model training, enabling the vehicle to accurately recognize road navigation arrows through the model and thus drive steadily based on the recognized road navigation arrows.

Description

Technical Field

[0001] This application relates to the field of vehicle technology, and in particular to a model training method, a vehicle control method, an electronic device, a control device, a vehicle, a computer-readable storage medium, and a computer program product. Background Technology

[0002] In related technologies, vehicles can achieve intelligent driving functions based on markings such as lines, arrows, text, and patterns on the road, such as recognizing road navigation arrows to control vehicle movement or lane changes. However, due to the high difficulty in recognizing road navigation arrows, there is a certain degree of recognition error, which in turn affects vehicle operation. Summary of the Invention

[0003] This application provides a model training method, a vehicle control method, an electronic device, a control device, a vehicle, a computer-readable storage medium, and a computer program product.

[0004] This application provides a model training method, including:

[0005] Obtain image samples of road navigation arrows and sample labels for the image samples, wherein the sample labels are used for the road navigation arrows in the image samples and the corresponding guiding directions of the road navigation arrows;

[0006] The initial model is trained based on the image samples and the sample labels to determine the recognition model, which is able to recognize the road navigation arrows and the guiding directions in the image samples.

[0007] Thus, in this embodiment, the model can be trained based on the obtained image samples of road navigation arrows and the sample labels of the image samples. During the training process, the model can be trained using the guiding direction corresponding to the road navigation arrows. As a result, the model can identify the road navigation arrows in the image samples and the guiding direction corresponding to the road navigation arrows. This avoids the situation where the model identifies two road navigation arrows with different or even opposite guiding directions as the same road navigation arrow, reducing the recognition error of road navigation arrows. While ensuring the model's ability to recognize road navigation arrows, it also reduces the training difficulty for the model to acquire this recognition ability to a certain extent. Moreover, it enables vehicles to accurately identify road navigation arrows through the model and thus drive steadily based on the identified road navigation arrows.

[0008] In some embodiments of this application, the sample label includes at least two key points within the image sample, one key point indicating the bottom of the road navigation arrow and the other key point indicating the top of the road navigation arrow, and the line formed by the at least two key points indicating the guiding direction.

[0009] Thus, in this embodiment of the application, the guiding direction of the road navigation arrow can be marked by at least two key points in the image sample, thereby reducing the difficulty and cost of marking the arrow guiding direction.

[0010] In some embodiments of this application, the sample label further includes a category label for the image sample and an arrow location label, wherein the arrow location label is used to indicate the location of the road navigation arrow in the image sample.

[0011] Thus, in this embodiment of the application, the label of the image sample can be composed of category label, arrow position label, and key point.

[0012] In some embodiments of this application, the number of image samples is a first number, and the step of training an initial model based on the image samples and the sample labels to determine the recognition model includes:

[0013] A preset data augmentation process is performed on a first number of image samples to determine a second number of image samples, wherein the second number is greater than the first number.

[0014] The recognition model is determined by training the model based on the second number of image samples and the sample labels.

[0015] Thus, in this embodiment of the application, the image samples can be subjected to preset data augmentation processing, so that the number of image samples increases from a first number to a second number, and the model is trained based on the second number of image samples and sample labels, thereby ensuring sufficient training of the model and guaranteeing the model performance.

[0016] In some embodiments of this application, the data enhancement processing includes at least one of image brightness adjustment processing, image blurring processing, image translation processing, image scaling processing, and image cropping processing.

[0017] In some embodiments of this application, the image samples include multiple samples, and the sample labels further include arrow position labels, which are used to indicate the location of the road navigation arrow in the image samples. The step of training an initial model based on the image samples and the sample labels to determine the recognition model includes:

[0018] Based on the arrow position labels of multiple image samples, determine the target anchor point detection box of the image sample;

[0019] The recognition model is determined by training a model based on the image samples, the sample labels, and the target anchor point detection boxes.

[0020] Thus, in this embodiment, a target anchor detection box suitable for the arrow recognition task in image samples can be determined based on the arrow position labels of multiple image samples. This allows the model to be trained based on the image samples, sample labels, and target anchor detection boxes, enabling the model to make predictions based on the target anchor detection boxes, thereby improving model performance.

[0021] In some embodiments of this application, determining the target anchor point detection box of the image sample based on the arrow position labels of the plurality of image samples includes:

[0022] Clustering is performed on the arrow position labels of multiple image samples to determine the target arrow position label among the multiple arrow position labels;

[0023] Based on the target arrow position label, determine the fitness of multiple preset anchor point detection boxes;

[0024] The multiple preset anchor point detection boxes are updated based on the fitness, and the target anchor point detection box is determined.

[0025] Thus, in the embodiments of this application, the target anchor detection box can be determined by clustering algorithm and fitness, thereby ensuring the effectiveness and reliability of the target anchor detection box.

[0026] In some embodiments of this application, the method further includes:

[0027] Based on the image size of the image that the model can recognize, the image sample and the arrow position label are resized.

[0028] Thus, in this embodiment, the image samples and arrow position labels can be resized according to the image size of the image that the model can recognize, so that when the model receives the image sample input during subsequent model training, it can process the image sample, thereby ensuring the robustness of the model training process.

[0029] In some embodiments of this application, the arrow location label includes a bounding box indicating the location of the road navigation arrow in the image sample, and the method further includes:

[0030] The annotation boxes of multiple image samples are filtered according to the preset annotation box size.

[0031] Thus, in this embodiment of the application, the annotation boxes of multiple image samples can be filtered according to the preset annotation box size to determine whether the model can identify arrow targets of a certain size during the early training and later inference.

[0032] In some embodiments of this application, the initial model includes a feature extraction layer, a feature enhancement layer, and a feature prediction layer;

[0033] The feature extraction layer is configured to perform feature extraction processing on the image samples to determine image features;

[0034] The feature enhancement layer is configured to perform a preset enhancement process on the image features to determine the enhanced image features;

[0035] The feature prediction layer is configured to predict road navigation arrows within the image sample based on the enhanced image features.

[0036] Thus, in this embodiment of the application, the initial model to be trained can be implemented based on a feature extraction layer, a feature enhancement layer, and a feature prediction layer.

[0037] In some embodiments of this application, the feature prediction layer is configured as follows:

[0038] Based on the enhanced image features, determine at least one of the following confidence levels: arrow location prediction, arrow direction prediction, arrow category prediction, and background pixel prediction within the input image; and / or

[0039] Based on the enhanced image features, the confidence levels of the predicted arrow location, arrow category, arrow guidance direction, and arrow guidance direction within the input image are determined.

[0040] Thus, in this embodiment of the application, the feature prediction layer can determine at least one of the confidence levels of the arrow location prediction, arrow guidance direction prediction, arrow category prediction, and background pixel prediction in the input image based on the enhanced image features, and / or determine the confidence levels of the arrow location prediction, arrow category prediction, arrow guidance direction prediction, and arrow guidance direction prediction in the input image based on the enhanced image features, thereby completing the prediction of the image sample.

[0041] In some embodiments of this application, the step of training an initial model based on the image samples and the sample labels to determine the recognition model includes:

[0042] Based on the image samples and the initial model, determine the prediction results for the image samples;

[0043] Based on the prediction results and the sample labels, determine the loss function;

[0044] The initial model is updated based on the loss function to determine the recognition model.

[0045] Thus, in this embodiment of the application, the prediction result of the image sample can be determined based on the image sample and the initial model, the loss function can be determined based on the prediction result and the sample label, and the initial model can be updated based on the loss function to determine the recognition model, thereby completing the training of the initial model.

[0046] This application provides a vehicle control method, including:

[0047] Based on the recognition model, road navigation arrows within the vehicle environment image are identified, wherein the recognition model is trained using the model training method described above.

[0048] Thus, in this embodiment, the model can be trained based on the obtained image samples of road navigation arrows and the sample labels of the image samples. During the training process, the model can be trained using the guiding direction corresponding to the road navigation arrows. As a result, the model can identify the road navigation arrows in the image samples and the guiding direction corresponding to the road navigation arrows. This avoids the situation where the model identifies two road navigation arrows with different or even opposite guiding directions as the same road navigation arrow, reducing the recognition error of road navigation arrows. While ensuring the model's ability to recognize road navigation arrows, it also reduces the training difficulty for the model to acquire this recognition ability to a certain extent. Moreover, it enables vehicles to accurately identify road navigation arrows through the model and thus drive steadily based on the identified road navigation arrows.

[0049] In some embodiments of this application, the vehicle environment image includes multiple images, and the step of identifying road navigation arrows within the vehicle environment image according to the recognition model includes:

[0050] Multiple vehicle environment images are stitched together to determine the stitched image;

[0051] Based on the recognition model, road navigation arrows within the stitched image are identified.

[0052] Thus, in this embodiment of the application, multiple vehicle environment images can be stitched together to determine a stitched image, and road navigation arrows within the stitched image can be identified according to a recognition model, thereby identifying each road navigation arrow in the vehicle's surrounding environment.

[0053] In some embodiments of this application, the step of stitching together multiple vehicle environment images to determine the stitched image includes:

[0054] The vehicle environment image is subjected to preset distortion correction processing to determine the processed vehicle environment image;

[0055] The multiple processed vehicle environment images are stitched together to determine the stitched image.

[0056] Thus, in this embodiment of the application, a preset distortion correction process can be performed on the vehicle environment image to determine the processed vehicle environment image, and multiple processed vehicle environment images can be stitched together to determine the stitched image, so that the stitched image can be stitched together based on the distortion-corrected vehicle environment image, thereby ensuring the reliability and rationality of the stitched image.

[0057] In some embodiments of this application, identifying road navigation arrows within a vehicle environment image based on a recognition model includes:

[0058] The vehicle environment image is preprocessed to determine the preprocessed vehicle environment image;

[0059] Based on the recognition model, road navigation arrows within the preprocessed vehicle environment image are identified.

[0060] Thus, in this embodiment of the application, the vehicle environment image can be preprocessed to improve the image quality, thereby determining the preprocessed vehicle environment image, and the road navigation arrows in the preprocessed vehicle environment image can be identified according to the recognition model, thereby ensuring the accuracy of the recognition model's prediction of the road navigation arrows.

[0061] In some embodiments of this application, identifying road navigation arrows within a vehicle environment image based on a recognition model includes:

[0062] Based on the recognition model, the vehicle environment image is identified to determine the predicted positions of multiple road navigation arrows in the vehicle environment image;

[0063] Non-maximum suppression processing is performed on the multiple road navigation arrow position predictions to determine the target road navigation arrow position prediction among the multiple road navigation arrow position predictions;

[0064] Based on the predicted location of the target road navigation arrow, identify the road navigation arrow within the vehicle environment image.

[0065] Thus, in this embodiment of the application, nonmaximum suppression processing can be used to filter the multiple road navigation arrow position predictions output by the model, and the road navigation arrows in the vehicle environment image can be determined by the target road navigation arrow position predictions obtained through filtering, thereby ensuring the accuracy and reliability of the road navigation arrow recognition results to a certain extent.

[0066] This application provides a control device, including:

[0067] The transceiver unit is configured to acquire image samples of road navigation arrows and sample labels of the image samples, wherein the sample labels are used for the road navigation arrows in the image samples and the corresponding guiding direction of the road navigation arrows;

[0068] The processing unit is configured to train an initial model based on the image samples and the sample labels to determine a recognition model, which is capable of recognizing the road navigation arrows and the guiding directions in the image samples.

[0069] This application provides a control device, including:

[0070] The processing unit is configured to identify road navigation arrows within a vehicle environment image based on a recognition model, wherein the recognition model is trained using the model training method described above.

[0071] This application provides an electronic device including a memory and a processor. The memory stores a computer program, which, when executed by the processor, implements the model training method or the vehicle control method described above.

[0072] This application provides a vehicle that includes the control device or electronic device described above.

[0073] This application provides a computer-readable storage medium storing a computer program that, when executed by one or more processors, implements the model training method or the vehicle control method described above.

[0074] This application provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the model training method or the vehicle control method described above.

[0075] The electronic device, control device, vehicle, computer-readable storage medium, and computer program product provided in this application can perform model training based on the acquired image samples of road navigation arrows and the sample labels of the image samples. This allows the model to be trained using the guiding direction corresponding to the road navigation arrow during the training process. Consequently, the model can recognize the road navigation arrows within the image samples and their corresponding guiding directions, thus avoiding the situation where the model identifies two road navigation arrows with different or even opposite guiding directions as the same road navigation arrow. This reduces the recognition error of road navigation arrows. Furthermore, while ensuring the model's ability to recognize road navigation arrows, it also reduces the training difficulty for the model to acquire this recognition ability to a certain extent. Moreover, it enables vehicles to accurately identify road navigation arrows through the model, and thus drive steadily based on the identified road navigation arrows.

[0076] Additional aspects and advantages of embodiments of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of embodiments of this application. Attached Figure Description

[0077] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, wherein:

[0078] Figure 1 This is a schematic diagram of a driving scenario;

[0079] Figure 2 This is a flowchart illustrating the model training method in some embodiments of this application;

[0080] Figure 3 This is a schematic diagram illustrating application scenarios in some embodiments of this application;

[0081] Figure 4 This is a schematic diagram illustrating application scenarios in some embodiments of this application;

[0082] Figure 5 Application scenario diagrams of some embodiments of this application;

[0083] Figure 6 This is a flowchart illustrating the model training method in some embodiments of this application;

[0084] Figure 7 This is a flowchart illustrating the model training method in some embodiments of this application;

[0085] Figure 8 This is a flowchart illustrating the model training method in some embodiments of this application;

[0086] Figure 9 This is a schematic diagram of the feature prediction layer in some embodiments of this application;

[0087] Figure 10 This is a schematic diagram of the feature prediction layer in some embodiments of this application;

[0088] Figure 11 This is a flowchart illustrating the model training method in some embodiments of this application;

[0089] Figure 12 Application scenario diagrams of some embodiments of this application;

[0090] Figure 13 Application scenario diagrams of some embodiments of this application;

[0091] Figure 14This is a flowchart illustrating a vehicle control method in certain embodiments of this application;

[0092] Figure 15 This is a flowchart illustrating a vehicle control method in certain embodiments of this application;

[0093] Figure 16 This is a flowchart illustrating a vehicle control method in certain embodiments of this application;

[0094] Figure 17 This is a flowchart illustrating a vehicle control method in certain embodiments of this application;

[0095] Figure 18 This is a flowchart illustrating a vehicle control method in some embodiments of this application. Detailed Implementation

[0096] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the embodiments of this application, and should not be construed as limiting the embodiments of this application.

[0097] In related technologies, one approach to identify road navigation arrows in driving lanes involves first acquiring a front view image of the real-world road captured by the vehicle's front-view camera, then parsing candidate navigation arrows from the front view image, identifying the arrow direction category of the candidate navigation arrows, calculating the rotation rectangle of the candidate navigation arrows based on the arrow direction category, and finally calculating the vector parameters of the candidate navigation arrows based on the rotation rectangle, thereby obtaining the arrow pointing direction.

[0098] However, this scheme involves multiple processing steps, including but not limited to semantic segmentation, cropping, binarization, parsing of candidate navigation arrows, direction recognition, rotating rectangle fitting, and vector parameter calculation. Therefore, the algorithm has high complexity and requires significant computational resources and processing power. Furthermore, the scheme involves the selection of many parameters, such as thresholds, error thresholds, and area thresholds. These parameters need to be reasonably adjusted and optimized to adapt to different road and environmental conditions, which correspondingly increases the difficulty of applying this scheme.

[0099] Another approach to identifying road navigation arrows on roads involves first capturing images of the road surface and the tip and base of the arrow to construct the current arrow. Then, a series of confidence criteria are used to determine whether the arrow can be identified as a road navigation arrow, thus performing the identification. However, in the complex and ever-changing real-world road environment, relying on pre-set conditions for identifying road navigation arrows may lead to misjudgments or omissions.

[0100] Furthermore, related technologies highlight schemes based on deep learning object detection networks for recognizing road navigation arrows. Consequently, deep learning has gradually become an application trend in the field of navigation arrow recognition. However, deep learning-based implementations generally have a limitation: they can only identify the position and category of the arrow, but have difficulty identifying the direction. Therefore, they cannot distinguish navigation arrows of the same category but with different directions. For details, please refer to [link to relevant documentation]. Figure 1 , Figure 1 This is a schematic diagram of a driving scenario, such as... Figure 1 Regarding the two arrows marked with black borders, both arrows are classified as straight arrows. However, the direction indicated by the left straight arrow is opposite to that indicated by the right straight arrow. Therefore, identifying only the location and category of the detection may lead to misidentification.

[0101] Based on the above description, it is evident that navigation arrow recognition schemes in related technologies suffer from several problems, such as poor real-time performance, low accuracy, and inability to obtain arrow direction. These problems collectively limit the effectiveness and reliability of these schemes in practical applications.

[0102] Based on the issues mentioned above, please refer to Figure 2 This application provides a model training method, including:

[0103] 01: Obtain image samples of road navigation arrows and sample labels for the image samples. The sample labels are used to identify the road navigation arrows in the image samples and the corresponding guiding directions of the road navigation arrows.

[0104] 02: Train the initial model based on image samples and sample labels to determine the recognition model. The recognition model can identify road navigation arrows and guidance directions in the image samples.

[0105] This application provides a control device. The model training method of this application can be implemented by the control device of this application. Specifically, the control device includes a transceiver unit and a processing unit. The transceiver unit is configured to acquire image samples of road navigation arrows and sample labels for the image samples. The sample labels are used to identify the road navigation arrows and their corresponding guidance directions in the image samples. The processing unit is configured to train an initial model based on the image samples and sample labels to determine a recognition model capable of recognizing the road navigation arrows and guidance directions in the image samples.

[0106] This application also provides an electronic device, which includes a memory and a processor. The model training method of this application can be implemented by the electronic device of this application. Specifically, the memory stores a computer program, and the processor is used to acquire image samples of road navigation arrows and sample labels of the image samples, and to train an initial model based on the image samples and sample labels to determine a recognition model. The recognition model is able to recognize the road navigation arrows and guiding directions in the image samples.

[0107] Specifically, in the embodiments of this application, the electronic device (or control device) can train the neural network model to be trained based on the obtained image samples of road navigation arrows and the sample labels of the image samples to update the parameters such as weights and biases in the model, so that the model can be trained to have the ability to "predict the road navigation arrow in the image after receiving an image, and predict the guidance direction corresponding to the road navigation arrow", thereby obtaining a recognition model that can be used by vehicles to recognize road navigation arrows.

[0108] To more clearly illustrate the recognition model in the embodiments of this application, please refer to the following: Figure 3 and Figure 4 , Figure 3 and Figure 4 These are all schematic diagrams illustrating application scenarios in certain embodiments of this application. That is, in the embodiments of this application, after receiving an image, the recognition model can output, as shown below... Figure 3 or Figure 4 The recognition results shown are as follows: the recognition model recognizes the acquired vehicle environment image, identifies the road navigation arrow in the vehicle environment image, and marks the position of the road navigation arrow in the image with a rectangle, that is, it displays the category of the road navigation arrow, such as straight_or_left (straight or left turn), and marks the guiding direction of the road navigation arrow with two key points, one before and one after.

[0109] Thus, in this embodiment, the model can be trained based on the obtained image samples of road navigation arrows and the sample labels of the image samples. During the training process, the model can be trained using the guiding direction corresponding to the road navigation arrows. As a result, the model can identify the road navigation arrows in the image samples and the guiding direction corresponding to the road navigation arrows. This avoids the situation where the model identifies two road navigation arrows with different or even opposite guiding directions as the same road navigation arrow, reducing the recognition error of road navigation arrows. While ensuring the model's ability to recognize road navigation arrows, it also reduces the training difficulty for the model to acquire this recognition ability to a certain extent. Moreover, it enables vehicles to accurately identify road navigation arrows through the model and thus drive steadily based on the identified road navigation arrows.

[0110] Exemplary, with Figure 1 For example, a vehicle can use a recognition model, either locally installed or mounted on a remote server, to identify the acquired images of its external environment, thereby recognizing... Figure 1 The road navigation arrow on the left is a straight arrow pointing south (or towards the bottom of the image), while... Figure 1 The road navigation arrow on the right is a straight arrow pointing north (or to the top of the image).

[0111] In some embodiments of this application, the sample label includes at least two key points within the image sample, one key point indicating the bottom of the road navigation arrow and the other key point indicating the top of the road navigation arrow, and the line formed by the at least two key points indicating the guiding direction.

[0112] Specifically, in this embodiment of the application, the direction of arrows in an image sample can be marked by key point annotation. Specifically, in one example, two key points can be marked in the image sample, wherein the first key point is located at the midpoint or corner of the bottom of the main trunk of the road navigation arrow, and the second key point is located at the midpoint or corner of the top of the main trunk of the road navigation arrow, and the line connecting the first key point to the second key point indicates the direction of the road navigation arrow.

[0113] For a clearer illustration of the implementation methods of this application, please refer to [link / reference]. Figure 5 , Figure 5 This application provides application scenario illustrations for certain embodiments. For example, using... Figure 5 Taking the image sample of the straight arrow located in the first row and first column as an example, in the image sample of the straight arrow, one key point (let the key point be P1) is located at the midpoint of the bottom of the main trunk of the straight arrow, and another key point (let the key point be P2) is located at the midpoint of the top of the main trunk of the straight arrow. The direction from P1 to P2 is the direction of the arrow.

[0114] Understandably, using arrows to mark key points allows for better visualization when facing... Figure 1 In the scenario shown, Figure 1 The arrow on the left points south (or downwards), and... Figure 1 The arrow on the right points north (or upwards). After model training, the model can recognize the direction of arrows on the road surface, thus correctly identifying arrows of the same type but different directions, such as when facing... Figure 1 In the scenario shown, the arrow on the left can be recognized as a straight arrow pointing south (or downwards), and... Figure 1 The arrow on the right is identified as a straight arrow pointing north (or upwards).

[0115] It is also understandable that using arrow directions based on key points can reduce the complexity and cost of the annotation process, thereby making the system's training and data preparation more efficient.

[0116] Thus, in this embodiment of the application, the guiding direction of the road navigation arrow can be marked by at least two key points in the image sample, thereby reducing the difficulty and cost of marking the arrow guiding direction.

[0117] In some embodiments of this application, the sample label also includes a category label for the image sample and an arrow position label, wherein the arrow position label is used to indicate the location of the road navigation arrow in the image sample.

[0118] Specifically, in this embodiment of the application, the annotation process can also annotate the position of the road navigation arrows in the sample image, and the types of road navigation arrows that can be annotated. For example, in... Figure 5 In the example shown, the black rectangle surrounding the road navigation arrow is the arrow location label.

[0119] And, in such Figure 5 In one example shown, the road navigation arrows can be divided into eight types: straight, left turn, right turn, "left or right turn", "straight or right turn", "straight or left turn", "straight, left or right turn", and U-turn.

[0120] In one example, road navigation arrows can be categorized into 12 types: straight, left turn, right turn, "straight or left turn", "straight or right turn", "left turn or right turn", U-turn, "straight or U-turn", "left turn or U-turn", "left turn or left merge", "right turn or right merge", and "straight, left turn or right turn".

[0121] Understandably, after labeling the image samples with category labels and arrow position labels, the model can also learn the ability to "identify the category and location of road navigation arrows in the image" through training.

[0122] Thus, in this embodiment of the application, the label of the image sample can be composed of category label, arrow position label, and key point.

[0123] Furthermore, in some embodiments of this application, the category labels, arrow position labels, and key point annotations corresponding to the image samples can all be determined through user operations. For example, in one example, the user can use an annotation tool to annotate the filtered image samples, that is, to annotate the category, position, and direction of the road navigation arrows in the image samples.

[0124] The location labels use rectangular boxes to ensure the navigation arrows are completely enclosed and to avoid overly large boxes. Furthermore, the category labels on the rectangles must match the actual navigation arrows to prevent labeling errors. When labeling key points, the first key point is located at the midpoint or corner of the bottom of the arrow's main stem, and the second key point is located at the midpoint or corner of the top of the arrow's main stem. The line connecting the first and second key points indicates the direction of the arrow. It is important to note that key points must be within their respective rectangular boxes and accurately labeled, similar to... Figure 3 The situation shown.

[0125] Please see Figure 6 In some embodiments of this application, the number of image samples is a first number, and step 02 includes:

[0126] 020: Perform preset data augmentation processing on a first number of image samples to determine a second number of image samples, where the second number is greater than the first number;

[0127] 021: Train the model based on the second number of image samples and sample labels to determine the recognition model.

[0128] The processing unit in this embodiment is configured to perform preset data augmentation processing on a first number of image samples, determine a second number of image samples, and perform model training based on the second number of image samples and sample labels to determine a recognition model, wherein the second number is greater than the first number.

[0129] The processor in this embodiment is further configured to perform preset data augmentation processing on a first number of image samples, determine a second number of image samples, and perform model training based on the second number of image samples and sample labels to determine a recognition model, wherein the second number is greater than the first number.

[0130] Specifically, in order to ensure the training effect of the model, in this embodiment of the application, the electronic device can perform data augmentation on the image samples to increase the number of image samples, so that the number of image samples increases from a first number to a second number. In this way, during the training process, the model can be trained with a sufficient number of image samples, thereby ensuring that the model is fully trained.

[0131] It is understandable that the specific methods of data augmentation can be set according to the actual situation. For example, in one example, data augmentation includes at least one of image brightness adjustment, image blurring, image panning, image scaling, and image cropping.

[0132] In one example, to simulate the change in arrow brightness caused by changes in parking lot lighting, in this embodiment of the application, the electronic device can randomly adjust the brightness of image samples in the training set to expand the training set, so that the training set includes image samples without brightness adjustment and image samples with brightness adjustment. It is understood that random brightness adjustment can simulate brightness changes in actual situations, making the image samples in the training set more closely resemble the real environment.

[0133] In one example, to simulate possible scene noise or image blurring, the electronic device can randomly blur the image samples in the dataset, thereby adding randomly blurred images to the training set to simulate image blurring caused by noise or the device.

[0134] In one example, to address the influence of factors such as the vehicle's direction of travel on the direction of the navigation arrow, and to expand the number of image samples in the dataset, image enhancement operations such as translation, scaling, and cropping are implemented to obtain more image variations, enabling the model to adapt to arrows in different situations with a sufficient number of image samples.

[0135] Thus, in this embodiment of the application, the image samples can be subjected to preset data augmentation processing, so that the number of image samples increases from a first number to a second number, and the model is trained based on the second number of image samples and sample labels, thereby ensuring sufficient training of the model and guaranteeing the model performance.

[0136] Please see Figure 7 In some embodiments of this application, the image samples include multiple samples, and the sample labels further include arrow position labels, which are used to indicate the location of road navigation arrows in the image samples. Step 02 includes:

[0137] 022: Determine the target anchor point detection box of the image sample based on the arrow position labels of multiple image samples;

[0138] 023: Train the model based on image samples, sample labels, and target anchor point detection boxes to determine the recognition model.

[0139] The processing unit in this embodiment is configured to determine the target anchor point detection box of the image sample based on the arrow position labels of multiple image samples, and to perform model training based on the image sample, sample label and target anchor point detection box to determine the recognition model.

[0140] The processor in this embodiment is also used to determine the target anchor point detection box of the image sample based on the arrow position labels of multiple image samples, and to perform model training based on the image sample, sample label and target anchor point detection box to determine the recognition model.

[0141] Specifically, to enable the model to better adapt to the detection of road navigation arrows of different sizes and shapes, in this embodiment, the anchor box suitable for the road navigation arrow, i.e., the target anchor point detection box, can be determined by the position label of the road navigation arrow in each image sample. Furthermore, during training, the model can be trained based on the image sample, the sample label of the image sample, and the determined target anchor point detection box; in other words, the model can predict the image sample using the target anchor point detection box, thereby updating the model based on the sample label of the image sample and the model prediction result.

[0142] Thus, in this embodiment, a target anchor detection box suitable for the arrow recognition task in image samples can be determined based on the arrow position labels of multiple image samples. This allows the model to be trained based on the image samples, sample labels, and target anchor detection boxes, enabling the model to make predictions based on the target anchor detection boxes, thereby improving model performance.

[0143] Please see Figure 8 In some embodiments of this application, step 022 includes:

[0144] 0220: Cluster the arrow position labels of multiple image samples to determine the target arrow position label among the multiple arrow position labels;

[0145] 0221: Determine the fitness of multiple preset anchor point detection boxes based on the target arrow position label;

[0146] 0222: Update multiple preset anchor point detection boxes based on fitness, and determine the target anchor point detection box.

[0147] The processing unit in this embodiment is configured to perform clustering processing on the arrow position labels of multiple image samples, determine the target arrow position label among the multiple arrow position labels, determine the fitness of multiple preset anchor point detection boxes based on the target arrow position label, update the multiple preset anchor point detection boxes based on the fitness, and determine the target anchor point detection box.

[0148] The processor in this embodiment is further configured to perform clustering processing on the arrow position labels of multiple image samples to determine the target arrow position label among the multiple arrow position labels, and to determine the fitness of multiple preset anchor point detection boxes based on the target arrow position label, and to update the multiple preset anchor point detection boxes based on the fitness to determine the target anchor point detection box.

[0149] Specifically, in order to improve the detection performance of the model, or in other words, to improve the accuracy and stability of the model for the road navigation arrow recognition task, in this embodiment of the application, the electronic device can determine the above-mentioned target anchor point detection box through program code used to implement clustering algorithms and genetic algorithms.

[0150] In one example, the electronic device performs clustering of multiple image samples using the k-means clustering algorithm.

[0151] In one example, suppose the arrow position label is Figure 3 Given a black rectangular bounding box and a sample size of m, the process by which the electronic device determines the target anchor point detection box includes:

[0152] Step (1): Use the k-means clustering algorithm to cluster the arrow position labels of m image samples to determine n arrow position labels. In one example, n is 9.

[0153] Step (2): Use a genetic algorithm to randomly mutate the width and height of the arrow position labels of these n image samples.

[0154] Step (3): Calculate the fitness of the mutated arrow position labels to evaluate their effectiveness. If the fitness of the mutated arrow position label is less than or equal to the fitness of the arrow position labels before and after the mutation, it indicates that the effectiveness of the arrow position label has decreased, and therefore the mutation of the arrow position label is ignored, or in other words, the mutated arrow position label is deleted. Conversely, if the fitness of the mutated arrow position label is higher than the fitness of the arrow position labels before and after the mutation, it indicates that the effectiveness of the arrow position label has increased, and therefore the mutated arrow position label is retained, or in other words, the original arrow position label is deleted.

[0155] Step (4): Repeat step (3) until the preset number of times is reached. In one example, the preset number of times is 1000.

[0156] Step (5): Sort the arrow position labels according to the area size, so that the sorted arrow position labels can be used as the final usable anchor point detection box, that is, the target anchor point detection box, for subsequent use by the model.

[0157] Thus, in the embodiments of this application, the target anchor detection box can be determined by clustering algorithm and fitness, thereby ensuring the effectiveness and reliability of the target anchor detection box.

[0158] In some embodiments of this application, the training method further includes:

[0159] Based on the image size that the model can recognize, the image samples and arrow position labels are resized.

[0160] The processing unit in this embodiment is configured to perform size adjustment processing on the image sample and the arrow position label according to the image size of the model-recognizable image.

[0161] The processor in this embodiment is also used to perform size adjustment processing on the image sample and the arrow position label according to the image size of the model-recognizable image.

[0162] Specifically, in order to ensure that the image samples can be processed by the model, the electronic device also adjusts the size of each image sample in this embodiment, so that the image samples after size adjustment can be accepted and processed by the model.

[0163] Specifically, the label at the arrow position is as follows: Figure 3 As shown in the black rectangle, the electronic device can read the width and height of each image sample in the dataset, as well as the width and height of the corresponding arrow position label for each image sample. Then, the electronic device can scale each image sample so that its size meets the requirements of the initial model (i.e., the image size that the model can recognize). It is understandable that when scaling the image sample, the size of the corresponding arrow position label can also be adjusted accordingly to ensure that, after the image sample is resized, the corresponding arrow position label accurately indicates the position of the road navigation arrow in the image sample.

[0164] Thus, in this embodiment, the image samples and arrow position labels can be resized according to the image size of the image that the model can recognize, so that when the model receives the image sample input during subsequent model training, it can process the image sample, thereby ensuring the robustness of the model training process.

[0165] In some embodiments of this application, the arrow location label includes a bounding box indicating the location of the road navigation arrow in the image sample, and the training method further includes:

[0166] The annotation boxes of multiple image samples are filtered according to the preset annotation box size.

[0167] The processing unit in this embodiment is configured to filter the annotation boxes of multiple image samples according to a preset annotation box size.

[0168] The processor in this embodiment is also used to filter the annotation boxes of multiple image samples according to a preset annotation box size.

[0169] Specifically, in order to ensure that the model can correctly identify navigation arrows on the road, in this embodiment of the application, the annotation boxes of multiple image samples can be filtered according to the preset annotation box size to determine whether the model can accurately identify arrow targets of a certain size during the early training and later inference.

[0170] In one example, the default label size refers to the size of a label box whose width and height are both greater than or equal to x pixels. In another example, x is greater than or equal to 2.

[0171] Furthermore, in one example, the electronic device can delete bounding boxes in each image sample whose width and height are both less than x pixels to complete the bounding box filtering process.

[0172] Thus, in this embodiment of the application, the annotation boxes of multiple image samples can be filtered according to the preset annotation box size to determine whether the model can identify arrow targets of a certain size during the early training and later inference.

[0173] Furthermore, in the embodiments of this application, the electronic device can first read the width and height of each image sample in the dataset, and read the width and height of the corresponding annotation box for each image sample.

[0174] Then, the electronic device scales the size of each image sample to ensure that each scaled image sample meets the image size requirements for model recognition. Understandably, while scaling the image sample, the size of the corresponding bounding box can also be adjusted accordingly to ensure that the bounding box accurately indicates the location of the arrow in the image sample.

[0175] Then, for each image sample, the electronic device can retain a bounding box with a width and height greater than or equal to x pixels (x≥2) to ensure that the model can recognize arrow targets of a certain size;

[0176] Next, the electronic device uses the k-means clustering algorithm to obtain n most suitable anchors (corresponding to the target arrow position labels mentioned above) from the labeled boxes corresponding to each image sample, in order to better adapt to the size and shape of the target. In one example, n is 9.

[0177] Next, the electronic device randomly mutates the width and height of the anchors using a genetic algorithm. If the effectiveness of the mutated result is better when evaluated by fitness, the anchors are updated. Conversely, if the effectiveness of the mutated result is worse, it is skipped. This process is repeated 1000 times by default to find the final result.

[0178] Finally, the electronic device sorts the resulting mutated anchors according to their area and returns them for use in subsequent models.

[0179] In some embodiments of this application, the initial model includes a feature extraction layer, a feature enhancement layer, and a feature prediction layer. The feature extraction layer is configured to perform feature extraction processing on the image sample to determine image features. The feature enhancement layer is configured to perform preset enhancement processing on the image features to determine enhanced image features. The feature prediction layer is configured to predict road navigation arrows in the image sample based on the enhanced image features.

[0180] Specifically, in the embodiments of this application, an initial model including a feature extraction layer, a feature enhancement layer and a feature prediction layer can be constructed according to the actual scenario and application requirements, and the initial model can be trained to determine the final deployable recognition model.

[0181] In essence, the feature extraction layer is used to extract features from the input image, such as during training, to extract features from image samples. Furthermore, the specific structure of the feature extraction layer can be customized based on the specific circumstances. For instance, in one example, the feature extraction layer could be designed based on model structures such as ResNet (Residual Network), MobileNet, or CSPDarknet, thereby extracting high-level features from the input image.

[0182] Furthermore, in the embodiments of this application, the feature enhancement layer is used to enhance the information representation of image features. In one example, the feature enhancement layer can obtain feature representations at different scales through structures such as FPN (Feature Pyramid Network) and PAN (Pyramid Attention Network). FPN can fuse features at different levels through upsampling and downsampling operations to adapt to different target sizes. PAN, on the other hand, achieves cross-level feature fusion through lateral connections and pyramid pooling operations, enabling the model to simultaneously focus on information at different levels.

[0183] In addition, the feature prediction layer can make predictions based on the output of the feature enhancement layer, thereby determining the predicted road guide arrows corresponding to the image samples.

[0184] Thus, in this embodiment of the application, the initial model to be trained can be implemented based on a feature extraction layer, a feature enhancement layer, and a feature prediction layer.

[0185] In one example, the feature prediction layer is configured as follows:

[0186] Based on enhanced image features, determine at least one of the following confidence levels: arrow location prediction, arrow direction prediction, arrow category prediction, and background pixel prediction within the input image; and / or

[0187] Based on enhanced image features, the confidence levels of the predicted arrow location, arrow category, arrow guidance direction, and arrow guidance direction within the input image are determined.

[0188] Specifically, in the embodiments of this application, the feature prediction layer can be designed using two approaches. One approach is to regress the navigation arrow rectangle, key point coordinates (i.e., guidance direction), and predict the confidence scores of each category and background pixel prediction in the image. The other approach is to regress the coordinates of the navigation arrow rectangle, the navigation arrow type, the confidence scores of each category, and the coordinates and confidence scores of the navigation arrow key points.

[0189] For a clearer illustration of the implementation methods of this application, please refer to the following: Figure 9 and Figure 10 , Figure 9 and Figure 10 These are all schematic diagrams of feature prediction layers in certain embodiments of this application. Specifically, the feature prediction layers in the embodiments of this application can be built based on the idea of ​​dense connections, i.e., as shown below. Figure 9 As shown, the feature prediction layer based on the dense connection approach includes two branches: one for regressing the navigation arrow rectangle and key point coordinates, and the other for predicting the confidence scores of each category and background.

[0190] Furthermore, in such Figure 9 In one example shown, the branches for regressing the navigation arrow rectangle and keypoint coordinates can be implemented using a 1x1 convolution. Figure 9 In this context, A represents the resolution of the predicted feature layer, and m1 represents the number of location regression parameters. The corresponding calculation expression is as follows:

[0191] m1 = (num bbox +num kpt )×num anchor

[0192] Where m1 represents the number of location regression parameters, and num... box num represents the number of parameters in the target location bounding box. kpt num represents the number of parameters for predicting keypoints. anchor This represents the number of anchors corresponding to the current predicted feature layer.

[0193] And, in such Figure 9 In one example shown, the regression branch used to predict the confidence levels for each category and background can also be constructed using a 1x1 convolution. Figure 9 In this context, A represents the resolution of the predicted feature layer, and m2 represents the number of confidence regression parameters. The corresponding calculation expression is as follows:

[0194] m2=(num label +1)×numanchor

[0195] Where m2 represents the number of confidence regression parameters, and num label num represents the number of target categories. anchor This represents the number of anchors corresponding to the current predicted feature layer, with 1 representing the background.

[0196] Furthermore, the feature prediction layer in the embodiments of this application can be built based on the idea of ​​separate prediction, i.e., as follows: Figure 10 As shown, the feature prediction layer based on the split prediction approach includes two branches: one for regressing the coordinates of the navigation arrow rectangle, the navigation arrow type, and the confidence level of each category; and the other for regressing the coordinates and confidence level of the navigation arrow key points.

[0197] Furthermore, in such Figure 10 In one example shown, the branches used to regress the coordinates of the navigation arrow rectangle, the navigation arrow type, and the confidence score for each category can be constructed using 1x1 convolutions. Figure 10 In this context, A represents the resolution of the predicted feature layer, and n1 represents the number of regression parameters for the target bounding box. The corresponding calculation expression is as follows:

[0198] n1 = (num bbox +num label_cof +1)×num anchor

[0199] Where n1 is the number of regression parameters for the target bounding box, and num box num represents the number of parameters in the target location bounding box. label_cof num represents the number of target categories. anchor This represents the number of anchors corresponding to the current predicted feature layer, where 1 represents the predicted category.

[0200] And, in such Figure 10 In one example shown, the branch used to regress the navigation arrow keypoints can also be constructed using a 1*1 convolution. Figure 10 In this context, A represents the resolution of the predicted feature layer, and n² represents the number of keypoint regression parameters. The calculation expression is as follows:

[0201] n2=(num kpt +num kpt_cof )×num anchor

[0202] Where n² is the number of regression parameters for key points, and num kpt num represents the number of parameters for predicting keypoints. kpt_cof Confidence score for each keypoint, num anchor This represents the number of anchors corresponding to the current predicted feature layer.

[0203] It should also be noted that, based on, for example Figure 9 or Figure 10 In the case of building the feature prediction layer with the structure shown, m1, m2, n1, and n2 can all be understood as the number of parameters that the initial model needs to learn.

[0204] Thus, in this embodiment of the application, the feature prediction layer can determine at least one of the confidence levels of the arrow location prediction, arrow guidance direction prediction, arrow category prediction, and background pixel prediction in the input image based on the enhanced image features, and / or determine the confidence levels of the arrow location prediction, arrow category prediction, arrow guidance direction prediction, and arrow guidance direction prediction in the input image based on the enhanced image features, thereby completing the prediction of the image sample.

[0205] Please see Figure 11 In some embodiments of this application, step 02 includes:

[0206] 024: Based on the image samples and the initial model, determine the prediction results for the image samples;

[0207] 025: Determine the loss function based on the prediction results and sample labels;

[0208] 026: Update the initial model based on the loss function to determine the recognition model.

[0209] The processing unit in this embodiment is configured to determine the prediction result of the image sample based on the image sample and the initial model, determine the loss function based on the prediction result and the sample label, update the initial model based on the loss function, and determine the recognition model.

[0210] The processor in this embodiment is further configured to determine the prediction result of the image sample based on the image sample and the initial model, and to determine the loss function based on the prediction result and the sample label, and to update the initial model based on the loss function to determine the recognition model.

[0211] Specifically, in this embodiment, the electronic device can input image samples into an initial model to obtain prediction results from the initial model for the image samples, such as the predicted location of road navigation arrows in the image samples, the predicted category of road navigation arrows in the image samples, the predicted direction of road navigation arrows in the image samples, and the predicted location of key points of road navigation arrows in the image samples. Then, based on the prediction results of the initial model for the image samples and the sample labels of the image samples, the electronic device can determine the difference between the prediction results and the sample labels of the image samples, thereby calculating the loss function value. Finally, the electronic device can update the weights, biases, and other parameters in the initial model using the loss function value.

[0212] Understandably, after multiple rounds of iterative input of image samples into the initial model, calculation of the loss function value, and updating of the parameters in the model, until the preset training termination conditions are met, the electronic device can output the initial model obtained from the last round of training as the final recognition model. These training termination conditions include, but are not limited to, reaching a preset number of iterations and achieving a preset model recognition accuracy.

[0213] In one example, the loss function values ​​include classification loss, confidence loss, bounding box localization loss, and keypoint localization loss.

[0214] Thus, in this embodiment of the application, the prediction result of the image sample can be determined based on the image sample and the initial model, the loss function can be determined based on the prediction result and the sample label, and the initial model can be updated based on the loss function to determine the recognition model, thereby completing the training of the initial model.

[0215] To more clearly illustrate the training process of the initial model in the embodiments of this application, please refer to the following: Figure 5 , Figure 12 and Figure 13 , Figure 12 and Figure 13 These are all schematic diagrams illustrating application scenarios in certain embodiments of this application, i.e. Figure 12 and Figure 13 As shown, the training process of the recognition model for the task of recognizing navigation arrows in parking lots may include...

[0216] like Figure 12 As shown in S10, multiple navigation arrow images are collected, and each navigation arrow image is labeled. This process specifically includes: first, determining the types and number of navigation arrows to be collected based on task requirements; then, capturing images of the parking lot environment outside the vehicle using a vehicle camera to obtain multiple navigation arrow images; next, filtering the collected navigation arrow images, removing those with duplicate scenes or unclear models; and finally, using annotation tools to label the remaining filtered navigation arrow images, indicating the type, location, and direction of the navigation arrows, ultimately forming a dataset. Furthermore, the specific annotation results of the navigation arrow images can be as follows: Figure 5 As shown.

[0217] like Figure 12As shown in S20, data augmentation was performed on the navigation arrow images within the dataset to expand the dataset. Specifically, to simulate the different brightness levels of the arrows caused by changes in parking lot lighting, images with randomly adjusted brightness were added to the dataset to simulate brightness variations in real-world situations. Simultaneously, to simulate potential scene noise or image blurring, images with randomly blurred effects were introduced into the dataset to simulate image blurring caused by noise or equipment. Furthermore, to address the influence of vehicle direction on the direction of the navigation arrows, and to expand the sample size of the dataset, operations such as translation, scaling, and cropping were performed on the images within the dataset to obtain more image variations, allowing the model trained on the dataset to better adapt to different situations.

[0218] like Figure 12 As shown in S30, suitable anchors for the dataset are obtained. Specifically, first, the width and height of each image in the dataset, as well as the width and height of the corresponding bounding boxes, are read. Then, the navigation arrow images in the dataset are scaled to a size that the model can process, and the size of the bounding boxes is adjusted accordingly to ensure that the bounding boxes of the navigation arrow images can fit the navigation arrow images. Next, bounding boxes with a width and height greater than or equal to x pixels (x≥2) are retained for bounding box filtering, thereby ensuring that the model can recognize targets of a certain size. Subsequently, the bounding boxes of each scaled image are analyzed using the k-means clustering algorithm to obtain k (e.g., 9) most suitable anchors. Next, a genetic algorithm is used to randomly mutate the width and height of the anchors, and the fitness of the mutated anchors is calculated. If the fitness determines that the mutated anchors are better, they are kept for updating; if the fitness determines that the mutated anchors are worse, they are deleted. This step is repeated multiple times (e.g., 1000 times), and the final mutated anchors are sorted according to their area and returned, thus completing the acquisition of the dataset anchors.

[0219] like Figure 13 and Figure 12 As shown in S40, a parking lot navigation arrow recognition model is built and trained. Specifically, firstly, the size, number of channels, and number of prediction branches of the input image are determined according to task requirements. Then, a feature extraction layer is constructed, such as selecting a suitable model from various deep learning network models (e.g., ResNet, MobileNet, CSPDarknet, etc.) based on the application environment and task requirements to determine the feature extraction layer used to extract high-level features of the input image. Next, structures such as FPN and PAN are used to build a feature enhancement layer, allowing the model to obtain feature representations at different scales through the feature enhancement layer. Finally, a feature prediction layer is built using approaches such as dense connections or split prediction (e.g., ...). Figure 9 and Figure 10 (As shown), the model is built. Finally, the model parameters are adjusted using the labeled dataset and the backpropagation algorithm to minimize the loss function, thereby obtaining the trained model weights, thus completing the training of the recognition model.

[0220] Please see Figure 14 Corresponding to the above model training method, this application also provides a vehicle control method, including:

[0221] 21: Based on the recognition model, identify the road navigation arrows in the vehicle environment image, wherein the recognition model is trained using the model training method described above.

[0222] This application provides a control device. The vehicle control method of this application can be implemented by the control device of this application. Specifically, the control device includes a processing unit. The transceiver unit is configured to identify road navigation arrows within a vehicle environment image based on a recognition model, wherein the recognition model is trained using the model training method described above.

[0223] This application also provides an electronic device, which includes a memory and a processor. The vehicle control method of this application can be implemented by the electronic device of this application. Specifically, the memory stores a computer program, and the processor is used to identify road navigation arrows in a vehicle environment image according to a recognition model, wherein the recognition model is trained using the model training method described above.

[0224] Specifically, in the embodiments of this application, the vehicle (or the electronic device in the vehicle, or the control device in the vehicle) can identify the collected vehicle environment images based on a recognition model that has been pre-trained and deployed locally in the vehicle or on a remote server. For example, it can identify the location of the road navigation arrow in the vehicle environment image, the guiding direction indicated by the road navigation arrow in the vehicle environment image, and the type of the road navigation arrow in the vehicle environment image.

[0225] Understandably, the training method for the recognition model can be found in the previous text, and will not be repeated here to avoid repetition.

[0226] Thus, in this embodiment, the model can be trained based on the obtained image samples of road navigation arrows and the sample labels of the image samples. During the training process, the model can be trained using the guiding direction corresponding to the road navigation arrows. As a result, the model can identify the road navigation arrows in the image samples and the guiding direction corresponding to the road navigation arrows. This avoids the situation where the model identifies two road navigation arrows with different or even opposite guiding directions as the same road navigation arrow, reducing the recognition error of road navigation arrows. While ensuring the model's ability to recognize road navigation arrows, it also reduces the training difficulty for the model to acquire this recognition ability to a certain extent. Moreover, it enables vehicles to accurately identify road navigation arrows through the model and thus drive steadily based on the identified road navigation arrows.

[0227] Please see Figure 15 In some embodiments of this application, the vehicle environment images include multiple images; therefore, step 21 includes:

[0228] 210: Perform stitching processing on multiple vehicle environment images to determine the stitched image;

[0229] 211: Based on the recognition model, identify the road navigation arrows within the stitched image.

[0230] The processing unit in this embodiment is configured to stitch together multiple vehicle environment images, determine the stitched image, and identify road navigation arrows within the stitched image based on a recognition model.

[0231] The processor in this embodiment is also used to stitch together multiple vehicle environment images, determine the stitched image, and identify road navigation arrows within the stitched image based on a recognition model.

[0232] Specifically, in order to ensure accurate identification of various road navigation arrows in the vehicle's surrounding environment, in this embodiment of the application, the vehicle can simultaneously capture images of the vehicle's environment from different directions using multiple cameras. Then, the vehicle environment images captured by each camera are stitched together to form a complete panoramic image (i.e., a stitched image). The image is then recognized using a recognition model to identify various road navigation arrows in the vehicle's surrounding environment.

[0233] In one example, a panoramic image (i.e., a stitched image) is similar to Figure 1 The image shown.

[0234] In one example, the vehicle is equipped with four fisheye cameras, one in front, one in the back, one in the left, and one in the right, and the positions and orientations of these four fisheye cameras can cover the four directions of the vehicle: front, back, left, and right.

[0235] Furthermore, in one example, the height and angle of the four fisheye cameras can be considered when installing them to ensure that targets such as the ground, obstacles, and navigation arrows around the vehicle can be captured. At the same time, to ensure synchronized data acquisition from the four fisheye cameras, the image acquisition frequency of these four fisheye cameras can be adjusted accordingly based on the hardware and application requirements.

[0236] Thus, in this embodiment of the application, multiple vehicle environment images can be stitched together to determine a stitched image, and road navigation arrows within the stitched image can be identified according to a recognition model, thereby identifying each road navigation arrow in the vehicle's surrounding environment.

[0237] Please see Figure 16 In some embodiments of this application, step 210 includes:

[0238] 2100: Perform preset distortion correction processing on the vehicle environment image to determine the processed vehicle environment image;

[0239] 2101: Perform stitching on multiple processed vehicle environment images to determine the stitched image.

[0240] Specifically, in order to ensure the reliability of the stitched image and thus the accurate prediction of the road navigation arrow, in this embodiment of the application, distortion correction can be performed on each image captured by the camera to ensure that distortions in the image (such as objects appearing smaller when farther away and larger when closer) are corrected, thereby eliminating distortions in the stitched image and ensuring the reliability and rationality of the stitched image.

[0241] In one example, the vehicle is equipped with four fisheye cameras (front, rear, left, and right). The positions and orientations of these four fisheye cameras cover the vehicle's front, rear, left, and right directions. The vehicle can then perform distortion correction on the raw images captured by these four fisheye cameras at the same time stamp, correcting distortions in each camera's image and obtaining a more accurate view. The vehicle can then stitch these corrected images together to form a complete panoramic image, i.e., a stitched image.

[0242] Thus, in this embodiment of the application, a preset distortion correction process can be performed on the vehicle environment image to determine the processed vehicle environment image, and multiple processed vehicle environment images can be stitched together to determine the stitched image, so that the stitched image can be stitched together based on the distortion-corrected vehicle environment image, thereby ensuring the reliability and rationality of the stitched image.

[0243] Please see Figure 17 In some embodiments of this application, step 21 includes:

[0244] 212: Preprocess the vehicle environment image and determine the preprocessed vehicle environment image;

[0245] 213: Based on the recognition model, identify the road navigation arrows in the preprocessed vehicle environment image.

[0246] The processing unit in this application embodiment is configured to preprocess a vehicle environment image, determine a preprocessed vehicle environment image, and identify road navigation arrows within the preprocessed vehicle environment image according to a recognition model.

[0247] The processor in this embodiment is further configured to preprocess the vehicle environment image, determine the preprocessed vehicle environment image, and identify road navigation arrows within the preprocessed vehicle environment image based on a recognition model.

[0248] Specifically, in order to provide accurate predictions for identification, in this embodiment of the application, the vehicle may preprocess the vehicle environment image to improve the image quality, so that the model can make predictions based on the higher quality vehicle environment image.

[0249] In one example, preprocessing includes at least one of histogram equalization, image resizing, normalization, mean subtraction, and channel swapping.

[0250] Thus, in this embodiment of the application, the vehicle environment image can be preprocessed to improve the image quality, thereby determining the preprocessed vehicle environment image, and the road navigation arrows in the preprocessed vehicle environment image can be identified according to the recognition model, thereby ensuring the accuracy of the recognition model's prediction of the road navigation arrows.

[0251] Please see Figure 18 In some embodiments of this application, step 21 includes:

[0252] 214: Based on the recognition model, identify the vehicle environment image and predict the positions of multiple road navigation arrows in the vehicle environment image;

[0253] 215: Perform non-maximum suppression processing on multiple road navigation arrow position predictions to determine the target road navigation arrow position prediction among the multiple road navigation arrow position predictions;

[0254] 216: Identify road navigation arrows within the vehicle environment image based on the predicted location of the target road navigation arrow.

[0255] The processing unit in this embodiment is configured to recognize a vehicle environment image according to a recognition model, determine multiple road navigation arrow position predictions in the vehicle environment image, perform non-maximum suppression processing on the multiple road navigation arrow position predictions, determine the target road navigation arrow position prediction among the multiple road navigation arrow position predictions, and recognize the road navigation arrows in the vehicle environment image based on the target road navigation arrow position prediction.

[0256] The processor in this embodiment is further configured to recognize a vehicle environment image according to a recognition model, determine multiple road navigation arrow position predictions in the vehicle environment image, perform non-maximum suppression processing on the multiple road navigation arrow position predictions, determine a target road navigation arrow position prediction among the multiple road navigation arrow position predictions, and recognize road navigation arrows in the vehicle environment image based on the target road navigation arrow position prediction.

[0257] Specifically, in the embodiments of this application, the prediction process of the recognition model for vehicle environment images can be divided into two stages: forward propagation and post-processing.

[0258] In one example, during the forward propagation phase, the model processes the vehicle environment image layer by layer based on a feature extraction layer, a feature enhancement layer, and a feature prediction layer to generate an initial prediction result. It should be noted that this initial prediction result includes the prediction of the road navigation arrow position, that is, the prediction of the location of the road navigation arrow in the vehicle environment image.

[0259] In one example, during the post-processing stage, the model can use the Non-Maximum Suppression (NMS) algorithm to filter the predicted road navigation arrow positions, removing one or more road navigation arrow position predictions with high overlap, and determining the final road navigation arrow recognition result based on the filtered road navigation arrow position predictions.

[0260] In one example, after receiving a vehicle environment image, the recognition model's predicted road navigation arrow position can be expressed as (xmin, ymin, xmax, ymax, keypoint1_x, keypoint1_y, keypoint2_x, keypoint2_y). Here, xmin, ymin, xmax, and ymax represent the two-dimensional coordinates of two corner points (e.g., the lower left and upper right corners) within a rectangle indicating the location of the road navigation arrow in the vehicle environment image. keypoint1_x, keypoint1_y, keypoint2_x, and keypoint2_y represent the two-dimensional coordinates of two key points of the road navigation arrow in the vehicle environment image.

[0261] Thus, in this embodiment of the application, nonmaximum suppression processing can be used to filter the multiple road navigation arrow position predictions output by the model, and the road navigation arrows in the vehicle environment image can be determined by the target road navigation arrow position predictions obtained through filtering, thereby ensuring the accuracy and reliability of the road navigation arrow recognition results to a certain extent.

[0262] To more clearly illustrate the reasoning process of the recognition model in the embodiments of this application, please refer to the following: Figure 3 , Figure 4 , Figure 12 , that is Figure 12 As shown in S50, the process of recognizing parking lot ground navigation arrows for panoramic images may include:

[0263] First, the vehicle uses four fisheye cameras installed at the front, rear, left, and right to capture images of the external environment at the same frequency.

[0264] Next, distortion correction is performed on the raw images acquired by the four fisheye cameras at the same time stamp. This ensures that distortions in the images are corrected, resulting in a more accurate view. Subsequently, these corrected images are stitched together to form a complete panoramic image.

[0265] Next, operations such as histogram equalization, image resizing, normalization, mean subtraction, and channel swapping are performed on the panoramic image to preprocess it, thereby enabling the model to make better predictions.

[0266] Then, the preprocessed panoramic image is input into the trained recognition model. Upon receiving the image input, the model processes the input image sequentially through a feature extraction layer, a feature enhancement layer, and a feature prediction layer to generate prediction results. Understandably, the model's prediction results include multiple bounding boxes, each representing the model's prediction of the location of the navigation arrow in the input image.

[0267] To improve the model's prediction accuracy, a non-maximum suppression algorithm can be used to filter the multiple bounding boxes output by the model, removing those with high overlap, and the final target detection result is determined by retaining the remaining bounding boxes (e.g., ...). Figure 3 and Figure 4 As shown in the image, it includes predictive information such as the type, location, and direction of navigation arrows, which can be used to achieve automatic navigation and route planning for vehicles in parking lots.

[0268] This application also provides a vehicle, which includes the above-described electronic device or control device.

[0269] This application also provides a computer-readable storage medium storing a computer program that, when executed by one or more processors, implements the model prediction method or the vehicle control method described above.

[0270] This application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the above-described model prediction method or the above-described vehicle control method.

[0271] In this specification, the terms "specifically," "furthermore," "particularly," "understandably," etc., refer to specific features, structures, materials, or characteristics described in connection with embodiments or examples that are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0272] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the function involved, as will be understood by those skilled in the art to which embodiments of this application pertain.

[0273] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A model training method, characterized in that, include: Obtain image samples of road navigation arrows and sample labels for the image samples, wherein the sample labels are used for the road navigation arrows in the image samples and the corresponding guiding directions of the road navigation arrows; The initial model is trained based on the image samples and the sample labels to determine the recognition model, which is able to recognize the road navigation arrows and the guiding directions in the image samples.

2. The method according to claim 1, characterized in that, The sample label includes at least two key points within the image sample, one key point indicating the bottom of the road navigation arrow and the other key point indicating the top of the road navigation arrow, and the line formed by the at least two key points indicating the guiding direction.

3. The method according to claim 1, characterized in that, The sample label also includes a category label for the image sample and an arrow position label, wherein the arrow position label is used to indicate the location of the road navigation arrow in the image sample.

4. The method according to claim 1, characterized in that, The number of image samples is a first quantity. The step of training an initial model based on the image samples and the sample labels to determine the recognition model includes: A preset data augmentation process is performed on a first number of image samples to determine a second number of image samples, wherein the second number is greater than the first number. The recognition model is determined by training the model based on the second number of image samples and the sample labels.

5. The method according to claim 4, characterized in that, The data enhancement processing includes at least one of image brightness adjustment processing, image blurring processing, image translation processing, image scaling processing, and image cropping processing.

6. The method according to claim 1, characterized in that, The image samples include multiple samples, and the sample labels further include arrow position labels, which are used to indicate the location of the road navigation arrow in the image samples. The step of training an initial model based on the image samples and the sample labels to determine the recognition model includes: Based on the arrow position labels of multiple image samples, determine the target anchor point detection box of the image sample; The recognition model is determined by training a model based on the image samples, the sample labels, and the target anchor point detection boxes.

7. The method according to claim 6, characterized in that, The step of determining the target anchor point detection box of the image sample based on the arrow position labels of the multiple image samples includes: Clustering is performed on the arrow position labels of multiple image samples to determine the target arrow position label among the multiple arrow position labels; Based on the target arrow position label, determine the fitness of multiple preset anchor point detection boxes; The multiple preset anchor point detection boxes are updated based on the fitness, and the target anchor point detection box is determined.

8. The method according to claim 6, characterized in that, The method further includes: Based on the image size of the image that the model can recognize, the image sample and the arrow position label are resized.

9. The method according to claim 6, characterized in that, The arrow location label includes a bounding box indicating the location of the road navigation arrow in the image sample, and the method further includes: The annotation boxes of multiple image samples are filtered according to the preset annotation box size.

10. The method according to claim 1, characterized in that, The initial model includes a feature extraction layer, a feature enhancement layer, and a feature prediction layer; The feature extraction layer is configured to perform feature extraction processing on the image samples to determine image features; The feature enhancement layer is configured to perform a preset enhancement process on the image features to determine the enhanced image features; The feature prediction layer is configured to predict road navigation arrows within the image sample based on the enhanced image features.

11. The method according to claim 10, characterized in that, The feature prediction layer is configured as follows: Based on the enhanced image features, determine at least one of the following confidence levels: arrow location prediction, arrow direction prediction, arrow category prediction, and background pixel prediction within the input image; and / or Based on the enhanced image features, the confidence levels of the predicted arrow location, arrow category, arrow guidance direction, and arrow guidance direction within the input image are determined.

12. The method according to any one of claims 1-11, characterized in that, The step of training an initial model based on the image samples and the sample labels to determine the recognition model includes: Based on the image samples and the initial model, determine the prediction results for the image samples; Based on the prediction results and the sample labels, determine the loss function; The initial model is updated based on the loss function to determine the recognition model.

13. A vehicle control method, characterized in that, include: According to the recognition model, road navigation arrows within a vehicle environment image are identified, wherein the recognition model is trained by the method described in any one of claims 1-12.

14. The method according to claim 13, characterized in that, The vehicle environment images include multiple images, and the step of identifying road navigation arrows within the vehicle environment images according to the recognition model includes: Multiple vehicle environment images are stitched together to determine the stitched image; Based on the recognition model, road navigation arrows within the stitched image are identified.

15. The method according to claim 14, characterized in that, The step of stitching together multiple vehicle environment images to determine the stitched image includes: The vehicle environment image is subjected to preset distortion correction processing to determine the processed vehicle environment image; The multiple processed vehicle environment images are stitched together to determine the stitched image.

16. The method according to claim 13, characterized in that, The step of identifying road navigation arrows within a vehicle environment image based on a recognition model includes: The vehicle environment image is preprocessed to determine the preprocessed vehicle environment image; Based on the recognition model, road navigation arrows within the preprocessed vehicle environment image are identified.

17. The method according to claim 13, characterized in that, The step of identifying road navigation arrows within a vehicle environment image based on a recognition model includes: Based on the recognition model, the vehicle environment image is identified to determine the predicted positions of multiple road navigation arrows in the vehicle environment image; Non-maximum suppression processing is performed on the multiple road navigation arrow position predictions to determine the target road navigation arrow position prediction among the multiple road navigation arrow position predictions; Based on the predicted location of the target road navigation arrow, identify the road navigation arrow within the vehicle environment image.

18. A control device, characterized in that, include: The transceiver unit is configured to acquire image samples of road navigation arrows and sample labels of the image samples, wherein the sample labels are used for the road navigation arrows in the image samples and the corresponding guiding direction of the road navigation arrows; The processing unit is configured to train an initial model based on the image samples and the sample labels to determine a recognition model, which is capable of recognizing the road navigation arrows and the guiding directions in the image samples.

19. A control device, characterized in that, include: The processing unit is configured to identify road navigation arrows within a vehicle environment image based on a recognition model, wherein the recognition model is trained by the method described in any one of claims 1-12.

20. An electronic device, characterized in that, The method includes a memory and a processor, wherein the memory stores a computer program, which, when executed by the processor, implements the method according to any one of claims 1-17.

21. A vehicle, characterized in that, The vehicle includes the device according to any one of claims 18-20.

22. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by one or more processors, implements the method of any one of claims 1-17.

23. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the method described in any one of claims 1-17.

Images