Environment perception method and apparatus, storage medium, and vehicle

Abstract

This disclosure relates to an environmental perception method, apparatus, storage medium, and vehicle. The method includes: acquiring image information of the current environment; inputting the image information into a pre-trained scene classification model and acquiring a classification result output by the scene classification model, the classification result including each scene in a plurality of scenes and a first confidence level corresponding to each scene; determining a perception result of the current environment based on the classification result and a recognition result output by at least one recognition model based on the image information, wherein the recognition result includes a recognized object and a second confidence level corresponding to the recognized object, wherein each scene corresponds to a pre-trained recognition model; and outputting the perception result if the perception result meets preset conditions. This disclosure can improve the efficiency and accuracy of a vehicle's perception of its surrounding environment.

Description

Technical Field

[0001] This disclosure relates to the field of autonomous driving technology, and more specifically, to an environmental perception method, device, storage medium, and vehicle. Background Technology

[0002] Visual vehicle environment perception methods in related technologies use a pre-acquired image dataset to train a single model for inference and perception. However, this method is only applicable to some general scenarios, and will fail to accurately perceive the environment around the vehicle in some long-tail scenarios with limited sample sizes. Summary of the Invention

[0003] To overcome the problems existing in related technologies, this disclosure provides an environmental sensing method, device, storage medium, and vehicle.

[0004] According to a first aspect of the present disclosure, an environment perception method is provided, the method comprising:

[0005] Obtain image information of the current environment;

[0006] The image information is input into a pre-trained scene classification model, and the classification result output by the scene classification model is obtained. The classification result includes each scene in multiple scenes and the first confidence level corresponding to each scene.

[0007] Based on the classification results and the recognition results output by at least one recognition model based on the image information, the perception result of the current environment is determined, wherein the recognition result includes the identified object and the second confidence level corresponding to the object, and each scene corresponds to a pre-trained recognition model;

[0008] If the perception result meets the preset conditions, then the perception result is output.

[0009] Optionally, determining the perception result of the current environment based on the classification result and the recognition result output by at least one recognition model based on the image information includes:

[0010] Based on the classification results, the scene with the highest confidence level is selected as the target scene from the multiple scenes;

[0011] The image information is input into the recognition model corresponding to the target scene, and the recognition result output by the recognition model corresponding to the target scene is obtained. The recognition result is used as the perception result of the current environment.

[0012] Optionally, the step of outputting the perception result if the perception result meets a preset condition includes:

[0013] If the second confidence level in the recognition result is greater than or equal to the first threshold, then the recognition result is output as the perception result.

[0014] Optionally, determining the perception result of the current environment based on the classification result and the recognition result output by at least one recognition model based on the image information includes:

[0015] The image information is input into the recognition model corresponding to each scene to obtain the recognition result corresponding to each scene. The recognition result includes the recognition object and the second confidence level corresponding to the recognition object.

[0016] For each identified object, the second confidence level of the identified object in each scenario and the first confidence level corresponding to each scenario are weighted to obtain the weighted second confidence level of the identified object.

[0017] Each identified object and its weighted second confidence level are determined as the perception result.

[0018] Optionally, the step of outputting the perception result if the perception result meets a preset condition includes:

[0019] If the weighted second confidence level is greater than or equal to the second threshold, then the recognition result corresponding to the weighted second confidence level is output as the perception result.

[0020] Optionally, the step of weighting the second confidence level of the identified object in each scenario and the first confidence level corresponding to each scenario to obtain the weighted second confidence level of the identified object includes:

[0021] The first confidence score corresponding to each scenario in the classification results is normalized to obtain the processed first confidence score.

[0022] For each identified object, the second confidence level of the identified object in each scenario and the processed first confidence level corresponding to each scenario are weighted to obtain the weighted second confidence level of the identified object.

[0023] Optionally, the step of inputting the image information into a pre-trained scene classification model and obtaining the classification result output by the scene classification model includes:

[0024] The region of interest is extracted from the image information using preset rules;

[0025] The region of interest is input into a pre-trained scene classification model, and the classification result output by the scene classification model is obtained.

[0026] Optionally, the method further includes:

[0027] Acquire multiple image samples of pre-labeled recognition objects;

[0028] The multiple image samples are labeled with scene annotations to obtain labeled image samples;

[0029] The scene classification model is trained based on the labeled image samples.

[0030] Optionally, before training the scene classification model based on the labeled image samples, the method further includes:

[0031] The number of image samples corresponding to each scene in the labeled image samples is determined to meet the preset requirements.

[0032] Optionally, the method further includes: for each of the plurality of scenarios, training a recognition model corresponding to the scenario based on the image samples corresponding to the scenario, wherein the plurality of scenarios includes general scenarios and long-tail scenarios, and the image samples corresponding to the general scenarios include the image samples labeled with the long-tail scenarios.

[0033] According to a second aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0034] According to a third aspect of the present disclosure, an environmental sensing device is provided, comprising:

[0035] A memory on which computer programs are stored;

[0036] A processor for executing the computer program in the memory to implement the steps of the method described in the first aspect.

[0037] According to a fourth aspect of the present disclosure, a vehicle is provided, including: the environmental sensing device described in the third aspect.

[0038] The technical solutions provided by the embodiments of this disclosure can include the following beneficial effects: acquiring image information of the current environment; inputting the image information into a pre-trained scene classification model and obtaining the classification result output by the scene classification model, the classification result including each scene in multiple scenes and the first confidence level corresponding to each scene; then determining the perception result of the current environment based on the classification result and the recognition result output by at least one recognition model based on the image information, wherein the recognition result includes the recognized object and the second confidence level corresponding to the recognized object, and each scene corresponds to a pre-trained recognition model; if the perception result meets the preset conditions, the perception result is output. Since the scene classification result can reflect the scene involved in the current environment, the recognition result of the recognition model suitable for the current environment can be selected through the scene classification result to determine the final environment perception result, avoiding the defect that a single recognition model cannot accurately identify the recognition object in a small number of sample scenes, thereby improving the accuracy and efficiency of environment perception.

[0039] Other features and advantages of this disclosure will be described in detail in the following detailed description section. Attached Figure Description

[0040] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the following detailed description to explain the present disclosure, but do not constitute a limitation thereof. In the drawings:

[0041] Figure 1 This is a flowchart illustrating an environmental perception method according to an exemplary embodiment.

[0042] Figure 2 This is a flowchart illustrating an environmental perception method according to another exemplary embodiment.

[0043] Figure 3 yes Figure 2 The flowchart of the training method for the scene classification model is shown in the embodiment.

[0044] Figure 4 yes Figure 2 The flowchart of the training method for the recognition model is shown in the embodiment.

[0045] Figure 5 yes Figure 2 The specific implementation flowchart of steps S240 to S270 is shown in the embodiment.

[0046] Figure 6 yes Figure 2 The embodiment shows a representation of the recognition model output in byte form.

[0047] Figure 7 yes Figure 2The embodiment illustrates a schematic diagram of the non-maximum value suppression process.

[0048] Figure 8 yes Figure 2 The embodiment shows a flowchart of an environmental perception method in a single scenario.

[0049] Figure 9 This is a block diagram illustrating an environmental sensing system according to an exemplary embodiment.

[0050] Figure 10 This is a schematic diagram of the structure of an environmental sensing device according to an exemplary embodiment.

[0051] Figure 11 This is a schematic diagram of the structure of a server according to an exemplary embodiment. Detailed Implementation

[0052] The specific embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit this disclosure.

[0053] It should be noted that all actions involving the acquisition of signals, information, or data in this disclosure are carried out in compliance with the relevant data protection laws and policies of the country where the location is situated, and with authorization from the owner of the relevant device.

[0054] Environmental perception methods in related technologies use a single model for inference and perception.

[0055] Long-tail scenarios refer to a wide variety of scenarios that occur with low probability or are sudden, resulting in a small sample size. While most car usage occurs on standard roads in normal weather, it can also encounter long-tail scenarios such as severe weather or illegally parked vehicles, leading to data imbalance. When collecting datasets for long-tail scenarios, the sample size is often small. If a single model is used for training, the limited number of samples in certain scenarios may prevent the model from learning the correct loss weights. Resampling and adjusting the loss function can increase the weights for long-tail scenario data, negatively impacting the model's performance in general scenarios.

[0056] To address the aforementioned issues, this embodiment proposes an environmental perception method, device, storage medium, and vehicle, which can classify scenes, train recognition models for different scenes, and use different recognition models for different scenes, thereby improving the vehicle's efficiency and accuracy in perceiving the environment.

[0057] The following explains the technical terms used in this embodiment:

[0058] Long-tail scenarios refer to scenarios that are diverse, have a low probability of occurrence, or are sudden, and the sample size of these scenarios is small.

[0059] Region of Interest (ROI): In machine vision and image processing, the region that needs to be processed from the image being processed is called the region of interest, which is delineated using shapes such as rectangles, circles, ellipses, and irregular polygons.

[0060] Figure 1 This is a flowchart illustrating an environmental perception method according to an exemplary embodiment, such as... Figure 1 As shown, this environmental method can be used in vehicles, and the method may include the following steps:

[0061] S110, acquire image information of the current environment.

[0062] In some embodiments, the vehicle is equipped with a camera that can capture images of the current environment in real time, thereby obtaining image information.

[0063] S120, the above image information is input into a pre-trained scene classification model, and the classification result output by the scene classification model is obtained. The classification result includes each scene in multiple scenes and the first confidence level corresponding to each scene.

[0064] In some implementations, the scene classification model can be trained based on multiple image samples, each of which is labeled with a scene tag. The vehicle can input image information into the scene classification model, which can then identify the current scene and output a classification result. It is understood that, since multiple scenes may coexist, the classification result may include a first confidence score for each of the multiple scenes. For example, the classification result may include: a first confidence score a1 for a rainy scene, a first confidence score a2 for a nighttime scene, and a first confidence score a3 for a standard highway driving scene.

[0065] S130, based on the above classification results and the recognition results output by at least one recognition model based on the above image information, determine the perception results of the current environment, wherein the recognition results include the recognized objects and the second confidence level corresponding to the recognized objects, wherein each scene corresponds to a pre-trained recognition model.

[0066] The recognition model is trained on multiple image samples labeled with the objects to be recognized. Based on the input image information, the model outputs the identified objects (e.g., obstacles, pedestrians, lane lines, traffic lights, etc.) and their corresponding second confidence scores, such as b1 for obstacles, b2 for pedestrians, and b3 for lane lines. Each recognition model corresponds to a specific scene. During model training, the image samples can be pre-labeled with scene tags. For example, images labeled with traffic light tags can be labeled with a standard highway driving scene. The recognition model can then be trained using image samples with these scene tags to ensure it corresponds to the specific scene.

[0067] In some implementations, since a higher first confidence level for a scene indicates a higher accuracy in identifying the current environment as belonging to that scene, the scene with the highest first confidence level can be selected based on the classification results. Then, the recognition model corresponding to that scene is used to identify the image information, and the recognition result is used as the perception result of the current environment. For example, if the scene with the highest first confidence level is a rainy scene, the image information can be input into the recognition model corresponding to the rainy scene to obtain the recognition result. For instance, the recognition result might be: the recognition model corresponding to the rainy scene identifies the object as a "pedestrian," and the second confidence level for "pedestrian" is 0.85.

[0068] S140, If the above perception result meets the preset conditions, then output the above perception result.

[0069] Using the example above, for instance, if the preset condition is that the second confidence level in the perception result is greater than or equal to the confidence threshold, and assuming the confidence threshold is 0.7, then it can be determined that the vehicle has perceived the presence of a "pedestrian" in the current environment, and the perception result of "pedestrian" can be output.

[0070] In some implementations, the method can also be applied to a server that can receive data uploaded by the vehicle, process the data, and output the final perception result to the vehicle.

[0071] As can be seen, in this embodiment, image information of the current environment is acquired; this image information is then input into a pre-trained scene classification model, and the classification result output by the scene classification model is obtained. The classification result includes each scene in multiple scenes and the first confidence level corresponding to each scene. Then, based on the classification result and the recognition result output by at least one recognition model based on the image information, the perception result of the current environment is determined. The recognition result includes the identified object and the second confidence level corresponding to the identified object. Each scene corresponds to a pre-trained recognition model. If the perception result meets preset conditions, the perception result is output. Since the scene classification result can reflect the scene involved in the current environment, the recognition result of the recognition model suitable for the current environment can be selected based on the scene classification result to determine the final environment perception result. This avoids the defect that a single recognition model cannot accurately identify the object in a small number of sample scenes, thereby improving the accuracy and efficiency of environment perception.

[0072] Figure 2 This is a flowchart illustrating an environmental perception method according to an exemplary embodiment, such as... Figure 1 As shown, this environmental method can be used in vehicles, and the method may include the following steps:

[0073] S210: Acquire multiple image samples of pre-calibrated recognition objects.

[0074] Optionally, the objects to be identified may include, but are not limited to, pedestrians, obstacles, vehicles, lane lines, traffic lights, etc. For example, images with pre-labeled objects can be obtained from an image database as image samples; alternatively, images can be captured on-site, and the objects to be identified in the images can be labeled and saved as image samples. Then, these multiple image samples can be used as an image training dataset.

[0075] S220, the above multiple image samples are labeled with scene annotations to obtain labeled image samples.

[0076] Following the example above, for the image training dataset obtained above, scene classification and annotation can be performed on each image sample in the dataset, labeling the scene to which each image sample belongs. For example, image samples with good lighting conditions, field of view, weather, and high visibility can be labeled as general scenes. For image samples that are difficult to identify, it is best to label them as other special scenes such as rain, night, sunset, snow, or fog, depending on the actual situation of the image sample. For some image samples that belong to multiple scenes, these coexisting scenes can all be labeled as the scene category of the image sample. In this way, the scene recognition model can learn features based on the characteristics of the scene.

[0077] S230, Based on the above-annotated image samples, the above scene classification model is trained.

[0078] In some implementations, the following may be included before step S230:

[0079] The number of image samples corresponding to each scene in the labeled image samples is determined to meet the preset requirements.

[0080] For example, since the number of image samples that can be collected for some long-tail or special scenarios may be relatively small, the number of image samples for each scenario can be adjusted to meet a preset requirement to ensure a balanced number of image samples for each scenario in the scene classification model. For instance, if the number of image samples for a certain scenario is within a specified range, it can be determined that the number of image samples for that scenario meets the preset requirement. If there are too many image samples for a particular scenario, some can be deleted; if there are too few, some can be added.

[0081] As an example, in practical applications, the specific implementation process of steps S210 to S230, i.e., the training method of the scene classification model, can be as follows: Figure 3 As shown, this can be achieved through steps 101 to 104.

[0082] Step 101: This processing step requires a pre-annotated image training dataset for perception tasks such as pedestrians, obstacles, vehicles, lane lines, and traffic lights. The annotation content of this image training dataset determines the type of recognition model task that can be trained. Only tasks annotated in this image training dataset can be trained correctly. This perception task corresponds to the recognition object in the above embodiment.

[0083] Step 102: In the preprocessing stage, using the labeled dataset, perform cropping and image enhancement operations on each image. Alternatively, perform small-angle rotations, translations, scaling, and other data enhancement operations on the images to extract a Region of Interest (ROI) and add it to the training data. The dimensions of this ROI should match the input dimensions of the scene classification model. In this step, preprocessed data with scene classification labels can be output.

[0084] Step 103: In the preprocessed data output from Step 102, corresponding to real-world situations, there will be relatively little data on special scenarios (long-tail scenarios) compared to general scenario data. The dataset output from Step 102 needs to be resampled. Specifically, resampling can include oversampling and undersampling (where undersampling involves removing samples from the majority class, and oversampling involves adding more examples to the minority class). Specifically, the non-general scenario data in the preprocessed data output from Step 102 is sampled multiple times (oversampling), or only a portion of the general scenario data is used as training data (undersampling), so that the distribution of each scenario data roughly conforms to an average distribution. Using the resampled preprocessed data, the scene classification model is trained, outputting the scene classification training model. This completes the training of the scene classification model (or attention model) for imbalanced data.

[0085] Alternatively, for the problem of imbalanced training data in multi-class classification tasks, besides resampling before training, a similar effect can be achieved by dynamically adjusting the weights of the loss during training. Taking FocalLoss as an example: the preprocessed data output from step 102 is used as input to the classification model, which outputs the classification result [a1, a2, ..., a...]. n ], a i The confidence level is given for each scenario.

[0086] Then, when calculating the loss value, use As the training loss function. Where p j γ represents the accuracy of the prediction for the j-th scene; j To control the weakening effect of training weights on high-accuracy samples in the j-th scene, this parameter needs to be pre-set before training; the larger the parameter, the stronger the weakening effect. α (t,j) To control the weights of positive and negative samples in the j-th scene. If a sample actually only belongs to the i-th scene, then p i =a i α (t,i) =α i For k≠i, then p k =1-a k α (t,k) =1-α k , where α k This represents the bias towards positive and negative samples in the preset k-th scenario. The larger this value is, the greater the weight of the positive sample.

[0087] Step 104: For the model output in Step 103, retain the required network structure and parameters, and solidify and transform it to generate a model file for inference.

[0088] In some embodiments, the method may further include:

[0089] For each of the above scenarios, a recognition model corresponding to the above scenario is trained based on the image samples corresponding to the above scenario. The above scenarios include general scenarios and long-tail scenarios. The image samples corresponding to the general scenarios include the image samples with the labels of the above long-tail scenarios.

[0090] As an example, in practical applications, the training method for the recognition model can be as follows: Figure 4 As shown, the specific steps may include:

[0091] Step 201: Use all image samples (the complete set) of the above image training dataset (hereinafter referred to as the dataset) to train a general scene recognition model. When training the general scene recognition model, image samples corresponding to long-tail scenes can also be included in the training dataset of the general model, thereby improving the generalization ability of the general scene inference process. In this step, the general scene perception training model can be output.

[0092] Step 202: Based on the annotation results of step 101 in the above embodiments, generate image data subsets corresponding to each scene from all images in the dataset (the complete set of image data). If an image sample exists in multiple scenes, it will be assigned to multiple scene data subsets.

[0093] Step 203: Retrain the general scene perception training model trained in Step 201 using data subsets corresponding to different scenarios (Step 202). Iterate the training until the model's error is reduced to a specified range, then output the training model corresponding to each scenario. Examples include a pedestrian detection model for snowy scenarios, a lane line recognition model for nighttime scenarios, and a vehicle detection model for foggy scenarios.

[0094] Taking the YOLOop recognition model in a rainy scene as an example, from the output of step 101, all images labeled as rainy scenes are extracted to form a subset of training data. Assume the training dataset input in step 101 labels targets such as pedestrians, vehicles, drivable areas, and lane lines. Since a single YOLOop model can simultaneously predict target detection, drivable areas, and lane lines, then after training in step 203, a recognition model capable of predicting pedestrians, vehicles, drivable areas, and lane lines in a rainy scene will be output.

[0095] Step 204: For the model output in step 203, retain the required network structure and parameters, and solidify and transform it to generate a model file for inference.

[0096] S240, acquire image information of the current environment.

[0097] The specific implementation of step S240 can be found in step 110, so it will not be repeated here.

[0098] S250, the above image information is input into a pre-trained scene classification model, and the classification result output by the scene classification model is obtained. The classification result includes each scene in the multiple scenes and the first confidence score corresponding to each scene.

[0099] In some implementations, step S250 may include the following specific implementations:

[0100] The region of interest is extracted from the above image information using preset rules; the region of interest is then input into a pre-trained scene classification model, and the classification result output by the scene classification model is obtained.

[0101] For example, a Region of Interest (ROI) can be extracted from an image sample, where the ROI's dimensions should match the input of the scene classification model. Alternatively, the ROI can be scaled to match the scene classification model's input. The ROI location should ideally encompass elements such as roads, sky, distant objects, and streetlights to ensure that image information including road type, weather, and lighting conditions is provided to the scene classification model. Furthermore, the ROI selection should avoid scene-irrelevant or repetitive image data, such as vehicle hoods or excessive sky images. Generally, selecting the central portion of the image is sufficient. Finally, the output image data should have the same dimensions and format as the input of the scene classification model.

[0102] S260, based on the above classification results and the recognition results output by at least one recognition model based on the above image information, determine the perception result of the current environment. The recognition result includes the identified object and the second confidence level corresponding to the identified object, and each scene corresponds to a pre-trained recognition model.

[0103] In some implementations, the specific implementation of step S260 may include:

[0104] Step S261A: Based on the above classification results, select the scenario with the highest first confidence level from the above multiple scenarios as the target scenario.

[0105] For example, based on the classification results, if the first confidence level for a rainy scene is 0.4, the first confidence level for a nighttime scene is 0.3, and the first confidence level for a standard highway driving scene is 0.3, then the rainy scene can be identified as the target scene.

[0106] Step S262A: Input the above image information into the recognition model corresponding to the target scene, obtain the recognition result output by the recognition model corresponding to the target scene, and use the recognition result as the perception result of the current environment.

[0107] Using the above example, the recognition model corresponding to the rainy scene can be used as the recognition model for the current environment. The above image information is input into the recognition model, and the output of the recognition model is used as the perception result of the current environment. For example, the perception result is: the second confidence level of the presence of "pedestrian" is 0.7.

[0108] In other embodiments, specific implementations of step S260 may include:

[0109] Step S261B: Input the above image information into the recognition model corresponding to each scene to obtain the recognition result corresponding to each scene. The recognition result includes the recognition object and the second confidence level corresponding to the recognition object.

[0110] For example, current recognition models include recognition model A for rainy scenes, recognition model B for nighttime scenes, and recognition model C for standard highway driving scenes. Image information x can be input into recognition model A, recognition model B, and recognition model C respectively. Assume that recognition model A outputs the following recognition results: a second confidence score of 0.9 for "pedestrian" and 0.6 for "obstacle"; recognition model B outputs the following recognition result: a second confidence score of 0.3 for "pedestrian"; and recognition model C outputs the following recognition result: a second confidence score of 0.1 for "obstacle".

[0111] Step S262B: For each identified object, the second confidence level of the identified object in each scenario and the first confidence level corresponding to each scenario are weighted and processed to obtain the weighted second confidence level of the identified object.

[0112] Continuing with the above example, the first confidence level for a rainy scene is 0.5; the first confidence level for a nighttime scene is 0.3; and the confidence level for a standard highway driving scene is 0.2. Therefore, the second confidence level for "pedestrian" weighted by the first confidence level is 0.5*0.9 + 0.3*0.3 = 0.54. The second confidence level for "obstacle" weighted by the first confidence level is 0.5*0.6 + 0.3*0.1 = 0.33. This process can be repeated to obtain the second confidence level of the identified object in each recognition result after weighting by the first confidence level.

[0113] As one implementation method, the specific implementation of step S262B may include:

[0114] The first confidence score for each scenario in the above classification results is normalized to obtain the processed first confidence score. For each identified object, the second confidence score of the identified object in each scenario and the processed first confidence score for each scenario are weighted to obtain the weighted second confidence score of the identified object.

[0115] In this embodiment, by normalizing the first confidence scores corresponding to different scenarios in the classification results, the second confidence scores can be easily weighted using the first confidence scores.

[0116] Step S263B: The weighted second confidence level is determined as the perceived result.

[0117] S270. If the above perception results meet the preset conditions, then output the above perception results.

[0118] In some implementations, step S270 may include the following specific implementations:

[0119] S273A, if the second confidence level in the above identification result is greater than or equal to the first threshold, then the above identification result is output as the above perception result.

[0120] Continuing with the example above, let's say the second confidence level for determining the presence of a "pedestrian" is 0.7. If the first threshold is 0.5, then it can be determined that the "pedestrian" has been identified in the current environment, and therefore the perception result of the pedestrian can be output. Optionally, the output can be displayed on a screen, announced via voice, uploaded to a server, etc., and is not limited here.

[0121] In some implementations, step S270 may include the following specific implementations:

[0122] S273B, if the weighted second confidence level is greater than or equal to the second threshold, then the recognition result corresponding to the weighted second confidence level is output as the perception result.

[0123] Continuing with the example above, if the second confidence level for determining the perception result as "pedestrian" is 0.54 and the second confidence level for "obstacle" is 0.33, and if the second threshold is 0.5, then it can be determined that the object being identified, "pedestrian," is being detected in the current environment, and therefore the perception result of "pedestrian" can be output. The first and second thresholds can be the same or different; this is not limited here.

[0124] In practical applications, the scene classification model and the recognition model can be a whole perception model, that is, the perception model includes two sub-models: the scene classification model and the recognition model.

[0125] As an example, in practical applications, the specific implementation process of steps S240 to S270 can be as follows: Figure 5 As shown:

[0126] S301: Acquire image information that matches the dimensions of the perception model through the vehicle's camera, and convert the image information (hereinafter also referred to as image data) into the format required for input to the perception model.

[0127] S302: Perform a cropping operation on the above image information.

[0128] Extract a Region of Interest (ROI), whose dimensions must match the input requirements of the scene classification model. Alternatively, scale the ROI to match the input dimensions of the scene classification model. The ROI location should ideally encompass elements such as roads, sky, distant objects, and streetlights to ensure that information about road type, weather, and lighting conditions is provided to the scene classification model. Furthermore, the ROI should avoid irrelevant or repetitive image data, such as vehicle hoods or excessive sky images. Generally, selecting the central portion of the image is sufficient. Finally, output image data with the same dimensions and format as the input to the scene classification model.

[0129] S303: After obtaining image data from S302, the pre-trained scene classification model outputs the classification result [a1, a2, ..., a...]. n ], a i Let a be the confidence level for each scenario, 0 ≤ a i ≤1, and ∑a i =1. From the overall architecture perspective, a i These correspond to weights for different scenes. Due to the simplicity of the task, this scene classification model can use a small-sized image cropped from the ROI using S302 for inference. The final step in the scene classification model is to normalize the output, which can be done using the softmax function. K is the length of the input parameters, e i This is the i-th input to the softmax function.

[0130] S304: In this step, image information obtained from S301 is input into the recognition models for each scene. The recognition models for each scene are then used to perform inference and output the recognized object and its confidence level for that scene. For example, in one embodiment, inference for each scene is performed without interaction between them. Each scene can use different model structures tailored to its own characteristics. For instance, in a sunset scene, a color space conversion can be added before the neural network; in a rain scene, image denoising or super-resolution processing can be added before the neural network. Taking object detection YOLO as an example, its model output byte format can be represented as... Figure 6 After non-maximum suppression (NMS), the starting coordinates, width, and class of the detection box can be obtained, and the confidence score can be the highest among the class scores. For some perceptual models that do not output a confidence score, the confidence score can be considered to be 1. As an example, the NMS process can be as follows: Figure 7 As shown.

[0131] S305: Scene classification results using the output of S303 [a1, a2, ..., a n The output of S304 is weighted to obtain the perception result and confidence level of the scene classification result.

[0132] S306: A threshold is used to judge the weighted confidence level. The threshold is mainly determined manually with reference to parameters such as accuracy and recall obtained from the overall model validation test, and is generally set to 0.7 to 0.95. The perceived results that are greater than the threshold are used as the final output.

[0133] In the above embodiments, to reduce the computational cost of the inference process of the scene classification model, the input image is selected and cropped before this process to reduce the computational load by reducing the image size. However, since inference operations are performed for each scene, the more scene models there are, the more computational load will increase.

[0134] To reduce computational load, another embodiment can use only the recognition model corresponding to the scene with the highest confidence, i.e., the scene with the highest probability. This embodiment is applicable to single-scene cases and does not need to consider the situation where multiple scenes coexist, such as sunset and rain. As an example, such as... Figure 8 As shown, the specific implementation of this embodiment can be as described in steps S401 to S405:

[0135] S401~S402: This step is the same as S301~S302.

[0136] S403: After inputting the extracted ROI image data into the scene classification model, the output scene classification result can be represented in the form of S303 [a1, a2, ..., an The scene classification model in this step can also be left unnormalized during inference, because S404 will take the maximum value in the output vector of S403 and use its corresponding scene model for inference.

[0137] S404: From the vector output by S403, select the recognition model corresponding to the scene with the highest confidence level as the inference model. Use the image data S401 obtained from the vehicle for recognition inference, and output the recognition result and confidence level.

[0138] S405: The confidence level obtained in step 404 can be judged using a confidence threshold. The confidence threshold can be determined by referring to parameters such as accuracy and recall obtained from the overall model validation test. Optionally, it is generally set to 0.7 to 0.95. As an example, if the identified objects with a confidence level greater than the confidence threshold obtained in step 404, they can be output as the final result.

[0139] As can be seen in this embodiment, the sample size for long-tail scenarios is relatively small, resulting in poor environmental perception performance when using a single model in such scenarios. Furthermore, resampling strategies can negatively impact the model's general representation capabilities. To address this, the vehicle environmental perception scenarios are classified, and corresponding data is used for retraining for each category. During inference, the scene classification result is used as the weight to select the appropriate scene model. This isolates general and long-tail scenarios, preventing data with few common features from being trained / inferred on a single model. This approach focuses more on long-tail scenarios, specifically handling special scenarios, resulting in higher recognition rates. It also offers greater controllability, allowing for targeted optimization for scenarios with high recognition difficulty. Additionally, to address the issue of uneven sample feature distribution and the difficulty in learning features with a small sample size, samples with similar feature distributions are used as a scene and training set, trained with a separate perception model. Since the training data feature distribution of non-general scene perception models is similar, the model learns more easily, leading to higher recognition rates.

[0140] Figure 9 This is a flowchart illustrating an environmental sensing system according to an exemplary embodiment, such as... Figure 9 As shown, the system 50 may include: an image information acquisition module 51, a classification result acquisition module 52, a perception result determination module 53, and an output module 54. Wherein:

[0141] Image information acquisition module 51 is used to acquire image information of the current environment.

[0142] The classification result acquisition module 52 is used to input the above image information into the pre-trained scene classification model and obtain the classification result output by the above scene classification model. The classification result includes each scene in multiple scenes and the first confidence level corresponding to each scene.

[0143] The perception result determination module 53 is used to determine the perception result of the current environment based on the classification result and the recognition result output by at least one recognition model based on the image information. The recognition result includes the recognized object and the second confidence level corresponding to the recognized object. Each scene corresponds to a pre-trained recognition model.

[0144] The output module 54 is used to output the above perception result if the above perception result meets the preset conditions.

[0145] In some implementations, the perception result determination module 53 includes:

[0146] The target scene determination submodule is used to select the scene with the highest first confidence score from the multiple scenes based on the above classification results as the target scene.

[0147] The perception result acquisition submodule is used to input the above image information into the recognition model corresponding to the above target scene, and to obtain the recognition result output by the recognition model corresponding to the above target scene, and to use the above recognition result as the perception result of the above current environment.

[0148] In some implementations, the output module 54 is specifically used to output the recognition result as the perception result if the second confidence level in the recognition result is greater than or equal to the first threshold.

[0149] In some implementations, the perception result determination module 53 includes:

[0150] The recognition result acquisition submodule is used to input the above image information into the recognition model corresponding to each scene to obtain the recognition result corresponding to each scene. The recognition result includes the recognition object and the second confidence level corresponding to the recognition object.

[0151] The second confidence level acquisition submodule is used to perform weighted processing on the second confidence level of the identified object in each scenario and the first confidence level corresponding to each scenario for each identified object, so as to obtain the weighted second confidence level of the identified object.

[0152] The perception result determination submodule is used to determine each of the above-mentioned identified objects and the weighted second confidence level of each of the above-mentioned identified objects as the above-mentioned perception result.

[0153] In some implementations, the output module 54 is specifically used to output the recognition result corresponding to the weighted second confidence level as the perception result if the weighted second confidence level is greater than or equal to the second threshold.

[0154] In some implementations, the second confidence level acquisition submodule is specifically used to normalize the first confidence level corresponding to each scenario in the above classification results to obtain the processed first confidence level; for each identified object, the second confidence level of the identified object in each scenario and the processed first confidence level corresponding to each scenario are weighted to obtain the weighted second confidence level of the identified object.

[0155] In some implementations, the classification result acquisition module 52 includes:

[0156] The Region of Interest Extraction (ROI) submodule is used to extract the region of interest from the above image information according to preset rules.

[0157] The classification result acquisition submodule is used to input the above-mentioned region of interest into the pre-trained scene classification model and obtain the classification result output by the above-mentioned scene classification model.

[0158] In some embodiments, the system 50 further includes:

[0159] The sample acquisition module is used to acquire multiple image samples of pre-labeled recognition objects;

[0160] The annotation module is used to annotate the above multiple image samples to obtain annotated image samples;

[0161] The training module is used to train the scene classification model based on the labeled image samples.

[0162] In some embodiments, the system 50 further includes:

[0163] The quantity requirement determination module is used to determine whether the image samples corresponding to each scene in the labeled image samples meet the preset quantity requirements.

[0164] In some embodiments, the system 50 further includes:

[0165] The recognition model training module is used to train a recognition model for each of the above multiple scenarios based on the image samples corresponding to the above scenarios. The above multiple scenarios include general scenarios and long-tail scenarios. The image samples corresponding to the above general scenarios include the image samples with the labels of the above long-tail scenarios.

[0166] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0167] Figure 10 This is a block diagram illustrating an environmental sensing device 700 according to an exemplary embodiment. (See diagram below.) Figure 10 As shown, the environmental sensing device 700 may include a processor 701 and a memory 702, which may be located in the vehicle terminal of the environmental sensing device. The environmental sensing device 700 may also include one or more of a multimedia component 703, an input / output (I / O) interface 704, and a communication component 705.

[0168] The processor 701 controls the overall operation of the environmental sensing device 700 to complete all or part of the steps in the environmental sensing method described above. The memory 702 stores various types of data to support the operation of the environmental sensing device 700. This data may include, for example, instructions for any application or method operating on the environmental sensing device 700, and application-related data such as contact data, sent and received messages, images, audio, video, etc. The memory 702 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. Multimedia component 703 may include a screen and an audio component. The screen may be, for example, a touchscreen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signals may be further stored in memory 702 or transmitted via communication component 705. The audio component also includes at least one speaker for outputting audio signals. I / O interface 704 provides an interface between processor 701 and other interface modules, such as a keyboard, mouse, buttons, etc. These buttons may be virtual or physical buttons. Communication component 705 is used for wired or wireless communication between the environmental sensing device 700 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, 4G, NB-IoT, eMTC, or other 5G technologies, or combinations thereof, is not limited here. Therefore, the corresponding communication component 705 may include: a Wi-Fi module, a Bluetooth module, an NFC module, etc.

[0169] In an exemplary embodiment, the environment sensing device 700 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the environment sensing method described above.

[0170] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, which, when executed by a processor, implement the steps of the above-described environment sensing method. For example, the computer-readable storage medium may be the memory 702 including program instructions, which may be executed by the processor 701 of the environment sensing device 700 to complete the above-described environment sensing method.

[0171] In another exemplary embodiment, a vehicle is also provided, which includes the environmental sensing device described in the above embodiments.

[0172] Figure 11 This is a block diagram illustrating a server 1900 according to an exemplary embodiment. For example, server 1900 may be provided as a server. (Refer to...) Figure 11 Server 1900 includes processors 1922, which may be one or more, and memory 1932 for storing computer programs executable by processors 1922. The computer programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, processor 1922 may be configured to execute the computer program to perform the aforementioned environment-aware method.

[0173] Additionally, server 1900 may include a power supply component 1926 and a communication component 1950. The power supply component 1926 can be configured to perform power management for server 1900, and the communication component 1950 can be configured to enable communication for server 1900, such as wired or wireless communication. Furthermore, server 1900 may include an input / output (I / O) interface 1958. Server 1900 can operate on an operating system, such as Windows Server, stored in memory 1932. TM Mac OSXTM Unix TM Linux TM etc.

[0174] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, which, when executed by a processor, implement the steps of the above-described environment-aware method. For example, the non-transitory computer-readable storage medium may be the memory 1932 including the program instructions, which may be executed by the processor 1922 of the server 1900 to complete the above-described environment-aware method.

[0175] In another exemplary embodiment, a computer program product is also provided, the computer program product comprising a computer program executable by a programmable device, the computer program having a code portion for performing the above-described environment perception method when executed by the programmable device.

[0176] The preferred embodiments of this disclosure have been described in detail above with reference to the accompanying drawings. However, this disclosure is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this disclosure, various simple modifications can be made to the technical solutions of this disclosure, and these simple modifications all fall within the protection scope of this disclosure.

[0177] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, this disclosure will not describe the various possible combinations separately.

[0178] Furthermore, various different embodiments of this disclosure can be combined in any way, as long as they do not violate the spirit of this disclosure, they should also be regarded as the content disclosed in this disclosure.

Claims

1. An environmental perception method, characterized in that, include: Obtain image information of the current environment; The image information is input into a pre-trained scene classification model, and the classification result output by the scene classification model is obtained. The classification result includes each scene in multiple scenes and the first confidence level corresponding to each scene. Based on the classification results and the recognition results output by at least one recognition model based on the image information, the perception result of the current environment is determined, wherein the recognition result includes the identified object and the second confidence level corresponding to the object, and each scene corresponds to a pre-trained recognition model; If the perception result meets the preset conditions, then the perception result is output; The step of determining the perception result of the current environment based on the classification result and the recognition result output by at least one recognition model based on the image information includes: The image information is input into the recognition model corresponding to each scene to obtain the recognition result corresponding to each scene. The recognition result includes the recognition object and the second confidence level corresponding to the recognition object. For each identified object, the second confidence level of the identified object in each scenario and the first confidence level corresponding to each scenario are weighted to obtain the weighted second confidence level of the identified object. Each identified object and its weighted second confidence level are determined as the perception result.

2. The method according to claim 1, characterized in that, The step of outputting the perception result if the perception result meets the preset conditions includes: If the weighted second confidence level is greater than or equal to the second threshold, then the recognition result corresponding to the weighted second confidence level is output as the perception result.

3. The method according to claim 1, characterized in that, The step of weighting the second confidence level of the identified object in each scenario and the first confidence level corresponding to each scenario to obtain the weighted second confidence level of the identified object includes: The first confidence score corresponding to each scenario in the classification results is normalized to obtain the processed first confidence score. For each identified object, the second confidence level of the identified object in each scenario and the processed first confidence level corresponding to each scenario are weighted to obtain the weighted second confidence level of the identified object.

4. The method according to any one of claims 1 to 3, characterized in that, The step of inputting the image information into a pre-trained scene classification model and obtaining the classification result output by the scene classification model includes: The region of interest is extracted from the image information using preset rules; The region of interest is input into a pre-trained scene classification model, and the classification result output by the scene classification model is obtained.

5. The method according to any one of claims 1 to 3, characterized in that, The method further includes: Acquire multiple image samples of pre-labeled recognition objects; The multiple image samples are labeled with scene annotations to obtain labeled image samples; The scene classification model is trained based on the labeled image samples.

6. The method according to claim 5, characterized in that, Before training the scene classification model based on the labeled image samples, the method further includes: The number of image samples corresponding to each scene in the labeled image samples is determined to meet the preset requirements.

7. The method according to any one of claims 1 to 3, characterized in that, The method further includes: For each of the multiple scenarios, a recognition model corresponding to the scenario is trained based on the image samples corresponding to the scenario. The multiple scenarios include general scenarios and long-tail scenarios. The image samples corresponding to the general scenarios include image samples with labels of the long-tail scenarios.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method according to any one of claims 1 to 7.

9. An environmental sensing device, characterized in that, include: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the steps of the method according to any one of claims 1 to 7.

10. A vehicle, characterized in that, include: The environmental sensing device as described in claim 9.

Images