Image display method and device, vehicle and computer readable storage medium

Abstract

The invention provides an image display method and device, a vehicle and a computer readable storage medium, and the method is applied to the technical field of vehicle control, and comprises the steps: obtaining an image generated by carrying out the three-dimensional modeling of an environment scene outside the vehicle, inputting the image into a target recognition model, obtaining a first image comprising the target objects in the vehicle and the environment outside the vehicle, and further judging whether the overlapped target objects exist in the first image or not; if there is an overlapped target object in the first image; if yes, keeping the relative direction relation between the overlapped target objects and the vehicle unchanged, and separating the overlapped target objects to obtain a second image; and displaying the second image through a display device of the vehicle. According to the method, a driver can clearly obtain the real position relation between the external environment object and the vehicle through the image displayed on the instrument panel, so that the driver can make a driving decision conveniently, and the driving safety and the driver experience are improved.

Description

Technical Field

[0001] The present application relates to the field of vehicle control technology, and more specifically, to an image display method, device, vehicle and computer-readable storage medium in the field of vehicle control technology. Background Art

[0002] At present, with the development and popularization of autonomous driving technology, more and more vehicles are equipped with instrument restoration world modules. The instrument restoration world module can process the surrounding environment information recognized by ADAS (Advanced Driver Assistance Systems) and perform three-dimensional modeling, and then display it on the dashboard in the form of an image to help the driver better understand the situation around the vehicle.

[0003] However, due to the limited recognition capabilities of vehicle hardware, environmental interference and imperfect data processing algorithms, data errors may occur, resulting in overlapping targets in the image displayed on the dashboard, which in turn provides the driver with erroneous information and affects driving decisions and safety.

[0004] Therefore, it is urgent to propose effective solutions to the above problems. Summary of the invention

[0005] The present application provides an image display method, device, vehicle and computer-readable storage medium. The present application enables the driver to clearly obtain the positional relationship between the real external environment and the vehicle through the image displayed on the dashboard, which facilitates the driver to make driving decisions and improves driving safety and driver experience.

[0006] In a first aspect, an image display method is provided, the image display method comprising: obtaining an initial image corresponding to a vehicle; wherein the initial image is an image generated by three-dimensional modeling of an external environment scene; using a pre-trained target recognition model to recognize the initial image to obtain a first image; wherein the first image includes the vehicle, a target object in the external environment and a bounding box of the target object; judging whether there is an abnormal object in the first image according to the respective bounding boxes of each target object; wherein the abnormal object includes a target object with an overlapping bounding box in the first image; if so, keeping the relative direction relationship between the target object in the abnormal object and the vehicle unchanged, adjusting the distance between the target objects in the abnormal object so that the bounding boxes of the target objects in the abnormal object do not overlap, to obtain a second image; and displaying the second image using a display device of the vehicle.

[0007] Through the above technical solution, an initial image generated by three-dimensional modeling of the vehicle's external environment scene is obtained, and the initial image is input into a pre-trained target recognition model to obtain a first image including a vehicle, a target object in the vehicle's external environment, and a bounding box of the target object. According to the bounding box of each target object, it is determined whether there is an abnormal object in the first image; wherein the abnormal object includes a target object whose bounding boxes overlap each other. If an abnormal object is detected in the first image; the relative direction relationship between the target object in the abnormal object and the vehicle is kept unchanged, and the distance between the target objects in the abnormal object is adjusted so that the bounding boxes of the target objects in the abnormal object do not overlap, and a second image is obtained; and the second image is displayed through the vehicle's display device. The present application obtains an initial image generated by three-dimensional modeling of the vehicle's external environment scene; the initial image is recognized by a pre-trained target recognition model, and when it is detected that there is overlap of target objects in the initial image, the relative direction relationship between the overlapping target objects and the vehicle is kept unchanged, and the distance between the overlapping target objects and the vehicle is adjusted to obtain a corrected image, so that there is no overlap of target objects in the corrected image, and the direction relationship between the target objects and the vehicle is accurate. The corrected image is displayed on the vehicle's dashboard so that the driver can clearly obtain the positional relationship between the actual external environment and the vehicle through the image displayed on the dashboard, which facilitates the driver to make driving decisions and improves driving safety and driver experience.

[0008] In combination with the first aspect, in some possible implementations, adjusting the distance between target objects in the abnormal objects so that the bounding boxes of the target objects in the abnormal objects do not overlap, and obtaining the second image includes: obtaining the distance between each target object in the abnormal objects and the vehicle; determining the first object and the second object according to the distance between each target object and the vehicle; keeping the position of the second object unchanged, and for each first object, keeping the relative direction relationship between the first object and the vehicle unchanged, moving the first object to increase the distance between the first object and the second object, so that the bounding boxes of the target objects in the abnormal objects do not overlap, and obtaining the second image.

[0009] In combination with the first aspect and the above implementation, in some possible implementations, determining the first object and the second object according to the distance between each target object and the vehicle includes: determining the target object corresponding to the minimum value of the distance between each target object and the vehicle as the second object; and determining the target objects other than the second object among the abnormal objects as the first object. Based on the above technical solution, determining the target object corresponding to the minimum distance among the distances between multiple target objects and the vehicle as the second object can keep the position of the target object closer to the vehicle unchanged, thereby improving the reliability of the position information of the target object in the displayed image.

[0010] In combination with the first aspect and the above implementation, in some possible implementations, moving the first object to increase the distance between the first object and the second object includes: moving the first object in a direction away from the vehicle to increase the distance between the first object and the second object. Based on the above technical solution, by moving the first object in a direction away from the vehicle, the driver can preferentially obtain the position information of the target object that is closer to the vehicle, which helps the driver make driving decisions.

[0011] In combination with the first aspect and the above-mentioned implementation, in some possible implementations, the target recognition model includes a classifier and a feature extraction layer connected in sequence, and there are multiple feature extraction layers. The image display method also includes: freezing the model parameters of all feature extraction layers, and using the target image sample set to train the classifier; when the classifier training is completed, unfreeze the model parameters of each feature extraction layer in sequence according to the preset connection order, and use the target image sample set to train each unfrozen feature extraction layer until the last feature extraction layer of all feature extraction layers is trained; wherein the preset connection order is a connection order from bottom to top or a connection order from top to bottom. Based on the above technical solution, the method of gradually unfreezing the feature extraction layers is used to train the target recognition model, which can not only prevent overfitting and improve model performance, but also improve training efficiency and model stability. By flexibly selecting the unfreezing order, it can better adapt to different task requirements and improve the generalization ability of the model.

[0012] In combination with the first aspect and the above implementation, in some possible implementations, the image display method further includes: obtaining multiple first image samples; wherein the multiple first image samples include multiple preset types of target objects; classifying the multiple first image samples to obtain multiple image sample sets; wherein each image sample set includes multiple preset types of target objects; and using the multiple image sample sets as target image sample sets. Based on the above technical solution, by using a small batch training method, the image samples are divided into multiple small batches for training, and a class-balanced sampling method is used to ensure that each small batch contains various categories of target objects, thereby improving the model's recognition ability for different categories and reducing the risk of model overfitting.

[0013] In combination with the first aspect and the above implementation, in some possible implementations, obtaining multiple first image samples includes: obtaining multiple second image samples; for each second image sample, performing enhancement processing on the second image sample to obtain multiple third image samples corresponding to the second image sample; wherein the enhancement processing includes at least one of rotation, flipping and scaling; and using the multiple second image samples and the multiple third image samples corresponding to each of the multiple second image samples as multiple first image samples. Based on the above technical solution, by performing data enhancement processing on image samples, the diversity of the data set can be increased, the training samples of the model can be expanded, and the risk of model overfitting can be reduced.

[0014] In a second aspect, an image display device is provided, the image display device comprising:

[0015] An acquisition module is used to acquire an initial image corresponding to the vehicle; wherein the initial image is an image generated by three-dimensional modeling of the environment scene outside the vehicle;

[0016] A processing module is used to use a pre-trained target recognition model to recognize an initial image to obtain a first image; wherein the first image includes a vehicle, a target object in an environment outside the vehicle, and a bounding box of the target object; according to the bounding box of each target object, it is determined whether there is an abnormal object in the first image; wherein the abnormal object includes a target object with an overlapping bounding box in the first image; if so, the relative direction relationship between the target object in the abnormal object and the vehicle is kept unchanged, and the distance between the target objects in the abnormal object is adjusted so that the bounding boxes of the target objects in the abnormal object do not overlap, thereby obtaining a second image; and the second image is displayed using a display device of the vehicle.

[0017] In a third aspect, a vehicle is provided, the vehicle comprising: a memory for storing executable program code;

[0018] A processor is used to call and run executable program code from a memory so that the vehicle executes the image display method in the above-mentioned first aspect or any possible implementation of the first aspect.

[0019] In a fourth aspect, a computer program product is provided, comprising: a computer program code, which, when executed on a computer, enables the computer to execute the image display method in the first aspect or any possible implementation of the first aspect.

[0020] In a fifth aspect, a computer-readable storage medium is provided, which stores a computer program code. When the computer program code runs on a computer, the computer executes the image display method in the above-mentioned first aspect or any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A schematic diagram of a first scenario of the image display method provided by an embodiment of the present application is shown;

[0022] Figure 2 A schematic flow chart of an image display method provided in an embodiment of the present application is shown;

[0023] Figure 3 A second scenario schematic diagram of the image display method provided in an embodiment of the present application is shown;

[0024] Figure 4 A schematic diagram of the architecture of a target recognition model provided in an embodiment of the present application is shown;

[0025] Figure 5 A schematic diagram showing a plurality of first image samples provided by an embodiment of the present application is shown;

[0026] Figure 6 A schematic structural diagram of an image display device provided in an embodiment of the present application is shown;

[0027] Figure 7 A schematic structural diagram of a vehicle provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0028] The technical solution in the present application will be described clearly and in detail below in conjunction with the accompanying drawings. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B: "and / or" in the text is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0029] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features.

[0030] At present, with the development and popularization of autonomous driving technology, more and more vehicles are equipped with instrument restoration world modules. The instrument restoration world module can process the surrounding environment information recognized by the ADAS system and perform three-dimensional modeling, and then display it on the dashboard in the form of an image to help drivers better understand the situation around the vehicle.

[0031] However, due to the limited recognition capabilities of vehicle hardware, environmental interference and imperfect data processing algorithms, data errors may occur, resulting in overlapping targets in the image displayed on the dashboard, which in turn provides the driver with erroneous information and affects driving decisions and safety.

[0032] For example, Figure 1 In the illustrated vehicle exterior environment scene, when vehicle A is driving straight through an intersection, there is a vehicle B on the right side of vehicle A, and a pedestrian C is in front of vehicle B. Pedestrian C is blocked by vehicle B, and the driver of vehicle A cannot see pedestrian C. If the model of vehicle B overlaps with the model of pedestrian C in the interface 101 displayed on the dashboard of vehicle A, the driver of vehicle A only sees the model of vehicle B but not the model of pedestrian C; at this time, if vehicle A continues to drive, it may hit pedestrian C.

[0033] Based on the problems existing in the above-mentioned related technologies, the embodiments of the present application provide an image display method, device, vehicle and computer-readable storage medium. The present application obtains an initial image generated by three-dimensional modeling of the external environment scene; uses a pre-trained target recognition model to recognize the initial image, and when it is detected that there is overlap of target objects in the initial image, keeps the relative direction relationship between the overlapping target objects and the vehicle unchanged, adjusts the distance between the overlapping target objects and the vehicle, and obtains a corrected image, so that there is no overlap of target objects in the corrected image, and the direction relationship between the target object and the vehicle is accurate. The corrected image is displayed on the dashboard of the vehicle, so that the driver can clearly obtain the position relationship between the external environment objects and the vehicle in reality through the image displayed on the dashboard, which is convenient for the driver to make driving decisions and improve driving safety and driver experience.

[0034] The following is an embodiment of an image display method provided in an embodiment of the present application.

[0035] Figure 2 A schematic flow chart of an image display method provided in an embodiment of the present application is shown. Figure 2 As shown, an embodiment of the present application provides an image display method, which is applied to a vehicle, the vehicle is equipped with a display device, the display device integrates the information display of the ADAS system, and can provide the driver with an image of the position relationship between the target object and the vehicle in the external environment, so that the driver can obtain accurate environmental information. The image display method includes the following schemes:

[0036] S210: Acquire an initial image corresponding to the vehicle.

[0037] S220: Using a pre-trained target recognition model to recognize the initial image to obtain a first image.

[0038] S230: Determine whether there is an abnormal object in the first image according to the respective bounding boxes of each target object; if so, execute S240; if not, execute S250.

[0039] S240: Keeping the relative direction relationship between the target object in the abnormal object and the vehicle unchanged, adjusting the distance between the target objects in the abnormal object so that the boundary boxes of the target objects in the abnormal object do not overlap, and obtaining a second image.

[0040] S250: using the first image as the second image;

[0041] S260: Displaying the second image using a display device of the vehicle.

[0042] In the embodiment of the present application, the initial image is an image generated by three-dimensional modeling of the vehicle exterior environment scene. Exemplarily, the image of the vehicle exterior environment scene can be acquired by a sensor module (such as a camera, a lidar, etc.) configured on the vehicle. Then, the acquired image of the vehicle exterior environment scene is subjected to three-dimensional modeling by a modeling software to obtain an initial image, in which the positional relationship between the target object in the vehicle exterior environment and the vehicle is restored. Among them, the target objects in the vehicle exterior environment include, for example, pedestrians, buildings, and other vehicles. For example, Figure 1 As shown, interface 101 is an initial image, and interface 101 restores the positional relationship between the target objects (vehicle B and pedestrian C) in the vehicle A's external environment and vehicle A. In interface 101, "vehicle A" is a model of vehicle A obtained by three-dimensional modeling of vehicle A, "vehicle B" is a model of vehicle B, and "pedestrian C" is a model of pedestrian C.

[0043] The target recognition model is pre-trained, and the initial image is input into the target recognition model, and the initial image is recognized by the target recognition model, and the target recognition model outputs a first image, wherein the first image includes the vehicle, multiple target objects in the environment outside the vehicle, and the bounding boxes of the multiple target objects. It should be understood that the first image includes the vehicle and the multiple target objects in the environment outside the vehicle refers to the model of the vehicle and the multiple target objects. The first image includes static and / or dynamic target objects. For example, Figure 1As shown, the interface 101 is recognized by using a pre-trained target recognition model to obtain a first image (interface 102). The interface 102 includes "vehicle B" and "pedestrian C", as well as the bounding boxes and categories of "vehicle B" and "pedestrian C". Among them, the bounding box 1021 is the bounding box of "vehicle B", and the vehicle is the category of "vehicle B". The bounding box 1022 is the bounding box of "pedestrian C", and the pedestrian is the category of "pedestrian C". The bounding box of the target object reflects the position of the target object in the first image. Therefore, it can be judged whether there is overlap of target objects in the first image according to whether the bounding boxes of the target objects overlap, that is, it can be judged whether there are abnormal objects in the first image according to the bounding boxes of multiple target objects; wherein the abnormal objects include target objects corresponding to the bounding boxes that overlap with each other in the first image. After the first image with abnormal objects is displayed through a display device, the first image seen by the driver has overlapping target objects, which makes it impossible for the driver to correctly judge the position of the target object, and there is a hidden danger to driving safety. Therefore, it is necessary to separate the overlapping target objects in the first image and correctly display the position of the target object.

[0044] Exemplarily, if it is determined that there are abnormal objects in the first image, it means that there are overlapping target objects in the first image, then S240 is executed. Figure 1 As shown, it is determined that the bounding box 1021 and the bounding box 1022 in the interface 102 overlap, that is, it is determined that there are abnormal objects in the interface 102, and the abnormal objects are "vehicle B" and "pedestrian C". "Vehicle B" and "pedestrian C" overlap each other, and then S240 is executed. The relative direction relationship between "vehicle B" and "pedestrian C" and "vehicle A" is kept unchanged, and the distance between "vehicle B" and "pedestrian C" and "vehicle A" is adjusted so that the bounding boxes of "vehicle B" and "pedestrian C" do not overlap, and the second image (interface 103) is obtained. In the interface 103, "vehicle B" and "pedestrian C" no longer overlap, and the driver of vehicle A can obtain more accurate environmental information through the corrected interface 103, and avoid pedestrian C, thereby enhancing driving safety.

[0045] If it is determined that there is no abnormal object in the first image, indicating that there are no overlapping target objects in the first image, S250 is executed to use the first image as the second image. After the second image is obtained, the second image is displayed using a display device of the vehicle. The display device of the vehicle is a vehicle dashboard, a central control screen, etc.

[0046] Through the above technical solution, an initial image generated by three-dimensional modeling of the vehicle's external environment scene is obtained, and the initial image is input into a pre-trained target recognition model to obtain a first image including a vehicle, a target object in the vehicle's external environment, and a bounding box of the target object. According to the bounding box of each target object, it is determined whether there is an abnormal object in the first image; wherein the abnormal object includes a target object whose bounding boxes overlap each other. If an abnormal object is detected in the first image; the relative direction relationship between the target object in the abnormal object and the vehicle is kept unchanged, and the distance between the target objects in the abnormal object is adjusted so that the bounding boxes of the target objects in the abnormal object do not overlap, and a second image is obtained; and the second image is displayed through the vehicle's display device. The present application obtains an initial image generated by three-dimensional modeling of the vehicle's external environment scene; the initial image is recognized by a pre-trained target recognition model, and when it is detected that there is overlap of target objects in the initial image, the relative direction relationship between the overlapping target objects and the vehicle is kept unchanged, and the distance between the overlapping target objects and the vehicle is adjusted to obtain a corrected image, so that there is no overlap of target objects in the corrected image, and the direction relationship between the target objects and the vehicle is accurate. The corrected image is displayed on the vehicle's dashboard so that the driver can clearly obtain the positional relationship between the actual external environment and the vehicle through the image displayed on the dashboard, which facilitates the driver to make driving decisions and improves driving safety and driver experience.

[0047] In a possible implementation, the adjusting the distance between the target objects in the abnormal objects so that the bounding boxes of the target objects in the abnormal objects do not overlap to obtain the second image includes:

[0048] Obtain the distance between each target object in the abnormal object and the vehicle;

[0049] Determine the first object and the second object according to the distance between each target object and the vehicle;

[0050] The position of the second object is kept unchanged, and for each first object, the relative direction relationship between the first object and the vehicle is kept unchanged, and the first object is moved to increase the distance between the first object and the second object so that the bounding boxes of the target objects in the abnormal objects do not overlap, thereby obtaining a second image.

[0051] For example, a coordinate system can be established with the midpoint of the front bumper of the vehicle as the origin, and the distance between the target object and the vehicle can be calculated by the coordinates of the target object in the coordinate system. For example, the coordinates of the target object M in the above coordinate system are (3, 4), and according to the calculation, it can be known that the distance between the target object M and the vehicle is 5.

[0052] In a possible implementation, determining the first object and the second object according to the distance between each target object and the vehicle includes:

[0053] Determine the target object corresponding to the minimum value of the distance between each target object and the vehicle as the second object;

[0054] The target object other than the second object among the abnormal objects is determined as the first object.

[0055] It should be understood that the target object with a smaller distance from the vehicle among the abnormal objects is closer to the vehicle, that is, the data collected by the vehicle's sensor is more accurate. Therefore, the reliability of the position information of the target object with a smaller distance from the vehicle among the abnormal objects is higher, and the reliability of the position information of the target object with a larger distance from the vehicle is lower. Therefore, the target object with higher reliability of position information is used as the second object, the position of the second object is kept unchanged, and the first object is moved to increase the distance between the first object and the second object, which can not only keep the position of the target object with higher reliability of position information unchanged, but also make the bounding boxes of the overlapping target objects no longer overlap.

[0056] Based on the above technical solution, the target object corresponding to the minimum distance between multiple target objects and the vehicle is determined as the second object, which can keep the position of the target object closer to the vehicle unchanged, thereby improving the reliability of the position information of the target object in the displayed image.

[0057] In a possible implementation manner, the moving the first object to increase the distance between the first object and the second object includes:

[0058] The first object is moved in a direction away from the vehicle to increase a distance between the first object and the second object.

[0059] It should be understood that in the process of moving the first object to increase the distance between the first object and the second object, the directional relationship between the first object and the vehicle needs to be kept unchanged. In addition, since the first object is a target object with low position reliability, when moving the first object, the first object is moved away from the vehicle to increase the distance between the first object and the second object, so that the driver can preferentially obtain the position information of the second object with higher position reliability. The directional relationship of the first object relative to the vehicle includes the heading angle of the first object relative to the vehicle. Figure 3As shown, for example, the target object D and the target object F in the first image overlap, and the target object F is the first object, and the target object D is the second object. When adjusting the position information of the target object F, the position of the target object D is kept unchanged, and the direction relationship of the target object F relative to the vehicle is kept unchanged, that is, when the target object F is moved, the heading angle of the target object F is determined according to the direction relationship of the target object F relative to the vehicle, and the heading angle is kept unchanged during the movement of the target object F until the target object D and the target object F do not overlap, so as to obtain the second image. Assuming that the direction relationship of the target object F relative to the vehicle is that the target object F is located in the southeast direction of the vehicle, it is determined that the heading angle of the target object F relative to the vehicle is 45° north by west, then the heading angle is kept unchanged at 45° north by west, and the position of the target object F is moved to increase the distance between the target object D and the target object F, until the target object D and the target object F do not overlap, so as to obtain the second image.

[0060] Based on the above technical solution, by moving the first object in a direction away from the vehicle, the driver can preferentially obtain the position information of the target object that is closer to the vehicle, which helps the driver make driving decisions.

[0061] In an embodiment of the present application, the first image also includes the confidence of each target object, and the confidence of the target object is used to indicate the degree of certainty of the target recognition model for each target object detected in the first image. It should be understood that the higher the confidence of the target object, the higher the probability that the target object actually exists, and the lower the confidence of the target object, the lower the probability that the target object actually exists. Therefore, before determining whether there is an abnormal object in the first image according to the respective bounding boxes of each target object, the present application sets a preset threshold to filter out the target objects corresponding to the confidence of the target objects in the abnormal objects whose confidence is less than the preset threshold, so as to filter out the target objects whose probability of actually existing in the first image is too low, and only retain the target objects corresponding to the confidence of the target objects in the abnormal objects whose confidence is greater than or equal to the preset threshold, so as to improve the accuracy of abnormal object detection.

[0062] For example, assume that the target recognition model detects three target objects D, F, and G in the first image, and gives the confidences of the three target objects: 0.9 (D), 0.8 (F), and 0.7 (G). Assuming that the preset threshold is 0.75, the target objects corresponding to the confidences of the three target objects that are less than the preset threshold are screened out, that is, the target object G corresponding to the confidence of 0.7 (G) is screened out. The target objects corresponding to the confidences of 0.9 (D) and 0.8 (F) are retained to improve the accuracy of abnormal object detection.

[0063] In one possible implementation, Figure 4As shown, the target recognition model includes a classifier and a feature extraction layer connected in sequence. There are multiple feature extraction layers, such as a high-level feature extraction layer, a middle-level feature extraction layer, and a low-level feature extraction layer.

[0064] It should be understood that the target recognition model in the embodiment of the present application is obtained by fine-tuning the pre-trained model, which is a model that is pre-trained through a large number of pictures of the vehicle's surrounding environment and can recognize and classify target objects around the vehicle. Since the pre-trained model is usually trained on a large-scale general data set, although the pre-trained model can extract general image features, its performance in specific fields may not be ideal. Therefore, by fine-tuning the pre-trained model, the pre-trained model can be better adapted to specific tasks and data sets.

[0065] Exemplarily, the embodiment of the present application uses a progressive fine-tuning method to fine-tune the pre-trained model. Freeze the feature extraction layer of the model and only train the classifier. Then, gradually unfreeze some feature extraction layers and make more detailed adjustments to the model. Specifically, model fine-tuning includes the following steps:

[0066] Freeze the model parameters of all feature extraction layers and train the classifier using the target image sample set.

[0067] Exemplarily, the model parameters of all feature extraction layers of the model are frozen; the feature extraction layer of the pre-trained model has been pre-trained on a large-scale image dataset and can extract general image features. Freezing the feature extraction layer can prevent overfitting to a specific vehicle recognition task during fine-tuning. While freezing the feature extraction layer, the target image sample set is used to train the classifier, the target image sample set is input into the pre-trained model, the frozen feature extraction layer processes the target image sample set, extracts the image features of the target image sample set, and transmits the extracted image features to the classifier of the pre-trained model, and the classifier is responsible for mapping the image features extracted by the feature extraction layer to specific target object categories around the vehicle. During the training process at this stage, only the classifier of the pre-trained model is trained, the model parameters of the feature extraction layer remain unchanged, and the feature extraction layer uses constant model parameters to extract image features of the target image sample set.

[0068] (2) When the classifier training is completed, the model parameters of each feature extraction layer are unfrozen in turn according to a preset connection order, and each unfrozen feature extraction layer is trained using the target image sample set until the training of the last feature extraction layer among all the feature extraction layers is completed; wherein the preset connection order is a connection order from bottom to top or a connection order from top to bottom.

[0069] Exemplarily, after the classifier training of the pre-trained model is completed, the feature extraction layers of the pre-trained model are gradually unfrozen according to the preset connection order, and each unfrozen feature extraction layer is trained using the target image sample set. Each layer is unfrozen for a certain training cycle, so that the pre-trained model can make more detailed adjustments to specific tasks while retaining the pre-training knowledge. The preset connection order can be from high to low or from low to high until the last feature extraction layer of all feature extraction layers is trained and the target recognition model is obtained. For example, Figure 4 As shown in the figure, in the order of connection from high to low, the high-level feature extraction layer of the pre-trained model is first unfrozen. The high-level feature extraction layer contains the most abstract features, which may involve the features of the entire object or scene. Such features usually need to be adjusted according to the new task. Therefore, the high-level feature extraction layer is usually unfrozen and trained first during the model fine-tuning process. After the high-level feature extraction layer is unfrozen and trained, the middle-level feature extraction layer of the pre-trained model is unfrozen. The middle-level feature extraction layer contains more general and more abstract features than the high-level feature extraction layer. It may involve partial structures of objects. Such features may be useful in some tasks, but may also need to be adjusted according to the new task. After the middle-level feature extraction layer is unfrozen and trained, the bottom-level feature extraction layer of the pre-trained model is unfrozen. The bottom-level feature extraction layer usually contains some convolutional layers and pooling layers, which are used to extract basic features in the image, such as edges, textures, etc. Such features are common in many tasks, so they are usually kept unchanged during the model fine-tuning process. During the training process at this stage, the model parameters of the feature extraction layer are constantly adjusted and optimized so that the pre-trained model can better adapt to specific vehicle environment recognition tasks and improve the accuracy of target object recognition.

[0070] It should be understood that during the entire fine-tuning process, the loss function value and accuracy of the pre-trained model can be monitored, and based on the monitoring results, the hyperparameters such as the learning rate and batch size of the pre-trained model can be continuously adjusted to obtain the best fine-tuning effect.

[0071] Based on the above technical solution, the method of gradually unfreezing the feature extraction layer is used to train the target recognition model, which can not only prevent overfitting and improve model performance, but also improve training efficiency and model stability. By flexibly selecting the unfreezing order, it can better adapt to different task requirements and improve the generalization ability of the model.

[0072] In a possible implementation, the image display method further includes:

[0073] Acquire a plurality of first image samples; classify the plurality of first image samples to obtain a plurality of image sample sets;

[0074] Multiple image sample sets are used as target image sample sets.

[0075] It should be understood that, since the types of target objects around the vehicle in the actual scene are diverse, the target image sample set needs to ensure that it covers target objects of various categories to improve the model's recognition ability of different categories. Therefore, the multiple first image samples in the embodiment of the present application include multiple preset types of target objects, and in the fine-tuning process, the embodiment of the present application adopts a small batch training method to divide the image samples into multiple image sample sets for training, and uses a class-balanced sampling method to ensure that each small batch sample set contains target objects of various categories, that is, each image sample set includes multiple preset types of target objects, so as to train the model's recognition ability of different categories, thereby improving the generalization ability of the model and reducing the risk of overfitting. Exemplarily, multiple first image samples are obtained; for example, multiple pictures of the vehicle's surroundings are obtained to obtain multiple first image samples. Classify the multiple first image samples to obtain multiple image sample sets; for example, the types of target objects in the multiple pictures of the vehicle's surrounding environment include three categories: pedestrians, vehicles, and buildings. Classify the multiple first image samples to obtain multiple image sample sets, and use the multiple image sample sets as target image samples. The categories of target objects contained in each image sample set cover the three categories of pedestrians, vehicles, and buildings.

[0076] In one possible implementation, the above-mentioned acquisition of multiple first image samples includes: acquiring multiple second image samples; for each second image sample, performing enhancement processing on the second image sample to obtain multiple third image samples corresponding to the second image sample; wherein the enhancement processing includes at least one of rotation, flipping and scaling; and using the multiple second image samples and the multiple third image samples corresponding to each of the multiple second image samples as multiple first image samples.

[0077] In an embodiment of the present application, in order to increase the generalization ability of the pre-trained model, data enhancement processing is performed on the collected image samples during the fine-tuning process to expand the image samples used for model training. Exemplarily, multiple second image samples are obtained; for each second image sample, the second image sample is enhanced to obtain multiple third image samples corresponding to the second image sample. The second image sample can be a photo of the vehicle's surrounding environment collected by the vehicle. The data enhancement processing includes at least one of rotation, flipping and scaling. Exemplarily, such as Figure 5 As shown, the second image sample is image a, image a is rotated to obtain image b, image a is flipped to obtain image c, and image a is scaled to obtain image d. Image b, image c, and image d are multiple third image samples corresponding to image a.

[0078] Based on the above technical solution, by performing data enhancement processing on image samples, the diversity of the data set can be increased, the training samples of the model can be expanded, the workload of collecting image samples can be reduced; and the risk of model overfitting can be reduced.

[0079] It should be understood that the target recognition model in the embodiment of the present application has an adaptive learning mechanism, which can collect feedback from the driver and information from other vehicle systems during the application of the target recognition model; based on the driver's feedback or information from other vehicle systems, the model parameters of the target recognition model are further adjusted to improve the recognition accuracy of the target recognition model and avoid erroneous images from being displayed. Among them, the driver's feedback, for example, includes: when the driver determines that an error display occurs, the driver can take a screenshot of the error screen and system time with one click. Other vehicle system information, for example, includes: steering angle, distance, trajectory and other information recorded by the vehicle system during the running test.

[0080] Based on the above technical solution, the present application obtains an initial image generated by three-dimensional modeling of the external environment scene; uses a pre-trained target recognition model to recognize the initial image, and when it is detected that there is overlap of target objects in the initial image, keeps the relative direction relationship between the overlapping target objects and the vehicle unchanged, adjusts the distance between the overlapping target objects and the vehicle, and obtains a corrected image, so that there is no overlap of target objects in the corrected image, and the direction relationship between the target objects and the vehicle is accurate. The corrected image is displayed on the dashboard of the vehicle, so that the driver can clearly obtain the position relationship between the external environment objects and the vehicle in reality through the image displayed on the dashboard, which is convenient for the driver to make driving decisions and improve driving safety and driver experience.

[0081] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the image display device of the present application, please refer to the embodiment of the image display method of the present application.

[0082] Figure 6 The structure diagram of an image display device provided in an embodiment of the present application is shown. The image display device 600 includes:

[0083] The acquisition module 610 is used to acquire an initial image corresponding to the vehicle; wherein the initial image is an image generated by three-dimensional modeling of the environment scene outside the vehicle;

[0084] The processing module 620 is used to use a pre-trained target recognition model to recognize the initial image to obtain a first image; wherein the first image includes a vehicle, a target object in an environment outside the vehicle, and a bounding box of the target object; according to the bounding box of each target object, it is determined whether there is an abnormal object in the first image; wherein the abnormal object includes a target object with an overlapping bounding box in the first image; if so, the relative direction relationship between the target object in the abnormal object and the vehicle is kept unchanged, and the distance between the target objects in the abnormal object is adjusted so that the bounding boxes of the target objects in the abnormal object do not overlap, to obtain a second image; and the second image is displayed using a display device of the vehicle.

[0085] In a possible implementation, the processing module 620 includes:

[0086] An acquisition unit, used for acquiring the distance between each target object in the abnormal object and the vehicle;

[0087] A calculation unit, configured to determine the first object and the second object according to the distance between each target object and the vehicle;

[0088] The moving unit is used to keep the position of the second object unchanged, and for each first object, keep the relative direction relationship between the first object and the vehicle unchanged, move the first object to increase the distance between the first object and the second object, so that the boundary boxes of the target objects in the abnormal objects do not overlap, and obtain the second image.

[0089] In a possible implementation, the computing unit is specifically configured to:

[0090] Determine the target object corresponding to the minimum value of the distance between each target object and the vehicle as the second object;

[0091] The target object other than the second object among the abnormal objects is determined as the first object.

[0092] In a possible implementation, the mobile unit is specifically configured to:

[0093] The first object is moved in a direction away from the vehicle to increase a distance between the first object and the second object.

[0094] In a possible implementation, the target recognition model includes a classifier and a feature extraction layer connected in sequence, and there are multiple feature extraction layers. The image display device 600 further includes:

[0095] A training module is used to freeze the model parameters of all feature extraction layers and use the target image sample set to train the classifier; and, when the classifier training is completed, unfreeze the model parameters of each feature extraction layer in turn according to a preset connection order, and use the target image sample set to train each unfrozen feature extraction layer until the training of the last feature extraction layer of all feature extraction layers is completed; wherein the preset connection order is a connection order from bottom to top or a connection order from top to bottom.

[0096] In a possible implementation, the acquisition module 610 is further configured to:

[0097] Acquire a plurality of first image samples; wherein the plurality of first image samples include a plurality of target objects of preset types;

[0098] Classifying a plurality of first image samples to obtain a plurality of image sample sets; wherein each image sample set includes a plurality of target objects of preset types;

[0099] Multiple image sample sets are used as target image sample sets.

[0100] In a possible implementation, the acquisition module 610 is specifically configured to:

[0101] acquiring a plurality of second image samples;

[0102] For each second image sample, performing enhancement processing on the second image sample to obtain a plurality of third image samples corresponding to the second image sample; wherein the enhancement processing includes at least one of rotation, flipping and scaling;

[0103] The plurality of second image samples and the plurality of third image samples corresponding to the plurality of second image samples are used as the plurality of first image samples.

[0104] Figure 7 It is a structural schematic diagram of a vehicle provided in an embodiment of the present application.

[0105] For example, Figure 7 As shown, the vehicle 700 includes: a memory 701 and a processor 702, wherein the memory 701 stores an executable program code 7011, and the processor 702 is used to call and execute the executable program code 7011 to perform an image display method.

[0106] In this embodiment, the functional modules of the vehicle can be divided according to the above method example. For example, each functional module can be corresponded, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0107] In the case of dividing each functional module according to each function, the vehicle may include: an acquisition module, a processing module, a control module, etc. It should be noted that all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module, which will not be repeated here.

[0108] The vehicle provided in this embodiment is used to execute the above-mentioned image display method, so the same effect as the above-mentioned implementation method can be achieved.

[0109] In the case of an integrated unit, the vehicle may include a processing module and a storage module. The processing module may be used to control and manage the actions of the vehicle. The storage module may be used to support the vehicle in executing relevant program codes and data.

[0110] The processing module may be a processor or a controller, which may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of the present application. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, etc. The storage module may be a memory.

[0111] This embodiment further provides a computer-readable storage medium, in which a computer program code is stored. When the computer program code is executed on a computer, the computer executes the above-mentioned related method steps to implement an image display method in the above-mentioned embodiment.

[0112] This embodiment further provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the above-mentioned related steps to implement an image display method in the above-mentioned embodiment.

[0113] In addition, the vehicle provided in the embodiments of the present application may specifically be a chip, component or module, and the vehicle may include a connected processor and memory; wherein the memory is used to store instructions, and when the vehicle is running, the processor may call and execute instructions so that the chip executes an image display method in the above-mentioned embodiments.

[0114] The vehicle, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding image display method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding image display method provided above, and will not be repeated here.

[0115] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0116] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0117] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image display method, characterized in that: The image display method comprises: Acquire an initial image corresponding to the vehicle; wherein the initial image is an image generated by three-dimensional modeling of the environment scene outside the vehicle; Using a pre-trained target recognition model to recognize the initial image, to obtain a first image; wherein the first image includes the vehicle, a target object in an environment outside the vehicle, and a bounding box of the target object; Determining whether there is an abnormal object in the first image according to the respective bounding boxes of each of the target objects; wherein the abnormal object includes a target object with an overlapping bounding box in the first image; If yes, keep the relative direction relationship between the target object in the abnormal object and the vehicle unchanged, adjust the distance between the target objects in the abnormal object so that the bounding boxes of the target objects in the abnormal object do not overlap, and obtain a second image; The second image is displayed using a display device of the vehicle.

2. The image display method according to claim 1, wherein: The step of adjusting the distance between the target objects in the abnormal objects so that the bounding boxes of the target objects in the abnormal objects do not overlap to obtain the second image comprises: Obtaining the distance between each target object in the abnormal object and the vehicle; Determining a first object and a second object according to a distance between each target object and the vehicle; The position of the second object is kept unchanged, and for each of the first objects, the relative direction relationship between the first object and the vehicle is kept unchanged, and the first object is moved to increase the distance between the first object and the second object so that the bounding boxes of the target objects in the abnormal objects do not overlap, thereby obtaining a second image.

3. The image display method according to claim 2, wherein: Determining the first object and the second object according to the distance between each target object and the vehicle comprises: Determine the target object corresponding to the minimum value of the distance between each target object and the vehicle as the second object; The target object other than the second object in the abnormal object is determined as the first object.

4. The image display method according to claim 2, wherein: The moving the first object to increase the distance between the first object and the second object comprises: The first object is moved in a direction away from the vehicle to increase a distance between the first object and the second object.

5. The image display method according to claim 1, wherein: The target recognition model includes a classifier and a feature extraction layer connected in sequence, and the feature extraction layer has multiple, The image display method further includes: Freezing the model parameters of all the feature extraction layers and training the classifier using a target image sample set; When the classifier training is completed, the model parameters of each feature extraction layer are unfrozen in turn according to the preset connection order, and the target image sample set is used to train each unfrozen feature extraction layer until the training of the last feature extraction layer among all the feature extraction layers is completed; wherein the preset connection order is a connection order from bottom to top or a connection order from top to bottom.

6. The image display method according to claim 5, characterized in that: The image display method further includes: Acquire a plurality of first image samples; wherein the plurality of first image samples include a plurality of target objects of preset types; Classifying the plurality of first image samples to obtain a plurality of image sample sets; wherein each image sample set includes the plurality of preset types of target objects; The multiple image sample sets are used as the target image sample sets.

7. The image display method according to claim 6, wherein: The acquiring of a plurality of first image samples comprises: acquiring a plurality of second image samples; For each second image sample, performing enhancement processing on the second image sample to obtain a plurality of third image samples corresponding to the second image sample; wherein the enhancement processing includes at least one of rotation, flipping and scaling; The plurality of second image samples and the plurality of third image samples respectively corresponding to the plurality of second image samples are used as the plurality of first image samples.

8. An image display device, characterized in that: The image display device comprises: An acquisition module is used to acquire an initial image corresponding to the vehicle; wherein the initial image is an image generated by three-dimensional modeling of the environment scene outside the vehicle; A processing module is used to use a pre-trained target recognition model to recognize the initial image to obtain a first image; wherein the first image includes the vehicle, target objects in the vehicle environment and bounding boxes of the target objects; according to the bounding boxes of each target object, it is determined whether there are abnormal objects in the first image; wherein the abnormal objects include target objects with overlapping bounding boxes in the first image; if so, the relative direction relationship between the target objects in the abnormal objects and the vehicle is kept unchanged, and the distance between the target objects in the abnormal objects is adjusted so that the bounding boxes of the target objects in the abnormal objects do not overlap, to obtain a second image; and the second image is displayed using a display device of the vehicle.

9. A vehicle, characterized in that: The vehicle comprises: A memory for storing executable program codes; A processor, configured to call and run the executable program code from the memory, so that the vehicle executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed, the method according to any one of claims 1 to 7 is implemented.

Images