Image processing method, computer storage medium and electronic equipment

Abstract

The invention discloses an image processing method, a computer storage medium and electronic equipment. The image processing method comprises the following steps: acquiring a road simulation image and a real road image under multiple scenes of automatic driving; a first lane line detection model is trained according to the road simulation image to obtain a trained first lane line detection model, and the trained first lane line detection model is used for generating a real scene false label; and based on a transfer learning principle, training a second lane line detection model according to the real road image and the real scene pseudo label to obtain a trained second lane line detection model, the second lane line detection model being used for detecting a lane line when the vehicle is running. By adopting the method, lane line detection of real road images in various scenes can be met, the transfer learning effect of a lane line detection model is improved, and the training difficulty of the lane line detection model in a complex scene is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of electronic equipment, and in particular to an image processing method, a computer storage medium and an electronic equipment. Background Art

[0002] Existing transfer learning technology uses a model pre-training method to fuse the features of two road images with similar scenes from the feature level to generate transfer target data and use the data to train the lane line detection model. For example, the features of lane line images that are blocked by shadows and those that are not blocked are fused to train the lane line detection model. However, existing transfer learning is only applicable to road images with similar scenes, that is, it is impossible to transfer the features of other scenes or road condition images, and it is impossible to generate lane line detection models for multiple scenes, and thus it is impossible to meet the lane line detection requirements of road images in multiple scenes. In addition, the above method solves the transfer learning problem from the data level, which makes it difficult to control the quality of the transfer target data, reduces the effect of transfer learning, and makes it more difficult to train lane line detection models for complex scenes. Summary of the invention

[0003] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, one purpose of the present invention is to propose an image processing method, which can meet the lane line detection of real road images in various scenarios, improve the transfer learning effect of the lane line detection model, and reduce the training difficulty of the lane line detection model in complex scenarios.

[0004] A second objective of the present invention is to provide a computer storage medium.

[0005] A third objective of the present invention is to provide an electronic device.

[0006] In order to solve the above problems, an embodiment of the first aspect of the present invention provides an image processing method, including: acquiring road simulation images and real road images in multiple scenarios of autonomous driving; training a first lane line detection model according to the road simulation images to obtain a trained first lane line detection model, and the trained first lane line detection model is used to generate a real scene pseudo label; based on the principle of transfer learning, training a second lane line detection model according to the real road image and the real scene pseudo label to obtain a trained second lane line detection model, and the second lane line detection model is used to detect lane lines when the vehicle is driving.

[0007] According to the image processing method of the embodiment of the present invention, a first lane line detection model is trained based on road simulation images in multiple scenarios, so that the first lane line detection model can identify lane lines in road simulation images in multiple scenarios, and then the real scene pseudo labels and real road images of the trained first lane line detection model are used for training the second lane line detection model by transfer learning, so that the second lane line detection model satisfies the lane line detection of real road images in multiple scenarios and can improve the lane line detection effect. Therefore, compared with the method of training the lane line detection model by migrating target data, the second lane line detection model is trained by using real scene pseudo labels and real road images in the present application, which can meet the lane line detection of real road images in multiple scenarios, and the model is trained by using real scene pseudo labels, so that the transfer learning of the second lane line detection model is no longer limited by the quality of the migration target data, thereby improving the transfer learning effect of the second lane line detection model and reducing the training difficulty of the lane line detection model for complex scenarios.

[0008] In some embodiments, the first lane line detection model is trained according to the road simulation image to obtain a trained first lane line detection model, including: reconstructing the road simulation image data using the real road image to obtain a reconstructed road simulation image; training the first lane line detection model according to the reconstructed road simulation image to obtain the trained first lane line detection model.

[0009] In some embodiments, data of the road simulation image is reconstructed using the real road image to obtain a reconstructed road simulation image, including: constructing a generative adversarial network; and data of the road simulation image is reconstructed using the real road image through the generative adversarial network to obtain a reconstructed road simulation image.

[0010] In some embodiments, the generative adversarial network includes a generator and a discriminator, and uses the real road image to reconstruct the road simulation image through the generative adversarial network to obtain a reconstructed road simulation image, including: inputting the road simulation image into the generator to extract road simulation feature information; inputting the road simulation feature information and the real road image into the discriminator to obtain simulation similarity; using the simulation similarity to supervise the training of the generator until the generator converges; and using the converged generator to obtain the reconstructed road simulation image.

[0011] In some embodiments, the first lane line detection model is trained according to the reconstructed road simulation image to obtain the trained first lane line detection model, including: in a first training stage, the first lane line detection model is pre-trained using the reconstructed road simulation image; in a second training stage, the first lane line detection model is post-trained using the reconstructed road simulation image and the real road image to obtain the trained first lane line detection model.

[0012] In some embodiments, the method further includes: during the training process, obtaining a first training loss value of the first lane line detection model; and using the first training loss value to supervise the training of the first lane line detection model until the first lane line detection model converges.

[0013] In some embodiments, when training the second lane line detection model, the method also includes: obtaining a second training loss value of the second lane line detection model; constructing a discriminator network structure; inputting the real scene pseudo label output by the first lane line detection model and the lane line prediction result output by the second lane line detection model into the discriminator network structure to obtain a third training loss value; and using the second training loss value and the third training loss value to supervise the training of the second lane line detection model until the second lane line detection model converges.

[0014] In some embodiments, the discriminator network structure includes a layer of Channels Max Pooling structure, three layers of Conv+Spatail Attention structure, three layers of Spatail Attention structure and a layer of fully connected layer structure.

[0015] A second aspect of the present invention provides a computer storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the image processing method according to any one of claims 1 to 8.

[0016] The third aspect of the present invention provides an electronic device, comprising: at least one processor; a memory communicatively connected to at least one of the processors; wherein the memory stores a computer program executable by at least one of the processors, and when at least one of the processors executes the computer program, the image processing method described in the above embodiment is implemented.

[0017] The electronic device according to the embodiment of the present invention can meet the lane line detection of real road images in various scenarios by executing the image processing method of the above embodiment, improve the transfer learning effect of the lane line detection model, and reduce the training difficulty of the lane line detection model in complex scenarios.

[0018] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0020] Figure 1 is a flowchart of an image processing method according to an embodiment of the present invention;

[0021] Figure 2 is a schematic diagram of transfer learning training of a lane detection model according to an embodiment of the present invention;

[0022] Figure 3 is a schematic diagram of a discriminator network structure according to an embodiment of the present invention;

[0023] Figure 4 is a flowchart of an image processing method according to another embodiment of the present invention;

[0024] Figure 5 is a structural block diagram of an electronic device according to an embodiment of the present invention.

[0025] Reference numerals:

[0026] Electronic device 10;

[0027] Processor 1; Memory 2. DETAILED DESCRIPTION

[0028] Embodiments of the present invention are described in detail below. The embodiments described with reference to the accompanying drawings are exemplary. Embodiments of the present invention are described in detail below.

[0029] In order to solve the above problems, an embodiment of the first aspect of the present invention provides an image processing method, which can meet the lane line detection of real road images in various scenarios, improve the transfer learning effect of the lane line detection model, and reduce the training difficulty of the lane line detection model in complex scenarios.

[0030] Reference below Figure 1 The image processing method according to the embodiment of the present invention is described as follows: Figure 1 As shown, the method includes: step S1-step S3.

[0031] Step S1, obtaining road simulation images and real road images in multiple scenarios of autonomous driving.

[0032] Among them, the autonomous driving scenario is the road scene on which the autonomous driving vehicle travels. The multiple scenarios of autonomous driving include at least urban roads, highways, national highway scenes, rural roads, tunnel scenes, intersection scenes, overpass scenes, rainy road scenes, daytime road scenes, sunny road scenes, night road scenes, etc., and there is no restriction on this.

[0033] Specifically, a 3D virtual simulation system for autonomous driving is built. The 3D virtual simulation system for autonomous driving batch generates road simulation images in multiple scenes of autonomous driving, wherein the road simulation images are virtual road images, and the road simulation images include various types of lane lines, traffic participants, climate conditions, time periods, etc. Various types of lane lines include yellow lines, double lines, dotted lines, etc. Various traffic participants include pedestrians, vehicles, obstacles, etc. Climate conditions include rainy days, snowy days, foggy days, etc. Time periods include daytime, night, etc. The 3D virtual simulation system for autonomous driving can also collect real road images in multiple scenes of autonomous driving. Real road images are images of actual road scenes. Real road images include various types of lane lines, traffic participants, climate conditions, time periods, etc. Real road images do not need to be labeled with reference labels and lane line labels. In addition, road simulation images and real road images in multiple scenes of autonomous driving can also be downloaded through the Internet.

[0034] Step S2: training the first lane line detection model according to the road simulation image to obtain a trained first lane line detection model, and the trained first lane line detection model is used to generate a real scene pseudo label.

[0035] Specifically, the first lane line detection model is trained by a road simulation image to obtain a trained first lane line detection model, wherein the road simulation image contains labels of various reference objects, for example, the road simulation image contains labels of lane lines, and the labels of lane lines are the type and position information of lane lines. Then, a real scene pseudo label is generated by the trained first lane line detection model, that is, the first lane line detection model is trained by using annotated road simulation images, and then the labels of reference objects in the real road image are predicted by using the trained first lane line detection model. The real scene pseudo label is the label of the reference object in the real road image predicted by the first lane line detection model. Therefore, the first lane line detection model in the present application generates a real scene pseudo label by using a road simulation image, which can effectively improve the quality of the real scene pseudo label.

[0036] Step S3, based on the transfer learning principle, the second lane line detection model is trained according to the real road image and the real scene pseudo label to obtain a trained second lane line detection model, and the second lane line detection model is used to detect the lane line when the vehicle is driving.

[0037] Specifically, in order to solve the above problems, the present application first uses an autonomous driving 3D virtual simulation system to batch generate road simulation images of multiple scenes to train the first lane line detection model. At this time, the first lane line detection model can identify the lane lines of multi-scene road simulation images and generate real scene pseudo labels. Then, the second lane line detection model can transfer the parameters or structure of the trained first lane line detection model through transfer learning. That is to say, the principle of transfer learning is that after the training of the first lane line detection model is completed, the real scene pseudo labels generated by the first lane line detection model are used in the training process of the second lane line detection model, so that the second lane line detection model can detect the lane lines of multi-scene real road images, which can also effectively improve the training efficiency of the second lane line detection model and reduce the resources and time required for model training. In addition, in order to further improve the lane line detection effect of the second lane line detection model, real road images under various scenarios are input into the second lane line detection model to output corresponding lane line detection results, and then the lane line detection results of the second lane line detection model and the real scene pseudo labels are used to optimize the training of the second lane line detection model. That is, the difference between the lane line detection results of the second lane line model and the first lane line detection model is used to adjust the parameters or weights of the second lane line detection model to obtain the trained second lane line detection model, thereby improving the lane line detection effect of the second lane line detection model. Therefore, compared with the method of training the lane line detection model by migrating target data, the present application uses real scene pseudo labels and real road images to train the second lane line detection model, which can meet the lane line detection of real road images in various scenarios, and uses real scene pseudo labels to train the model so that the transfer learning of the second lane line detection model is no longer limited by the quality of the migration target data, thereby improving the transfer learning effect of the second lane line detection model and reducing the training difficulty of the lane line detection model in complex scenes. Moreover, by training the second lane line detection model with real scene pseudo labels, the learning space of the second lane line detection model can be isolated from that of the first lane line detection model, so that the second lane line detection model is only trained using real scene data, so that it is not affected by the data characteristics of the road simulation image, and the purpose of migrating from the simulation scene to the real scene learning is achieved. In addition, the second lane line detection model no longer requires manually labeled real road images, thereby avoiding manpower and time.

[0038] According to the image processing method of the embodiment of the present invention, a first lane line detection model is trained based on road simulation images in multiple scenarios, so that the first lane line detection model can identify lane lines in road simulation images in multiple scenarios, and then the real scene pseudo labels and real road images of the trained first lane line detection model are used for training the second lane line detection model by transfer learning, so that the second lane line detection model satisfies the lane line detection of real road images in multiple scenarios and can improve the lane line detection effect. Therefore, compared with the method of training the lane line detection model by migrating target data, the second lane line detection model is trained by using real scene pseudo labels and real road images in the present application, which can meet the lane line detection of real road images in multiple scenarios, and the model is trained by using real scene pseudo labels, so that the transfer learning of the second lane line detection model is no longer limited by the quality of the migration target data, thereby improving the transfer learning effect of the second lane line detection model and reducing the training difficulty of the lane line detection model for complex scenarios.

[0039] In some embodiments, the first lane line detection model is trained according to the road simulation image to obtain the trained first lane line detection model, including: reconstructing the road simulation image data using the real road image to obtain the reconstructed road simulation image; training the first lane line detection model according to the reconstructed road simulation image to obtain the trained first lane line detection model.

[0040] The existing transfer learning technology solution is to use model pre-training to fuse and learn two differently distributed features from the feature level. However, this solution has a better effect on lane line detection in similar scenes and is suitable for migrating from daytime detection to nighttime detection scenes. However, it has a poor effect on migrating from road simulation images to real road images. This is because the spatial distribution gap between the two types of data features is large. When the two features are fused, the feature learning space becomes very large, resulting in poor migration effect.

[0041] In order to improve the transfer learning effect from simulation data to real scene data, on the one hand, the road simulation image is converted into a road simulation image with real scene features through data reconstruction, so that the road simulation image is close to the real road image, that is, the real road image is obtained, wherein the real road image can be an image consistent with the scene in the road simulation image, for example, the scene in the road simulation image contains a rainy scene, and the real road image used for reconstruction also contains a rainy scene, and then the real road image is used to reconstruct the road simulation image, that is, the feature space of the road simulation image is mapped to the feature space in the real road image, so as to obtain the reconstructed road simulation image, thereby reducing the difference in the distribution of scene data in the two images and improving the efficiency of transfer learning. Among them, data reconstruction is to use the real road image to convert the road simulation image into a road simulation image with the features of the real road image, while retaining objects such as pedestrians, vehicles, road surfaces or lane lines in the road simulation image, and not changing the position, quantity, and category of the objects, thereby using the real road image to reconstruct the road simulation image, and because the scene consistency of the road simulation image used for reconstruction and the real road image is high, so that the reconstructed road simulation image is more real, which can effectively improve the accuracy of model transfer learning detection.

[0042] In the embodiment, in the related art, when the lane line detection model is trained by the real road image, it is necessary to manually label the reference objects in the real road image, but manual labeling consumes a lot of manpower and time. In order to solve this problem, the reconstructed road simulation image is used to train the first lane line detection model in the present application. That is to say, since the road simulation image itself contains the label of the reference object, the lane line detection model trained by the road simulation image can only detect the lane line of the road simulation image. Therefore, the road simulation image needs to be reconstructed so that the road simulation image has the characteristics of the real road image, and then the reconstructed road simulation image is used to train the first lane line detection model, that is, the first lane line detection model is trained by the road simulation image similar to the real scene, so that the trained first lane line detection model can recognize the lane line in the real road image. Therefore, the reconstructed road simulation image is used to train the lane line detection model in the present application, so that there is no need to manually label the real road image, avoiding the consumption of manpower and time. In addition, since the authenticity of the road simulation image used for model training is high, the accuracy of model transfer learning detection can be effectively improved.

[0043] In some embodiments, data of a road simulation image is reconstructed using a real road image to obtain a reconstructed road simulation image, including: constructing a generative adversarial network; and data of a road simulation image is reconstructed using a real road image through a generative adversarial network to obtain a reconstructed road simulation image.

[0044] Specifically, data reconstruction can be achieved through a deep learning model, wherein the deep learning model includes a generative adversarial network (GAN) and a diffusion model, etc. Taking the reconstruction of a road simulation image by a generative adversarial network as an example, the specific contents are as follows: first, a generative adversarial network is constructed, wherein the generative adversarial network is used to generate new data similar to the real data distribution, and the real road image is used to reconstruct the road simulation image through the generative adversarial network to obtain a reconstructed road simulation image, that is, a road simulation image with a scene distribution consistent with the real road image is generated, thereby reducing the difference in the scene data distribution in the two images and improving the efficiency of transfer learning.

[0045] In some embodiments, a generative adversarial network includes a generator and a discriminator, and a real road image is used to reconstruct data of a road simulation image through a generative adversarial network to obtain a reconstructed road simulation image, including: inputting the road simulation image into the generator to extract road simulation feature information; inputting the road simulation feature information and the real road image into the discriminator to obtain simulation similarity; using the simulation similarity to supervise the training of the generator until the generator converges; and using the converged generator to obtain the reconstructed road simulation image.

[0046] Specifically, the generative adversarial network includes a generator and a discriminator. The generator is used to generate a road simulation image similar to the real road image, and the discriminator is used to judge the similarity between the real road image and the road simulation image. Figure 2 As shown, the road simulation image is input into the generator to obtain the extracted road simulation feature information; the road simulation feature information and the real road image are input into the discriminator to obtain the simulation similarity Loss1, wherein the simulation similarity Loss1 is the similarity between the road simulation image and the real road image, that is, the discriminator is used to judge the similarity between the road simulation image generated by the generator and the real road image, and then the simulation similarity Loss1 is used to supervise the training of the generator until the generator converges, and the smaller the simulation similarity Loss1, the more similar the road simulation image generated by the generator is to the real road image. Therefore, the training process of the generator is also the process of reducing the simulation similarity Loss1, until the simulation similarity Loss1 gradually tends to the most ideal simulation similarity. At this time, the road simulation image generated by the generator is closest to the real road image, and the training of the generator is completed. Therefore, the reconstructed road simulation image is obtained by using the converged generator, that is, the converged generator will reconstruct a road simulation image that is very close to the real scene, thereby reducing the difference in scene data distribution in the two images and improving the efficiency of transfer learning.

[0047] In an embodiment, the simulation similarity Loss1 can be expressed as:

[0048]

[0049] Among them, G is the generator, D1 is the first discriminator, E is the mean, x is the real road image, and z is the road simulation image. The training of the generator is to minimize the loss of the generator G and maximize the loss of the discriminator D1, thereby completing the generator training.

[0050] In some embodiments, the first lane line detection model is trained according to the reconstructed road simulation image to obtain a trained first lane line detection model, including: in a first training stage, the first lane line detection model is pre-trained using the reconstructed road simulation image; in a second training stage, the first lane line detection model is post-trained using the reconstructed road simulation image and the real road image to obtain a trained first lane line detection model.

[0051] Specifically, in order to improve the detection effect of the first lane line detection model on the lane lines in the real road image, the first lane line detection model is trained through two training stages. That is, in the first training stage, the first lane line detection model is pre-trained using the reconstructed road simulation image, that is, the first lane line detection model is trained using the road simulation image similar to the real scene, so that the trained first lane line detection model can recognize the lane lines in the real road image. In the second training stage, the first lane line detection model is post-trained using the reconstructed road simulation image and the real road image to obtain the trained first lane line detection model. That is, the first lane line detection model is trained by generating simulation data in the early stage of training, and then the real scene data is added for fusion training, so that it can learn from the second lane line detection model and improve the accuracy of scene detection in the real road image.

[0052] In some embodiments, during the training process, such as Figure 2As shown, the first training loss value Loss2 of the first lane line detection model is obtained, wherein the first training loss value Loss2 is the difference between the real scene pseudo label output by the first lane line detection model and the real scene correct label. The first training loss value is used to supervise the training of the first lane line detection model until the first lane line detection model converges. The smaller the first training loss value Loss2 is, the more accurate the real scene pseudo label output by the first lane line detection model is. Therefore, the training process of the first lane line detection model is also the process of reducing the first training loss value Loss2, until the first training loss value Loss2 gradually tends to the most ideal first training loss value, then the training of the first lane line detection model is completed. At this time, the real scene pseudo label generated by the first lane line detection model is very accurate. After the training of the first lane line detection model is completed, the first lane line detection model achieves the best detection effect on the lane line in the road simulation image, and at the same time, it can also have a good detection effect when detecting the lane line in the real road image.

[0053] In some embodiments, the calculation formula of the first training loss value Loss2 is as follows:

[0054]

[0055] Among them, w1 and w2 are weight hyperparameters, L cls is the lane line category cross entropy loss, L reg is the MSE loss for lane line point fitting, n is the number of lane line points, y cls,i The i-th point category label value, p i is the predicted probability of the i-th point category, x reg,i is the label value of the i-th point in the x direction, x i is the predicted value in the x direction, y reg,i is the label value of the y direction of the i-th point, is the predicted value in the y direction.

[0056] In some embodiments, Figure 2 As shown, when training the second lane line detection model, it is obtained by the following steps:

[0057] First, obtain the second training loss value Loss3 of the second lane line detection model, that is, the lane line detection loss. The second training loss value is the difference between the lane line prediction result output by the second lane line detection model and the actual lane line result. The size of the second training loss value can reflect the training degree and accuracy of the second lane line detection model.

[0058] Secondly, construct the discriminator network structure.

[0059] Then, the real scene pseudo label output by the first lane line detection model and the lane line prediction result output by the second lane line detection model are input into the discriminator network structure to obtain a third training loss value Loss4, wherein the third training loss value is the difference between the lane line prediction result output by the second lane line detection model and the real scene pseudo label.

[0060] Finally, the second training loss value and the third training loss value are used to supervise the training of the second lane line detection model until the second lane line detection model converges. In other words, the training effect of the second lane line detection model is evaluated by the second training loss value and the third training loss value.

[0061] Specifically, the real road image is input into the second lane line detection model to output the lane line prediction result of the real road image, and the second training loss value is used to evaluate the difference between the lane line prediction result output by the second lane line detection model and the actual lane line result, and the smaller the second training loss value is, the more accurate the lane line prediction result output by the second lane line detection model is. Therefore, the training process of the second lane line detection model is to continuously reduce the second training loss value until the second training loss value gradually tends to the most ideal second training loss value. At this time, the lane line prediction result output by the second lane line detection model is very accurate.

[0062] At the same time, without considering the background and obstacles, since the lane line feature space distribution of the road simulation image and the real road image is consistent, the third training loss value is used to supervise the training of the second lane line detection model, that is, the third training loss value is used to constrain the second lane line detection model, so that the lane line prediction result output by the second lane line detection model is similar to or the same as the lane line prediction result output by the first lane line detection model, that is, the real scene pseudo label. Specifically, since the first lane line detection model can detect the lane lines of the real road image, and the lane line detection of the real road image is accurate, and the first lane line detection model can generate the real scene pseudo label of the real road image, therefore, the second lane line detection model is trained by the real scene pseudo label, so that the second lane line detection model has the ability to detect the lane lines of the real road image. By training the second lane line detection model by the real scene pseudo label, the learning space of the second lane line detection model and the first lane line detection model can be isolated, so that the second lane line detection model is trained only with the real scene data, so that it is not affected by the data characteristics of the road simulation image, and the purpose of migration to the real scene learning is achieved.

[0063] At the same time, in order to improve the detection effect of the second lane line detection model on the lane lines in the real road image, the second lane line detection model is trained using the third training loss value, that is, the parameters or weights of the second lane line detection model are adjusted by the third training loss value until the third training loss value converges. Therefore, the training process of the second lane line detection model is also the process of optimizing (reducing) the third training loss value, until the third training loss value gradually tends to the most ideal third training loss value, that is, reducing the difference between the lane line prediction results output by the second lane line detection model and the first lane line detection model, until the difference between the lane line prediction results gradually tends to When the lane line prediction result output by the second lane line detection model is similar to or consistent with the lane line prediction result output by the first lane line detection model, that is, the real scene pseudo label, it means that the lane line detection effect of the second lane line detection model has reached the best state, that is, the second lane line detection model has achieved the detection effect of the first lane line detection model on the lane line in the real road image. Therefore, after multiple rounds of iterations, the second lane line detection model can learn the lane line features of the real scene from the first lane line detection model, complete the learning process of the second lane line detection model using the asymptotic migration of data distribution, and the second lane line detection model obtains better migration learning effect.

[0064] In addition, the training loss value can be understood as the ratio of the intersection and union between the predicted box and the true box.

[0065] In the embodiment, the calculation formula of the third training loss value Loss3 is as follows:

[0066]

[0067] Among them, M2 is the second lane line detection model, D2 is the second discriminator, E is the mean, x r is the real scene pseudo label output by the first lane line detection model, and M2(x) is the lane line prediction result output by the second lane line detection model.

[0068]

[0069] Among them, w3 and w4 are weight hyperparameters.

[0070] In an embodiment, Figure 2As shown in the figure, while the real scene pseudo labels generated by the first lane line detection model are used to train the second lane line detection model, the real scene pseudo labels generated by the second lane line detection model are also used for the fusion training of the first lane line detection model. That is to say, during the training process, the first lane line detection model and the second lane line detection model generate real scene pseudo labels for each other to learn. The improvement of one of the first lane line detection model and the second lane line detection model will promote the learning enhancement of the other model. Since the first lane line detection model has the correct real scene pseudo labels to supervise, the two models can avoid non-convergence problems during the mutual training process.

[0071] In an embodiment, the learning strategies of the first lane line detection model and the second lane line detection model are adjusted by the evaluation results of the second training loss value and the third training loss value. The learning strategy adopted for the first lane line detection model is to gradually increase the real scene data of the real scene image to improve the detection performance. Since the effect of the real scene pseudo-label generated when the second lane line detection model starts training is poor, it is necessary to reduce the real scene data used to train the first lane line detection model at the beginning of training. As the accuracy of detecting the second lane line detection model is improved, the proportion of the real scene data used to train the first lane line detection model can be increased. However, in order to ensure the learning ability of the first lane line detection model for road simulation data and labels, an upper limit value of the number of real scene data used to train the first lane line detection model is set. The upper limit value can be 50% to ensure that the first lane line detection model can accurately learn the simulation label data, thereby avoiding the model from falling into a local optimum. The learning strategy of the second lane line detection model is adjusted according to the evaluation result. Specifically, since the second lane line detection model has poor lane line detection effect in the early stage of training, adding the discriminator network structure in advance to enhance the learning effect may easily lead to model convergence difficulties. Therefore, the discriminator network structure is added in the late stage of training to converge the second lane line detection model, thereby ensuring the convergence of the second lane line detection model, enhancing the learning effect of the second lane line detection model, and improving the detection accuracy of the second lane line detection model.

[0072] Therefore, the method of asymptotic migration of data distribution and learning strategy constraints in this application enables the second lane line detection model to gradually complete the optimization training of the real scene.

[0073] In an embodiment, the learning strategies of the first lane line detection model and the second lane line detection model are adjusted by the evaluation results of the second training loss value and the third training loss value. By adjusting the learning strategies of the two lane line detection models, the training process of transfer learning is optimized to further improve the detection accuracy of the lane line detection model.

[0074] In the embodiment, the existing transfer learning technical solution is to use model pre-training to fuse and learn two differently distributed features from the feature level. However, this solution has a better effect on lane line detection in similar scenes and is suitable for migrating from daytime detection to nighttime detection scenes, but the detection effect is not good for migrating from simulation data to real scene data. This is because the spatial distribution difference of the features of the two data is large, and when the two features are fused, the feature learning space becomes very large, resulting in poor migration effect. In order to improve the transfer learning effect from simulation data to real scene data, the application isolates the influence of simulation data by learning from each other between two lane line detection models, that is, the second lane line detection model only needs to learn the lane line detection of the first lane line detection model for real road images, and does not need to learn the lane line detection of the first lane line detection model for road simulation images, thereby reducing the feature space distribution, which is conducive to the rapid training and convergence of the second lane line detection model.

[0075] In some embodiments, in order to improve the performance of the discriminator, the discriminator network structure designed by the present application is as follows: Figure 3 As shown in the figure, the discriminator network structure includes a Channels Max Pooling structure, a three-layer Conv+Spatail Attention structure, a three-layer Spatail Attention structure and a fully connected layer structure. Specifically, since the single Channel in the output result of the first lane line detection model or the second lane line detection model contains sparse information about the lane line, multiple channels are fused into a Channels MaxPooling structure, which is beneficial to subsequent feature learning. Since the lane line has a specific spatial distribution structure, the addition of a three-layer Spatail Attention structure can enhance the lane line distribution structure learning ability, and the fully connected layer is used to output the lane line prediction results. In addition, it should be noted that the first layer of the discriminator network structure is a Channels Max Pooling structure, the second to fourth layers are Conv+Spatail Attention structures, and the last layer is a fully connected layer.

[0076] In an embodiment, the image processing method of the present application is suitable for transfer learning of an autonomous driving visual perception model. The image processing method of the present application can also be used for transfer learning of models such as traffic sign detection, road obstacle detection, and drivable area segmentation. The transfer learning process is the same as the transfer learning process of the second lane line detection model, and is not described in detail here.

[0077] Exemplarily, the road simulation image contains labels of traffic signs and road obstacles, so the first detection model can be trained using the road simulation image to obtain a trained first detection model. The trained first detection model is used to generate pseudo labels. The pseudo labels generated by the first detection model are used in the training process of the second detection model, so that the second detection model can detect obstacles or traffic signs in real road images.

[0078] Reference below Figure 2 The image processing method of an embodiment of the present invention is illustrated by way of example, and the specific contents are as follows.

[0079] Step S4, input the road simulation image and the label of the road simulation image, and execute step S6.

[0080] Step S5, input the real road image, and execute steps S6 and S8.

[0081] Step S6, acquiring a real road image and a road simulation image, and reconstructing data of the road simulation image using the real road image to obtain a reconstructed road simulation image.

[0082] Step S7: training a first lane detection model according to the reconstructed road simulation image.

[0083] Step S8, obtaining a real road image, and inputting the real road image into a first lane line detection model to generate a real scene pseudo label.

[0084] Step S9: training a second lane line detection model using real scene pseudo labels.

[0085] Step S10: Evaluate the second lane line detection model using the second training loss value and the third training loss.

[0086] Step S11, determine whether the detection accuracy of the second lane line detection model reaches a preset condition, for example, determine whether the third training loss value reaches the most ideal third training loss value. If so, execute step S13, otherwise, execute step S12.

[0087] Step S12, adjust the learning strategies of the first lane line detection model and the second lane line detection model during the training process, and execute step S7 and step S9.

[0088] Step S13, training ends.

[0089] A second aspect of the present invention provides a computer storage medium having a computer program stored thereon, wherein the computer program implements the image processing method of the above embodiment when executed by a processor.

[0090] A third aspect of the present invention provides an electronic device 10, such as Figure 5 As shown, the electronic device 10 includes: at least one processor and a memory communicatively connected to the at least one processor.

[0091] The memory stores a computer program that can be executed by at least one processor, and the image processing method of the above embodiment is implemented when the at least one processor executes the computer program.

[0092] The electronic device according to the embodiment of the present invention can meet the lane line detection of real road images in various scenarios by executing the image processing method of the above embodiment, improve the transfer learning effect of the second lane line detection model, and reduce the training difficulty of the lane line detection model in complex scenarios.

[0093] In the description of this specification, any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code including one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present invention belong.

[0094] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0095] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0096] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0097] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0098] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present invention. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

[0099] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example.

[0100] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. An image processing method, characterized in that: include: Obtain road simulation images and real road images in multiple scenarios of autonomous driving; Training the first lane line detection model according to the road simulation image to obtain a trained first lane line detection model, wherein the trained first lane line detection model is used to generate a real scene pseudo label; Based on the transfer learning principle, the second lane line detection model is trained according to the real road image and the real scene pseudo label to obtain a trained second lane line detection model, and the second lane line detection model is used to detect lane lines when the vehicle is driving.

2. The image processing method according to claim 1, characterized in that: Training a first lane line detection model according to the road simulation image to obtain a trained first lane line detection model includes: Reconstructing the road simulation image using the real road image to obtain a reconstructed road simulation image; The first lane line detection model is trained according to the reconstructed road simulation image to obtain the trained first lane line detection model.

3. The image processing method according to claim 2, characterized in that: Reconstructing data of the road simulation image using the real road image to obtain a reconstructed road simulation image includes: Building a Generative Adversarial Network; The real road image is used to reconstruct data of the road simulation image through the generative adversarial network to obtain a reconstructed road simulation image.

4. The image processing method according to claim 3, characterized in that: The generative adversarial network includes a generator and a discriminator, and uses the real road image to reconstruct the road simulation image through the generative adversarial network to obtain a reconstructed road simulation image, including: Inputting the road simulation image into the generator to obtain extracted road simulation feature information; Inputting the road simulation feature information and the real road image into the discriminator to obtain simulation similarity; Supervising the training of the generator using the simulation similarity until the generator converges; The reconstructed road simulation image is obtained by using the converged generator.

5. The image processing method according to claim 2, characterized in that: Training a first lane line detection model according to the reconstructed road simulation image to obtain the trained first lane line detection model includes: In a first training stage, the reconstructed road simulation image is used to perform preliminary training on a first lane line detection model; In the second training stage, the first lane line detection model is post-trained using the reconstructed road simulation image and the real road image to obtain the trained first lane line detection model.

6. The image processing method according to claim 5, characterized in that: The method further comprises: During the training process, obtaining a first training loss value of the first lane line detection model; The first training loss value is used to supervise the training of the first lane line detection model until the first lane line detection model converges.

7. The image processing method according to any one of claims 1 to 6, characterized in that: When training the second lane line detection model, the method further includes: Obtain a second training loss value of the second lane line detection model; Construct the discriminator network structure; Inputting the real scene pseudo label output by the first lane line detection model and the lane line prediction result output by the second lane line detection model into the discriminator network structure to obtain a third training loss value; The second training loss value and the third training loss value are used to supervise the training of the second lane line detection model until the second lane line detection model converges.

8. The image processing method according to claim 7, characterized in that: The discriminator network structure includes a layer of Channels Max Pooling structure, three layers of Conv+SpatailAttention structure, three layers of SpatailAttention structure and a layer of fully connected layer structure.

9. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image processing method according to any one of claims 1 to 8 is implemented.

10. An electronic device, characterized in that: include: at least one processor; a memory communicatively coupled to at least one of the processors; The memory stores a computer program executable by at least one of the processors, and when at least one of the processors executes the computer program, the image processing method according to any one of claims 1 to 8 is implemented.

Images