Training method, identification method, system and device of lane early warning model and medium

Abstract

The invention discloses a training method, a recognition method, a system, a device and a medium for a lane early warning model, and the training method comprises the steps: obtaining training data which comprises a marked source domain image and an unmarked target domain image; determining image confrontation loss according to the source domain image, the target domain image, the two generators and the two discriminators; according to the source domain image and the target domain image, the two generators determine image loop consistency loss; determining cross-domain consistency loss according to the target domain image, the two image recognizers and the two generators; determining image recognition loss according to the source domain image, the two image recognizers and the first generator; total loss is determined according to the image confrontation loss, the image circulation consistency loss, the cross-domain consistency loss and the image recognition loss, and parameters of the vehicle early warning model are updated according to the total loss. According to the embodiment of the invention, the model can be trained by using a limited number of annotation data, the prediction accuracy is improved, and the method can be widely applied to the technical field of image processing.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a training method, recognition method, system, device and medium for a lane warning model. Background Art

[0002] In the field of automobile hazard warning, image recognition technology plays a vital role. It can help drivers identify potential dangers in time and take corresponding preventive measures. However, the widespread application and further development of this technology face the key problem of limited annotated data. The limited annotated data seriously restricts the learning ability and accuracy of image recognition models. In the automobile hazard warning system, the image recognition model needs to learn how to identify various traffic scenes and potential dangers through a large amount of annotated data. However, due to the high cost of obtaining annotated data and the cumbersome and time-consuming annotation process, the available annotated data set is relatively small, which is difficult to meet the needs of model training. This makes it impossible for the model to accurately identify potential dangers when facing complex and changing traffic scenes, thus affecting the performance of the warning system. Summary of the invention

[0003] In view of this, an object of an embodiment of the present invention is to provide a training method, recognition method, system, device and medium for a lane warning model, which can use a limited amount of labeled data to train the model and improve the accuracy of prediction.

[0004] On the one hand, an embodiment of the present invention provides a training method for a lane warning model, wherein the vehicle warning model includes an image conversion module and an image recognition module, wherein the image conversion module includes a first generator, a second generator, a first discriminator and a second discriminator, and the image recognition module includes a first image recognizer and a second image recognizer, and the training method includes:

[0005] Acquire training data, where the training data includes labeled source domain images and unlabeled target domain images;

[0006] Determining an image adversarial loss based on the source domain image, the target domain image, the first generator, the first discriminator, the second generator, and the second discriminator;

[0007] Determine an image cycle consistency loss according to the source domain image, the target domain image, the first generator, and the second generator;

[0008] determining a cross-domain consistency loss based on a target domain image, the first image recognizer, the second generator, and a second image recognizer;

[0009] Determining an image recognition loss based on a source domain image, the first image recognizer, the first generator, and the second image recognizer;

[0010] The total loss is determined according to the image adversarial loss, the image cycle consistency loss, the cross-domain consistency loss and the image recognition loss, and the parameters of the vehicle warning model are updated according to the total loss until the preset requirements are met, thereby determining the vehicle warning model.

[0011] Optionally, determining the image adversarial loss according to the source domain image, the target domain image, the first generator, the first discriminator, the second generator, and the second discriminator includes:

[0012] Inputting the source domain image into a first generator to obtain a target domain conversion image, and generating a first adversarial loss value according to the target domain conversion image, the target domain image, and the first discriminator;

[0013] Inputting the target domain image into a second generator to obtain a source domain conversion image, and generating a second adversarial loss value according to the source domain conversion image, the source domain image, and the second discriminator;

[0014] An image adversarial loss is determined according to the first adversarial loss value and the second adversarial loss value.

[0015] Optionally, determining the image cycle consistency loss according to the source domain image, the target domain image, the first generator, and the second generator includes:

[0016] Inputting the source domain image into a first generator to obtain a target domain conversion image, inputting the target domain conversion image into a second generator to obtain a source domain restoration image, and determining a first cycle consistency loss according to the source domain image and the source domain restoration image;

[0017] Inputting the target domain image into the second generator to obtain a source domain conversion image, inputting the source domain conversion image into the first generator to obtain a target domain restoration image, and determining a second cycle consistency loss according to the target domain image and the target domain restoration image;

[0018] An image cycle consistency loss is determined according to the first cycle consistency loss and the second cycle consistency loss.

[0019] Optionally, determining the cross-domain consistency loss according to the target domain image, the first image recognizer, the second generator, and the second image recognizer includes:

[0020] determining a first recognition result according to the target domain image and the first image recognizer;

[0021] Inputting the target domain image into a second generator to obtain a source domain conversion image, and determining a second recognition result according to the source domain conversion image and the second image recognizer;

[0022] A cross-domain consistency loss is determined according to the first recognition result and the second recognition result.

[0023] Optionally, determining the image recognition loss according to the source domain image, the first image recognizer, the first generator, and the second image recognizer includes:

[0024] determining a first recognition loss based on the source domain image and the first image recognizer;

[0025] Inputting the source domain image into a first generator to obtain a target domain conversion image, and determining a second recognition loss according to the target domain conversion image and the second image recognizer;

[0026] An image recognition loss is determined according to the first recognition loss and the second recognition loss.

[0027] On the other hand, an embodiment of the present invention provides a method for recognizing a lane warning image, the method comprising:

[0028] A target domain image is obtained, and the target domain image is input into the trained second image recognizer to obtain a lane warning image recognition result.

[0029] On the other hand, an embodiment of the present invention provides a training system for a lane warning model, including:

[0030] The first module is used to obtain training data, where the training data includes labeled source domain images and unlabeled target domain images;

[0031] A second module, configured to determine an image adversarial loss according to the source domain image, the target domain image, the first generator, the first discriminator, the second generator, and the second discriminator;

[0032] A third module is used to determine an image cycle consistency loss according to the source domain image, the target domain image, the first generator, and the second generator;

[0033] A fourth module, configured to determine a cross-domain consistency loss based on a target domain image, the first image recognizer, the second generator, and the second image recognizer;

[0034] A fifth module, configured to determine an image recognition loss according to a source domain image, the first image recognizer, the first generator, and the second image recognizer;

[0035] The sixth module is used to determine the total loss based on the image adversarial loss, the image cycle consistency loss, the cross-domain consistency loss and the image recognition loss, and update the parameters of the vehicle warning model according to the total loss until the preset requirements are met, thereby determining the vehicle warning model.

[0036] On the other hand, an embodiment of the present invention provides a lane warning image recognition device, including:

[0037] at least one processor;

[0038] at least one memory for storing at least one program;

[0039] When the at least one program is executed by the at least one processor, the at least one processor implements the above-mentioned training method or recognition method.

[0040] On the other hand, an embodiment of the present invention provides a computer-readable storage medium, in which a program executable by a processor is stored. When the program executable by the processor is executed by the processor, it is used to perform the above-mentioned training method or recognition method.

[0041] On the other hand, an embodiment of the present invention provides a vehicle, which includes the above-mentioned lane warning model training system or the above-mentioned lane warning image recognition device.

[0042] The implementation of the embodiment of the present invention includes the following beneficial effects: the vehicle warning model includes an image conversion module and an image recognition module, the image conversion module includes a first generator, a second generator, a first discriminator and a second discriminator, the image recognition module includes a first image recognizer and a second image recognizer, and the training method includes: obtaining training data, the training data includes annotated source domain images and unlabeled target domain images; determining image adversarial loss according to the source domain image, the target domain image, the first generator, the first discriminator, the second generator and the second discriminator; determining image cycle consistency loss according to the source domain image, the target domain image, the first generator and the second generator; determining image cycle consistency loss according to the target domain image, the first image recognizer, the second generator and the second discriminator; The image recognizer determines the cross-domain consistency loss; the image recognition loss is determined according to the source domain image, the first image recognizer, the first generator, and the second image recognizer; the total loss is determined according to the image adversarial loss, the image cycle consistency loss, the cross-domain consistency loss and the image recognition loss, and the parameters of the vehicle warning model are updated according to the total loss until the preset requirements are met, and the vehicle warning model is determined; the image conversion module and the image recognition module are merged into an end-to-end adversarial learning model, and the mutual constraints of accuracy are achieved through multiple loss functions, and the image recognizer is trained with labeled source domain images through the method of transfer learning domain adaptation, so that the image recognizer can achieve the purpose of predicting unlabeled target domain images, thereby realizing unsupervised training. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a schematic diagram of the structure of a cycle-consistent generative adversarial network provided by an embodiment of the present invention;

[0044] Figure 2 is a flowchart of the steps of a lane warning model training method provided by an embodiment of the present invention;

[0045] Figure 3 is a flowchart of the steps of another lane warning model training method provided by an embodiment of the present invention;

[0046] Figure 4 is a flowchart of steps for determining image adversarial loss provided by an embodiment of the present invention;

[0047] Figure 5 is a schematic flow chart of steps for determining image cyclic consistency loss provided by an embodiment of the present invention;

[0048] Figure 6 is a flowchart of steps for determining cross-domain consistency loss provided by an embodiment of the present invention;

[0049] Figure 7 is a flowchart of steps for determining image recognition loss provided by an embodiment of the present invention;

[0050] Figure 8 is a schematic diagram of the structure of a generator provided by an embodiment of the present invention;

[0051] Fig. 9 is a schematic structural diagram of a residual module provided by an embodiment of the present invention;

[0052] Fig.10 is a structural schematic diagram of an image recognizer provided by an embodiment of the present invention;

[0053] Fig.11 It is a structural block diagram of a lane warning model training system provided by an embodiment of the present invention;

[0054] Fig.12 It is a structural block diagram of a lane warning image recognition device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only provided for the convenience of explanation and description, and the order between the steps is not limited in any way. The execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0056] It should be noted that, although the functional module division is performed in the device schematic diagram and the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in a different order from the module division in the device or the order in the flow chart. The terms "first", "second", etc. in the specification and claims and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0057] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0058] See also Figure 1 CycleGAN is a model based on the generative adversarial network. The network contains two generators G X→Y , G Y→X , two discriminators D T and D S . CycleGAN simultaneously learns G X→Y and G Y→X Two generators are required to convert source domain data X to target domain data Y and then convert it back to the style of the source domain. The core part is to propose a cycle consistency loss, which is used to measure the difference between the reconstructed image and the input image. While completing the image conversion between the two domains, it ensures that the semantics of the generated image does not change. CycleGAN can be trained without paired data and convert the source domain dataset X to the target domain dataset Y.

[0059] See also Figure 2 The embodiment of the present invention provides a training method for a lane warning model, wherein the vehicle warning model includes an image conversion module and an image recognition module, wherein the image conversion module includes a first generator, a second generator, a first discriminator and a second discriminator, and the image recognition module includes a first image recognizer and a second image recognizer, and the training method includes:

[0060] S100: Acquire training data, where the training data includes labeled source domain images and unlabeled target domain images.

[0061] The labeled source domain images include source domain images and their corresponding recognition results, and the unlabeled target domain images refer to target domain images to be recognized. The labeled source domain images and the unlabeled target domain images are determined according to actual applications, and this embodiment does not impose any specific restrictions.

[0062] S200, determining an image adversarial loss according to a source domain image, a target domain image, a first generator, a first discriminator, a second generator, and a second discriminator.

[0063] The generator is used to convert an image from one domain to another domain. The first generator is used to convert a source domain image into a target domain image, and the second generator is used to convert a target domain image into a source domain image. The discriminator is used to determine the degree of similarity between images. The first discriminator is used to determine the degree of similarity between a target domain image and an image converted from a source domain image, and the second discriminator is used to determine the degree of similarity between a source domain image and an image converted from a target domain image. Image adversarial loss determines the degree of similarity between an image after domain conversion and an image in another domain.

[0064] S300, determining an image cycle consistency loss according to a source domain image, a target domain image, a first generator, and a second generator.

[0065] Image cycle consistency loss refers to the similarity between an image converted from a first domain to another domain and then from another domain back to the first domain and the original image. The source domain image can be converted twice by sequentially converting through the first generator and the second generator, and the target domain image can be converted twice by sequentially converting through the second generator and the first generator.

[0066] S400, determining a cross-domain consistency loss according to a target domain image, a first image recognizer, a second generator, and a second image recognizer.

[0067] Cross-domain consistency loss refers to the difference between different recognition results after the target domain image has been converted to another domain through the second generator. The images of the two domains are respectively passed through two image recognizers to generate two different results. The cross-domain consistency loss can be determined based on the two results.

[0068] S500, determining an image recognition loss according to a source domain image, a first image recognizer, a first generator, and a second image recognizer.

[0069] Image recognition loss refers to the difference between different recognition results after the source domain image has been converted to another domain by the first generator. The images in the two domains are respectively passed through two image recognizers to generate two different results. The image recognition loss can be determined based on the two results.

[0070] S600, determining the total loss according to the image adversarial loss, the image cycle consistency loss, the cross-domain consistency loss and the image recognition loss, updating the parameters of the vehicle warning model according to the total loss until the preset requirements are met, and determining the vehicle warning model.

[0071] The specific calculation of the total loss is determined according to the actual application, and this embodiment does not impose any specific restrictions. Image adversarial loss, image cycle consistency loss, cross-domain consistency loss, and image recognition loss can correspond to different weights respectively. In a specific embodiment, the total loss function of the unsupervised image recognition method based on transfer learning is expressed as:

[0072] L=λ1·L task +λ2·L img +λ3·L cycle +λ4·L CDC

[0073] Among them, λ1, λ2, λ3, and λ4 all represent weight coefficients, L represents the total loss, and L task represents the image recognition loss, L img represents the image adversarial loss, L cycle represents the image cycle consistency loss, L CDC Indicates cross-domain consistency loss.

[0074] The preset requirements for stopping the training of the vehicle warning model are determined according to the actual application, and this embodiment does not impose specific restrictions, such as the total loss reaching a preset value or the number of training times reaching the requirement.

[0075] It should be noted that the specific structures of the generator, discriminator and image recognizer are determined according to actual applications and are not specifically limited in this embodiment.

[0076] Optionally, see Figure 3 and Figure 4 , according to the source domain image (Source), the target domain image (Target), the first generator (G S->T ), the first discriminator (D T ), the second generator (G T<-S ) and the second discriminator (D S )Determine the image adversarial loss, including:

[0077] S210, input the source domain image (Source) to the first generator (GS->T ), the target domain conversion image (Target(fake)) is obtained, and according to the target domain conversion image (Target(fake)), the target domain image (Target) and the first discriminator (D T ) Generate the first adversarial loss value (L img1 );

[0078] S220, input the target domain image (Target) into the second generator (G T<-S ), and obtain the source domain conversion image (Source(fake)). According to the source domain conversion image (Source(fake)), the source domain image (Source) and the second discriminator (D S ) Generate the second adversarial loss value (L img2 );

[0079] S230, according to the first confrontation loss value (L img1 ) and the second adversarial loss value (L img2 ) determines the image adversarial loss.

[0080] The embodiment of the present invention adopts pixel-level rather than feature-level domain adaptation in the model to obtain better conversion results. In the model of the embodiment of the present invention, there are two generators and two discriminators, one generator converts the image from the source domain to the target domain, and the other generator converts the image from the target domain to the source domain. s and X t→s The discriminator is D s , X t and X s→t The discriminator D T The generator and discriminator are the same as in CycleGAN. The purpose of the generator and discriminator is to make the generated image more similar to the target domain image X t In a specific embodiment, the calculation methods of the first adversarial loss value and the second adversarial loss value are respectively as follows:

[0081]

[0082] L img Pixel-level adversarial loss, in order to narrow the gap between the source and target domains, increases the similarity between the source and target domains when the discriminator in the network cannot determine the gap between the two domains.

[0083] Optionally, see Figure 3 and Figure 5 , according to the source domain image (Source), the target domain image (Target), the first generator (G S->T ), the second generator (G T<-S ) determines the image cycle consistency loss, including:

[0084] S310, input the source domain image (Source) to the first generator (G S->T ), and obtain the target domain conversion image (Target(fake)), and input the target domain conversion image (Target(fake)) into the second generator (G T<-S ), obtain the source domain restored image (Re-Source), and determine the first cycle consistency loss (L according to the source domain image (Source) and the source domain restored image (Re-Source) cycle1 );

[0085] S320, input the target domain image (Target) to the second generator (G T<-S ), and obtain the source domain conversion image (Source(fake)), and input the source domain conversion image (Source(fake)) into the first generator (G S->T ), obtain the target domain restored image (Re-Target), and determine the second cycle consistency loss (L according to the target domain image (Target) and the target domain restored image (Re-Target) cycle2 );

[0086] S330, according to the first cycle consistency loss (L cycle1 ) and the second cycle consistency loss (L cycle2 ) determines the image cycle consistency loss.

[0087] In the model, cycle consistency loss is used for image-to-image conversion. After transforming from one domain to another, the embodiment of the present invention performs an inverse transformation to obtain the same image, i.e., G t→s (G s→t (X s ))≈X s and G s→t (G t→s (X t ))≈X t In a specific embodiment, the formula is as follows:

[0088]

[0089] Although the generator and the discriminator have good performance in converting image styles, they are not sensitive to the content of the image. The embodiment of the present invention applies cycle consistency loss in the model, which can be achieved by using G s→t X S Convert to X s→t , and then use G t→s Generate X s→t→s , thereby improving the generator. X S and Xs→t→s Should have identical imaging and content.

[0090] L cycle Cycle consistency loss. When an image is converted from one domain to another, in order to make the converted image more similar to the image in the target domain, the embodiment of the present invention adds a cycle consistency loss so that the converted image is reversely converted, and the reconstructed image should be infinitely close to the original image.

[0091] Optionally, see Figure 3 and Figure 6 , according to the target domain image (Target), the first image recognizer (FRCN (Source)), the second generator (G T<-S ) and the second image recognizer (FRCN(Target)) determine the cross-domain consistency loss, including:

[0092] S410, determining a first recognition result according to the target domain image (Target) and the first image recognizer (FRCN (Source));

[0093] S420, input the target domain image (Target) to the second generator (G T<-S ), obtaining a source domain conversion image (Source(fake)), and determining a second recognition result according to the source domain conversion image (Source(fake)) and the second image recognizer (FRCN(Target));

[0094] S430: Determine cross-domain consistency loss according to the first recognition result and the second recognition result.

[0095] The goal of the embodiment of the present invention is to make the trained network applicable to the unlabeled data in the target domain. To achieve this goal, firstly, the image X in the target domain T dataset is t Convert to X t→s , X t→s The generator G in the pixel-level domain adaptation network t→s Generate (ie X t→s =G t→s (X t )). The data X belonging to the target domain t is transformed into data X belonging to the source domain t→s , so the color and style of the image may be different, but the content and annotations in the image are the same. In order to generate the two prediction results f t (i.e. F t (X t )) and f t→s (i.e. F s (X t→s )) is exactly the same, and enhances ft and f t→s The embodiment of the present invention uses cross-domain consistency loss (CDCloss) to constrain the consistency of the two image recognition networks F s and G t The output (i.e. f t and f t→s ). In a specific embodiment, the cross-domain consistency loss can be expressed as:

[0096]

[0097] Where w and h are the width and height of the image. t→s From X t The result of the conversion, which means f t→s and f t Have the same label.

[0098] In the model proposed in the embodiment of the present invention, CDCloss is applied to maintain the accuracy of the image recognition network in the target domain and the source domain. t→s Indicates F s Network in X t→s The prediction results on s It is a prediction network of the source domain. The training data of the network is the labeled source domain data, so the network prediction result is relatively accurate. In the embodiment of the present invention, CDCloss is added to the model to correct the unlabeled target domain network F. t In X t The prediction result f t And improve F t The accuracy of the network prediction results. In addition, the back propagation of the image recognition network FRCN can also correct the output of CycleGAN, thereby improving the accuracy of the generator.

[0099] L CDC Cross-domain consistency loss, which imposes consistency constraints on image recognition tasks and t and its transformed X t→s Adjust and adjust F during training s Network t→s The prediction results and F t Network t Implement constraints on the prediction results.

[0100] Optionally, see Figure 3 and Figure 7 , according to the source domain image (Source), the first image recognizer (FRCN (Source)), the first generator (G S->T), the second image recognizer (FRCN(Target)) determines the image recognition loss, including:

[0101] S510, determining a first recognition loss according to a source domain image (Source) and a first image recognizer (FRCN (Source));

[0102] S520, input the source domain image (Source) to the first generator (G S->T ), obtaining a target domain conversion image (Target(fake)), and determining a second recognition loss according to the target domain conversion image (Target(fake)) and the second image recognizer (FRCN(Target));

[0103] S530: Determine an image recognition loss according to the first recognition loss and the second recognition loss.

[0104] In the image recognition task, the embodiment of the present invention uses a fully residual convolutional neural network (FRCN). The difference between FRCN and other pixel-level segmentation networks is that it can use context information to obtain more accurate predictions. In the model of the embodiment of the present invention, two FRCN networks F s and F t , and applied to the source domain and the target domain. In order to use the annotations in the source domain to guide the training of the image recognition network in the task, the embodiment of the present invention adds L task In the source domain, each set of images and annotations (X s ,Y s ) are all matched. During the training process, the present invention uses X s Transform to X s→t (i.e. X s→t =G t→s (X s )). At this time, X s and X s→t should have the same true annotations. In summary, the image recognition loss can be expressed as follows:

[0105]

[0106] Among them, f s (w,h)=F s (X s ) and f t→s (w,h)=F s (X t→s ) is X s and X t→s The recognition result. Represents Y sis the average of (w,h), α and β are predefined constants used to adjust the weights of the losses from different parts of the model output.

[0107] Image feature migration, that is, the image conversion module, has two generators and two discriminators, G s→t , G t→s , D t and D s In a specific embodiment, see Figure 8 , G s→t and G t→s It is a Resnet-based generator, consisting of 9 Resnet blocks between downsampling and upsampling operations. Its structure is similar to CycleGAN, and consists of an encoder, a transformer, and a decoder. The encoder performs convolution downsampling to reduce the size of the input representation and increase the number of channels. The transformer consists of 9 resnet blocks, each of which has 2 convolution blocks. Each convolution block contains a convolution layer, a regularization layer, and a nonlinear layer (RELU). Its function is to combine different compact features of the image, and then determine how to transform the feature vector of the image from the source domain to the target domain based on these features. Finally, the size of the feature is sampled using the decoder by utilizing inverse convolution.

[0108] D t and D s For G s→t and G t→s The discriminator takes the image as input and predicts whether it is the original image or the image generated by the generator. After extracting features from the image, the discriminator composed of 5 convolutional layers determines whether the above features belong to a specific category. The main purpose of the discriminator is to reduce the gap between the source domain and the target domain, thereby mapping the data from the source domain to the target domain until the discriminator cannot distinguish between the images in the source domain and the generated images. In addition, since the Cycle Consistency Generative Adversarial Network (CycleGAN) model has two pairs of generators and discriminators, the model is able to convert the image back to the original domain, which can better preserve the image content, making the image recognition results more accurate.

[0109] The image recognition module consists of two image recognition networks operating on the source domain and the target domain, including but not limited to the fully residual convolutional neural network (FRCN). Fig.10, the structure of the FRCN used is similar to U-net, which is divided into two parts, namely the contraction path and the expansion path. The main function of the contraction path is to extract high-level features of the image and gradually increase the perception field by downsampling. The contraction path consists of a residual block after repeated superposition of 3×3 convolutions and a 2×2 downsampling layer. Before downsampling the feature map, the feature map channel is doubled to 256 using 1×1 convolution. Different from FRCN, all downsampling operations used in the method of the embodiment of the present invention are maximum pooling layers. In the process of extracting global features of the image, local feature information is inevitably lost. The biggest difference between the expansion path and the contraction path is that the downsampling in the contraction path is replaced by upsampling. This embodiment uses bilinear interpolation to upsample the feature map. In order to compensate for the loss of high-resolution information in the downsampling layer and improve the positioning accuracy, the high-resolution feature map on the contraction path is connected to the corresponding upsampled feature map on the expansion path. The biggest difference between the expansion path and the contraction path is that the downsampling in the contraction path is replaced by upsampling, and the method uses bilinear interpolation to upsample the feature map. To compensate for the large loss of high-resolution information during downsampling and improve localization accuracy, the high-resolution feature maps are connected from the contracting path to the corresponding upsampled feature maps in the expanding mode. Feature map concatenation: In the expanding path, upsampling layers are used to expand the size of the feature maps, however, the upsampling and downsampling layers are not completely "symmetrical". For example, after the downsampling operation, the size of the 25×25 feature map becomes 12×12. However, when the 12×12 feature map is upsampled, only a 24×24 feature map is obtained. In order to maintain the size during this process, the upsampled feature maps are padded to match the size of the corresponding feature maps in the contracting path. The two sets of feature maps are then cascaded and passed to the following convolutional layer to infer the output.

[0110] In addition, the biggest difference between FRCN and U-Net is that FRCN uses residual blocks for both the contraction path and the expansion path. Fig. 9 , each residual block can be expressed as output = input + F (input, W), where the F function includes activation function layer, convolution layer, regularization layer, activation function layer, convolution layer and scaling layer in sequence. F represents the ELU-Conv-Dropout-ELU-Conv series of operations, where W represents the weight coefficient in the F operation to prevent the gradient explosion problem, and the output of F is reduced by a certain proportion and then used as the output of the residual block.

[0111] The implementation of the embodiment of the present invention includes the following beneficial effects: the vehicle warning model includes an image conversion module and an image recognition module, the image conversion module includes a first generator, a second generator, a first discriminator and a second discriminator, the image recognition module includes a first image recognizer and a second image recognizer, and the training method includes: obtaining training data, the training data includes annotated source domain images and unlabeled target domain images; determining image adversarial loss according to the source domain image, the target domain image, the first generator, the first discriminator, the second generator and the second discriminator; determining image cycle consistency loss according to the source domain image, the target domain image, the first generator and the second generator; determining image cycle consistency loss according to the target domain image, the first image recognizer, the second generator and the second discriminator; The image recognizer determines the cross-domain consistency loss; the image recognition loss is determined according to the source domain image, the first image recognizer, the first generator, and the second image recognizer; the total loss is determined according to the image adversarial loss, the image cycle consistency loss, the cross-domain consistency loss and the image recognition loss, and the parameters of the vehicle warning model are updated according to the total loss until the preset requirements are met, and the vehicle warning model is determined; the image conversion module and the image recognition module are merged into an end-to-end adversarial learning model, and the mutual constraints of accuracy are achieved through multiple loss functions, and the image recognizer is trained with labeled source domain images through the method of transfer learning domain adaptation, so that the image recognizer can achieve the purpose of predicting unlabeled target domain images, thereby realizing unsupervised training.

[0112] See also Fig.11 , an embodiment of the present invention provides a training system for a lane warning model, comprising:

[0113] The first module is used to obtain training data, which includes labeled source domain images and unlabeled target domain images;

[0114] A second module is used to determine the image adversarial loss according to the source domain image, the target domain image, the first generator, the first discriminator, the second generator and the second discriminator;

[0115] The third module is used to determine the image cycle consistency loss according to the source domain image, the target domain image, the first generator, and the second generator;

[0116] A fourth module, for determining a cross-domain consistency loss based on the target domain image, the first image recognizer, the second generator, and the second image recognizer;

[0117] A fifth module, configured to determine an image recognition loss according to the source domain image, the first image recognizer, the first generator, and the second image recognizer;

[0118] The sixth module is used to determine the total loss based on image adversarial loss, image cycle consistency loss, cross-domain consistency loss and image recognition loss, and update the parameters of the vehicle warning model according to the total loss until the preset requirements are met, thereby determining the vehicle warning model.

[0119] It can be seen that the contents of the above training method embodiments are all applicable to the present training system embodiments, the functions specifically implemented by the present training system embodiments are the same as those of the above training method embodiments, and the beneficial effects achieved are also the same as those achieved by the above training method embodiments.

[0120] An embodiment of the present invention provides a method for recognizing a lane warning image, the method comprising:

[0121] The target domain image is obtained, and the target domain image is input into the trained second image recognizer to obtain a lane warning image recognition result.

[0122] After the vehicle warning model is trained according to the above method, the second image recognizer in the vehicle warning model is extracted, and then the target domain image is input into the extracted second image recognizer to obtain the lane warning image recognition result.

[0123] See also Fig.12 , an embodiment of the present invention provides a lane warning image recognition device, comprising:

[0124] at least one processor;

[0125] at least one memory for storing at least one program;

[0126] When at least one program is executed by at least one processor, the at least one processor implements the above-mentioned training method or recognition method.

[0127] Among them, the memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. The memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a remote memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0128] It can be seen that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0129] In addition, the embodiment of the present application also discloses a computer program product or a computer program, which is stored in a computer-readable storage medium. The processor of the computer device can read the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device performs the above method.

[0130] The embodiment of the present invention further provides a computer-readable storage medium, which stores a program executable by a processor, and the program executable by the processor is used to implement the above method when executed by the processor. Similarly, the contents of the above method embodiment are all applicable to the present storage medium embodiment, and the functions specifically implemented by the present storage medium embodiment are the same as those of the above method embodiment, and the beneficial effects achieved are also the same as those achieved by the above method embodiment.

[0131] It is understood that all or some steps and systems in the disclosed method above can be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, and the computer-readable medium can include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those of ordinary skill in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassette, magnetic tape, disk storage or other magnetic storage device, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0132] On the other hand, an embodiment of the present invention provides a vehicle, which includes the above-mentioned lane warning model training system or the above-mentioned lane warning image recognition device.

[0133] The embodiment further provides a vehicle, which includes the training system of the lane warning model or the recognition device of the lane warning image or the electric drive assembly of the computer device (or electronic device). Specifically, the vehicle can be a private car, such as a sedan, SUV, MPV or pickup truck. The vehicle can also be an operating vehicle, such as a van, bus, small truck or large trailer. The vehicle can be a gasoline vehicle or a new energy vehicle. When the vehicle is a new energy vehicle, it can be a hybrid vehicle or a pure electric vehicle.

[0134] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0135] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0136] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0137] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A training method for a lane warning model, characterized in that: The vehicle warning model includes an image conversion module and an image recognition module, the image conversion module includes a first generator, a second generator, a first discriminator and a second discriminator, the image recognition module includes a first image recognizer and a second image recognizer, and the training method includes: Acquire training data, where the training data includes labeled source domain images and unlabeled target domain images; Determining an image adversarial loss based on the source domain image, the target domain image, the first generator, the first discriminator, the second generator, and the second discriminator; Determine an image cycle consistency loss according to the source domain image, the target domain image, the first generator, and the second generator; determining a cross-domain consistency loss based on a target domain image, the first image recognizer, the second generator, and a second image recognizer; Determining an image recognition loss based on a source domain image, the first image recognizer, the first generator, and the second image recognizer; The total loss is determined according to the image adversarial loss, the image cycle consistency loss, the cross-domain consistency loss and the image recognition loss, and the parameters of the vehicle warning model are updated according to the total loss until the preset requirements are met, thereby determining the vehicle warning model.

2. The training method according to claim 1, characterized in that: The determining the image adversarial loss according to the source domain image, the target domain image, the first generator, the first discriminator, the second generator and the second discriminator comprises: Inputting the source domain image into a first generator to obtain a target domain conversion image, and generating a first adversarial loss value according to the target domain conversion image, the target domain image, and the first discriminator; Inputting the target domain image into a second generator to obtain a source domain conversion image, and generating a second adversarial loss value according to the source domain conversion image, the source domain image, and the second discriminator; An image adversarial loss is determined according to the first adversarial loss value and the second adversarial loss value.

3. The training method according to claim 1, characterized in that: The determining the image cycle consistency loss according to the source domain image, the target domain image, the first generator, and the second generator includes: Inputting the source domain image into a first generator to obtain a target domain conversion image, inputting the target domain conversion image into a second generator to obtain a source domain restoration image, and determining a first cycle consistency loss according to the source domain image and the source domain restoration image; Inputting the target domain image into the second generator to obtain a source domain conversion image, inputting the source domain conversion image into the first generator to obtain a target domain restoration image, and determining a second cycle consistency loss according to the target domain image and the target domain restoration image; An image cycle consistency loss is determined according to the first cycle consistency loss and the second cycle consistency loss.

4. The training method according to claim 1, characterized in that: The determining the cross-domain consistency loss according to the target domain image, the first image recognizer, the second generator and the second image recognizer comprises: determining a first recognition result according to the target domain image and the first image recognizer; Inputting the target domain image into a second generator to obtain a source domain conversion image, and determining a second recognition result according to the source domain conversion image and the second image recognizer; A cross-domain consistency loss is determined according to the first recognition result and the second recognition result.

5. The training method according to claim 1, characterized in that: The determining of the image recognition loss according to the source domain image, the first image recognizer, the first generator, and the second image recognizer comprises: determining a first recognition loss based on the source domain image and the first image recognizer; Inputting the source domain image into a first generator to obtain a target domain conversion image, and determining a second recognition loss according to the target domain conversion image and the second image recognizer; An image recognition loss is determined according to the first recognition loss and the second recognition loss.

6. A method for recognizing lane warning images, characterized in that: The identification method comprises: Acquire a target domain image, input the target domain image into a second image recognizer trained according to any one of claims 1 to 5, and obtain a lane warning image recognition result.

7. A lane warning model training system, characterized in that: include: The first module is used to obtain training data, where the training data includes labeled source domain images and unlabeled target domain images; A second module, configured to determine an image adversarial loss according to the source domain image, the target domain image, the first generator, the first discriminator, the second generator, and the second discriminator; A third module is used to determine an image cycle consistency loss according to the source domain image, the target domain image, the first generator, and the second generator; A fourth module, configured to determine a cross-domain consistency loss based on a target domain image, the first image recognizer, the second generator, and the second image recognizer; A fifth module, configured to determine an image recognition loss according to a source domain image, the first image recognizer, the first generator, and the second image recognizer; The sixth module is used to determine the total loss based on the image adversarial loss, the image cycle consistency loss, the cross-domain consistency loss and the image recognition loss, and update the parameters of the vehicle warning model according to the total loss until the preset requirements are met, thereby determining the vehicle warning model.

8. A lane warning image recognition device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a program executable by a processor, characterized in that: The processor-executable program is used to perform the method according to any one of claims 1 to 6 when executed by the processor.

10. A vehicle, characterized in that: The vehicle includes the training system of the lane warning model as described in claim 7 or the recognition device of the lane warning image as described in claim 8.

Images