An environment recognition model training method, driving control method and related device

Abstract

Embodiments of the present invention provide an environment recognition model training method, a driving control method and related devices. The environment recognition model training method obtains an image of an environment that interrupts the automatic driving mode as a style image, and inputs the style image and the original image in the original training image set into an image style transfer network, thereby transferring the style of the image of the environment that interrupts the automatic driving mode to the original image collected during the vehicle driving process, thereby solving the problem that the environment recognition model is difficult to obtain training data. In addition, the trained environment recognition model can be used to identify the environment image that interrupts the automatic driving mode in the actual driving scene, and the image can be used as a new training image to update the environment recognition model, so that the environment recognition model can continuously improve its ability to identify various environments that interrupt the automatic driving mode, improve the accuracy of identifying the environment that interrupts the automatic driving mode, and ensure the safety performance of automatic driving.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of automatic driving control technology, and in particular to an environment recognition model training method, a driving control method and related devices. Background Art

[0002] During driving, autonomous vehicles determine whether the driving environment is suitable for autonomous driving by sensing the driving environment around the vehicle in real time, such as weather, roads, and other driving environments. When the driving environment is not suitable for autonomous driving mode, the autonomous driving mode is interrupted. Therefore, it is crucial to identify whether the driving environment is suitable for autonomous driving.

[0003] With the development of artificial intelligence technology, neural networks have been widely used in target recognition and environmental perception. If the environment that interrupts the autonomous driving mode is to be identified through the neural network, a large amount of data containing the environment that interrupts the autonomous driving mode is required to train the neural network. For example, images of environments that are not suitable for the autonomous driving mode, such as bad weather and deteriorating roads, are required as training data. However, in the actual driving process of autonomous vehicles, such images are relatively sparse, which makes it difficult to obtain training data, and the environmental recognition ability of the obtained neural network is poor. Summary of the invention

[0004] The embodiment of the present invention proposes an environment recognition model training method to solve the problem that it is difficult to obtain training data when training the environment recognition model and the environment recognition ability of the neural network is poor.

[0005] In a first aspect, an embodiment of the present invention provides an environment recognition model training method, comprising:

[0006] Acquire a style image including a specified style, where the specified style is an image style of a preset environment for interrupting the automatic driving mode;

[0007] Inputting the style image and the original image in the original training image set into an image style transfer network to transfer the specified style to the original image, thereby obtaining a first training image set including the first image;

[0008] Using the first training image set to train an environment recognition model;

[0009] Inputting the acquired second image into the environment recognition model to obtain a preset environment contained in the second image and a first probability of containing the preset environment;

[0010] When the first probability is greater than a preset probability, adding the second image to the first training image set to obtain a second training image set;

[0011] The environment recognition model is updated using the second training image set.

[0012] In a second aspect, an embodiment of the present invention further provides a driving control method,

[0013] When the autonomous driving vehicle is driving in the autonomous driving mode, an image is collected to obtain an image to be recognized;

[0014] Inputting the image to be recognized into a pre-trained environment recognition model to obtain a probability that the image to be recognized contains a preset environment, where the preset environment is an environment that interrupts the automatic driving mode;

[0015] When the probability is greater than a preset probability, controlling the autonomous driving vehicle to exit the autonomous driving mode;

[0016] Wherein, the environment recognition model is trained by the environment recognition model training method as described in the first aspect.

[0017] In a third aspect, an embodiment of the present invention further provides an environment recognition model training device,

[0018] A style image acquisition module, used to acquire a style image containing a specified style, where the specified style is an image style of a preset environment for interrupting the automatic driving mode;

[0019] A first training image set acquisition module, used for inputting the style image and the original image in the original training image set into an image style transfer network to transfer the specified style to the original image, thereby obtaining a first training image set including the first image;

[0020] A model training module, used for training an environment recognition model using the first training image set;

[0021] A first environment recognition module, used for inputting the collected second image into the environment recognition model to obtain a preset environment contained in the second image and a first probability of containing the preset environment;

[0022] A second training image set acquisition module, configured to add the second image to the first training image set to obtain a second training image set when the first probability is greater than a preset probability;

[0023] A model updating module is used to update the environment recognition model using the second training image set.

[0024] In a fourth aspect, an embodiment of the present invention further provides a driving control device, comprising:

[0025] An image acquisition module, used to acquire images to be recognized when the autonomous driving vehicle is driving in the autonomous driving mode;

[0026] A second environment recognition module is used to input the image to be recognized into a pre-trained environment recognition model to obtain a probability that the image to be recognized contains a preset environment, where the preset environment is an environment that interrupts the automatic driving mode;

[0027] A vehicle control module, configured to control the autonomous driving vehicle to exit the autonomous driving mode when the probability is greater than a preset probability;

[0028] Wherein, the environment recognition model is trained by the environment recognition model training method as described in the first aspect.

[0029] In a fifth aspect, an embodiment of the present invention further provides an electronic device, the electronic device comprising:

[0030] one or more processors;

[0031] a memory for storing one or more computer programs,

[0032] When the one or more computer programs are executed by the one or more processors, the one or more processors implement the environment recognition model training method as described in the first aspect, and / or the driving control method as described in the second aspect.

[0033] In a sixth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the environment recognition model training method as described in the first aspect and / or the driving control method as described in the second aspect are implemented.

[0034] The embodiment of the present invention obtains an image containing a preset environment that interrupts the automatic driving mode as a style image, inputs the style image and the original image in the original training image set into an image style transfer network, and transfers the image style in the style image to the original image to obtain a first training image set containing a first image, so that the first image contains the image style of the preset environment that interrupts the automatic driving mode, and further uses the first training image set as training data for an environment recognition model, thereby transferring the style of the image containing the preset environment to the original image collected during the vehicle driving process by way of style transfer, and can conveniently and quickly obtain images containing various preset environments as training data for the environment recognition model. On the other hand, the trained environment recognition model can be used to identify images containing the preset environment in the actual driving scene, and the image can be used as new training data to update the environment recognition model, so that the environment recognition model can continuously improve its ability to recognize various preset environments, thereby improving the accuracy of recognizing the preset environment and ensuring the safety performance of automatic driving.

[0035] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A flowchart of an environment recognition model training method provided in Embodiment 1 of the present invention;

[0037] Figure 2 A schematic diagram of style migration image comparison provided in the first embodiment of the present invention;

[0038] Figure 3 is a flow chart of an environment recognition model training method provided by Embodiment 2 of the present invention;

[0039] Figure 4 is a schematic diagram of a region of interest provided in Embodiment 2 of the present invention;

[0040] Figure 5 is a flow chart of a driving control method provided by Embodiment 3 of the present invention;

[0041] Figure 6 A schematic diagram of the structure of an environment recognition model training device provided in Embodiment 4 of the present invention;

[0042] Figure 7 A schematic diagram of the structure of a driving control device provided in Embodiment 5 of the present invention;

[0043] Figure 8 A schematic diagram of the structure of an electronic device provided in Embodiment 6 of the present invention. DETAILED DESCRIPTION

[0044] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0045] Embodiment 1

[0046] Figure 1This is a flow chart of an environment recognition model training method provided in Embodiment 1 of the present invention. This embodiment can be applied to training an environment recognition model to identify the situation where the environment interrupts the automatic driving mode. The method can be executed by an environment recognition model training device. The environment recognition model training device can be implemented by software and / or hardware and can be configured in an electronic device, such as a computer. The method specifically includes the following steps:

[0047] S101: Acquire a style image containing a specified style.

[0048] The specified style may be the image style of the image when the image contains a preset environment, wherein the preset environment may be a driving environment that interrupts the autonomous driving mode, and the driving environment that interrupts the autonomous driving mode may be an environment that causes the perception system on the autonomous driving vehicle to be unable to accurately perceive. In one example, the preset environment may be a severe weather environment, such as heavy rain, heavy snow, haze, etc., which make it impossible for the camera to capture images and require manual driving. It may also be a traffic environment with a high degree of bumpiness on the road, or heavy traffic, dense vehicles, and dense pedestrians that require manual driving. Of course, the preset environment may also include other driving environments that cause switching from the autonomous driving mode to the manual driving model.

[0049] The purpose of obtaining the style image is to use the style in the style image, and there is no restriction on the image content. In an optional embodiment, images containing a preset environment can be searched on the Internet as style images, for example, images containing weather environments such as heavy rain, heavy snow, and haze can be searched as style images, and images of driving scenes such as bumpy roads and dense traffic can be searched as style images. The embodiment of the present invention does not limit the method of obtaining images containing a preset environment.

[0050] S102: Input the style image and the original image in the original training image set into an image style transfer network to transfer the specified style to the original image, thereby obtaining a first training image set including the first image.

[0051] An embodiment of the present invention can pre-train an image style transfer network, which can transfer the image style of one image to another image. In one example, the image style transfer network has two image input channels, one input channel is used to input the style image, and the other input channel is used to input the original image. After the style image and the original image are input, the image style transfer network extracts the image style in the style image and extracts the image content in the original image, and outputs a first image after fusing the image style and the image content, so that the first image contains the original image and has the style of the style image.

[0052] In one example, if Figure 2As shown, the original image is an image taken by the autonomous driving vehicle while driving on a sunny day, and the style image is an image of other scenes on a rainy day. After the style image and the original image are input into the image style transfer network, an image simulated of the autonomous driving vehicle taken while driving in the rain can be obtained, namely, the first image.

[0053] In practical applications, each specified style can correspond to an image style transfer network, and the image style transfer network can be trained by various training methods. For example, the image style transfer network can be trained based on the training method of the adversarial generative network, and can also be trained by other supervised training methods. The embodiment of the present invention does not limit the method of training the image style transfer network.

[0054] S103: Use the first training image set to train an environment recognition model.

[0055] After obtaining the first training image set, a label can be annotated for each image in the first training image set, for example, the preset environment contained in the annotated image and the probability of containing the preset environment. For example, for the image style of a heavy rain environment, it is determined whether the image contains a heavy rain environment. If so, it is marked as 1, otherwise it is marked as 0, and a probability is assigned at the same time. The probability represents the possibility of containing a heavy rain environment in the image.

[0056] When training the environment recognition model, the first image in the first training image set is input into the environment recognition model to obtain whether it contains the preset environment and the corresponding probability. The environment recognition model can be adjusted after the loss rate is calculated by probability until the loss rate is less than the preset threshold or the number of iterative training times is greater than the preset number. The training is stopped to obtain a trained environment recognition model. When the user inputs an image, the environment recognition model can predict the environment and probability contained in the image.

[0057] It should be noted that the environment recognition model can be a neural network such as CNN, DNN, RNN, etc., and the training method can adopt a supervised learning training method. The embodiment of the present invention does not limit the network structure and training method of the environment recognition model.

[0058] S104: Input the collected second image into an environment recognition model to obtain a preset environment contained in the second image and a first probability of containing the preset environment.

[0059] The second image is an image collected in an actual driving scene. Specifically, after the environment recognition model is trained, it can be applied to an autonomous driving vehicle. During driving, the autonomous driving vehicle can collect images of the surrounding environment through an image acquisition device such as a camera to obtain a second image, and input the collected second image into the environment recognition model to obtain the category of the preset environment and the first probability that the second image contains the preset environment. A second image may not contain a preset environment, or may contain at least one type of preset environment. When the environment recognition model detects that the second image contains a preset environment, it can output the preset environment and the first probability that the second image contains the preset environment.

[0060] S105. When the first probability is greater than a preset probability, add the second image to the first training image set to obtain a second training image set.

[0061] The preset probability is used to measure the confidence that the second image contains the preset environment. The higher the probability, the higher the confidence that the preset environment is contained in the predicted image. When the first probability is greater than the preset probability, it means that the confidence that the second image contains the preset environment is high, and the current driving environment is not suitable for the automatic driving mode. The second image can be added as new training data to the first training image set to obtain a second training image set.

[0062] S106: Use the second training image set to update the environment recognition model.

[0063] The second image is a real image of a preset environment that interrupts the autonomous driving mode. Compared with the first image obtained by image style transfer in the first training image set, the features of the preset environment contained in the second image are more realistic. The second image is also contained in the second training image set. Updating the environment recognition model through the second training image set can enable the environment recognition model to learn the ability to recognize the preset environment in the real driving scene, and can also improve the recognition accuracy of the environment recognition model for the preset environment.

[0064] The embodiment of the present invention obtains an image of an environment that interrupts the automatic driving mode as a style image, inputs the style image and the original image in the original training image set into an image style transfer network, and transfers the image style in the style image to the original image to obtain a first training image set containing a first image, so that the first image contains the image style of the environment that interrupts the automatic driving mode, and further uses the first training image set as training data for an environment recognition model, thereby solving the problem that the environment recognition model is difficult to obtain training data, and can conveniently and quickly obtain images of various environments that interrupt the automatic driving mode as training images. On the other hand, when the environment that interrupts the automatic driving mode in the actual driving scene can be identified through the trained environment recognition model, the image collected by the vehicle is used as a new training image to update the environment recognition model, so that the environment recognition model can continuously improve its ability to identify various environments that interrupt the automatic driving mode, improve the accuracy of identifying the environment that interrupts the automatic driving mode, and ensure the safety performance of automatic driving.

[0065] Embodiment 2

[0066] Figure 3 This is a flow chart of an environment recognition model training method provided in the second embodiment of the present invention. This embodiment is further optimized on the basis of the above embodiment. Figure 3 As shown, the environment recognition model training method includes:

[0067] S301: Acquire a style image containing a specified style.

[0068] The specified style can be the image style of the image when the image contains a preset environment. The preset environment can be classified into weather, lighting, road surface, sensor, external interference items, etc. For example, regarding lighting, when sunlight or other car lights directly hit the camera of the autonomous driving vehicle, the shooting picture is unclear and the environment cannot be judged; regarding external interference items, such as pseudo 3D images and water splashing on the road, they may cause environmental recognition errors. The preset environment can be set and updated according to the actual situation, so as to be flexibly applied to autonomous driving vehicles of different application scenarios and different models.

[0069] S302: Divide a plurality of original images in an original training image set into a plurality of image groups.

[0070] The original image is a real image of the environment acquired by the autonomous driving vehicle while driving. When the interval time of image acquisition is short or the vehicle speed is slow, images with small environmental differences may be acquired in a continuous time. If the style transfer of the same image style is performed on the original images with small environmental differences, the degree of environmental differences between the first images obtained may also be small. Therefore, the multiple images in the first training image set can be divided into multiple image groups, and the degree of environmental differences between the original images in each image group is large. When the number of original images is large enough, a random grouping method can be used to extract a preset number of original images from the original images as an image group. The advantage of such grouping is that the original data can be grouped simply and quickly, and the possibility of the original images in each image group being similar is small, that is, the degree of difference between the original images in the image group can also be large. In addition, when grouping the original images, the grouping can be first performed according to the acquisition time and / or acquisition position of the original images, and the original images with an acquisition time interval greater than the preset time and / or an acquisition position distance greater than the preset distance are first grouped into the same image group.

[0071] In an example of this embodiment, after the original images are initially grouped, the original images with smaller environmental differences can be screened out through the similarity between the original images in each image group to obtain the final image group, which can ensure that the environmental differences between the original images in the image group are larger.

[0072] S303: For each designated style, determine a target image group corresponding to the designated style.

[0073] Each image group may perform style transfer with at least one style image of a specified style. Before performing image style transfer, a corresponding target image group may be determined for each specified style.

[0074] S304: Input the style image containing the specified style and the original image in the target image group into an image style transfer network to transfer the specified style to the original image, thereby obtaining a first training image set containing the first image.

[0075] The original images with large environmental differences in the same image group generally include different elements, including streets, buildings, trees, other vehicles, pedestrians, etc. The style image containing the specified style and the original image in the target image group are input into the image style transfer network, that is, after the style transfer of the same image style is performed on the original images in the image group, the first image with large environmental differences can also be obtained. The first training image set containing the first image is used as the training data of the environment recognition model, which improves the significance and richness of the training data, thereby improving the environment recognition ability of the environment recognition model.

[0076] In an optional embodiment, when performing image style transfer, for each specified style, the style image containing the specified style and the original image in the original training image set can be input into the image style transfer network. For example, if there are 100 original images, 30 style images of rainy day style and 100 original images are input into the style transfer network, and 100 first images of rainy day style can be obtained. In this embodiment, when performing image style transfer, for each style image, an original image can be randomly selected as the target image for its style transfer, or style transfer can be performed according to the arrangement order of the original images. All style images of the specified style use the same original image as the object of style transfer, which can reduce the demand for the number of original images and can also simply and quickly obtain a first image with a large amount of data.

[0077] In another optional embodiment, when performing image style transfer, a region of interest associated with a preset environment is also cut out from the original image in the original training image set to obtain a third image; the style image and the third image are input into the image style transfer network to obtain a fourth image, the fourth image is replaced into the region of interest of the original image to obtain the first image, and the collection of the first images is used as the first training image set.

[0078] The region of interest is the key area that the environment recognition model focuses on when analyzing the original image. For each preset environment, the associated region of interest may be the entire area of ​​the image or a part of the image. Figure 4 As shown, when a preset environment related to road conditions is to be identified, the associated area of ​​interest is the driving road area, i.e. Figure 4 The area surrounded by the dotted line in the middle is the area surrounded by the dotted line, while other areas such as roads, sky, trees, and buildings are non-key areas. Therefore, for the preset environment where the area of ​​interest is a part of the image, the area of ​​interest associated with the preset environment can be cut out from the original image to obtain a third image, and the third image is subjected to image style transfer of a specified style to obtain a fourth image, and then the fourth image is spliced ​​with the original image. By delineating the area of ​​interest in the image, the time for image style transfer of non-interest areas can be reduced, and the accuracy of image style transfer of the area of ​​interest can be increased.

[0079] S306: Use the first training image set to train an environment recognition model.

[0080] In this embodiment, the training of the environment recognition model includes: labeling the first image in the first training image set, the label including the preset environment contained in the first image and the second probability of containing the preset environment, initializing the environment recognition model, randomly extracting the first image and inputting it into the environment recognition model, obtaining the third probability that the first image contains the preset environment, using the second probability and the third probability to calculate the loss rate, and judging whether the loss rate is less than a preset threshold. If so, stopping the training of the environment recognition model to obtain the trained environment recognition model; if not, using the loss rate to adjust the model parameters of the environment recognition model, and returning to the step of randomly extracting the first image and inputting it into the environment recognition model.

[0081] The environment recognition model can be trained using methods such as gradient descent method and learning rate method.

[0082] When labeling the first image, only one label representing the preset environment is set for each first image. For a first image containing multiple preset environments, multiple first images can be copied, and each image is labeled with a different preset environment label. For each first image, the second probability that it contains the preset environment is 100%. Of course, the first training image set can also include other images. If the other images do not contain the preset environment, the labels of the other images can be set to any preset environment, but the second probability that they contain the preset environment is always 0%.

[0083] For the same first image, the loss rate is used to represent the difference between the known second probability and the predicted third probability. The smaller the loss rate, the smaller the difference between the second probability and the third probability, the better the training effect of the environment recognition model, and the higher the confidence of the preset environment predicted by the environment recognition model. Specifically, the sum of the standard deviations of the second probability and the third probability or the mean square error can be used as the loss rate. The preset threshold is used to determine whether the training effect of the environment recognition model meets the preset requirements, and the number of training times for the environment recognition model can also be set. When the number of training times reaches the preset number, the training ends.

[0084] S307: Input the collected second image into an environment recognition model to obtain the preset environment contained in the second image and a first probability of containing the preset environment.

[0085] The second image is an image collected in an actual driving scenario and can be applied to autonomous vehicles after the preliminary training of the environment recognition model is completed. The autonomous driving vehicle can collect the second image when it is in unmanned driving mode, or it can collect the second image when the vehicle is in unmanned driving mode and receives forced human intervention. For example, if the preset environment does not include a strong wind environment, and the vehicle recognition model cannot recognize the current strong wind environment, the driver brakes urgently or changes the route, then the current environment image can be collected. The current environment image may include an unknown environment that interrupts the autonomous driving mode. Using it as new training data can continuously improve the ability to identify various environments that interrupt the autonomous driving mode.

[0086] The environmental recognition model can recognize that the second image contains the category of the preset environment and the first probability. In an example of the present embodiment, when the environmental recognition model recognizes the second image, it can also divide the region of interest associated with the preset environment for each preset environment, and then input the region of interest into the corresponding environmental recognition model to confirm whether there is a preset environment in the region of interest and the first probability of the existence of the preset environment. By identifying the region of interest, the recognition accuracy can be improved, and the efficiency of image recognition can also be improved.

[0087] S308: When the first probability is greater than the preset probability, label the second image.

[0088] After obtaining a second image containing an image style of a preset environment, a label can be set for the second image according to the preset environment contained therein. The label includes the preset environment contained in the second image and the probability of containing the preset environment. After setting the label, the second image can also be sent to a folder for storing images of different preset environments. For example, a picture of a rainy environment is sent to a folder with a rainy style. By associating the second image with a specific folder, on the one hand, it can be distinguished from the first image, and on the other hand, it can facilitate manual classification and management of data.

[0089] S309: Add the labeled second image to the first training image set to obtain a second training image set.

[0090] The second training image set includes the first image in the first training image set and a newly added second image. The first image is obtained by style migration from the original data, and the second image is real data collected by the autonomous driving vehicle while driving. Compared with the first image, the second image is more realistic and has a stronger ability to express the characteristics of the preset environment. Using the second image as newly added training data can improve the quality of the training data of the environment recognition model.

[0091] In an example of this embodiment, after the data volume of the second image reaches a preset data volume, each time a new second image is added, a corresponding number of first images can be randomly eliminated from the second training image set, and the first images can be gradually replaced with real second images to further improve the quality of the training data of the environment recognition model. In addition, when all the images in the second training image set are second images, the images can also be updated periodically.

[0092] S310: Use the second training image set to update the environment recognition model.

[0093] Part or all of the images in the second training image set are real images of a preset environment that interrupts the autonomous driving mode, which are collected by the vehicle during driving. Compared with the first training image set, the data in the second training image set is more realistic and has higher data quality. Using the image set in the second training image set to update the environment recognition model can enhance the environment recognition capability of the environment recognition model, i.e., improve the safety of autonomous driving.

[0094] The embodiment of the present invention can expand the environmental differences between the original images in each image group by obtaining an image containing a preset environment that interrupts the automatic driving mode as a style image, and then grouping the original images in the original training image set. When the style of the preset environment in the style image is transferred to the corresponding target image group, the environmental differences between the first images in each image group are also large, which can improve the significance and richness of the training data of the environment recognition model. On the other hand, the environment recognition model can be used to identify the second image containing the preset environment in the actual driving scene. After labeling the second image, it is added to the first training image set to obtain the second training image set, which is convenient for the staff to manage the second image. The first image can also be eliminated as the second image increases. The second image is more realistic than the first image. In this way, the quality of the training data can be improved, so that the environment recognition model can continuously improve its ability to recognize various preset environments, that is, improve the safety performance of automatic driving.

[0095] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0096] Embodiment 3

[0097] Figure 5This is a flow chart of a driving control method provided in Embodiment 3 of the present invention. This embodiment is applicable to the case of controlling an autonomous driving vehicle. The method can be executed by a driving control device. The driving control device can be implemented by software and / or hardware and can be configured in an autonomous driving vehicle, such as in an onboard computer of an autonomous driving vehicle. The method specifically includes the following steps:

[0098] S501. While the autonomous driving vehicle is driving in the autonomous driving mode, an image is collected to obtain an image to be recognized.

[0099] S502: Input the image to be recognized into a pre-trained environment recognition model to obtain the probability that the image to be recognized contains a preset environment.

[0100] The preset environment is an environment in which the automatic driving mode is interrupted. If a to-be-recognized image contains multiple preset environments, there is a probability that there are multiple preset environments.

[0101] S503: When the probability is greater than a preset probability, control the autonomous driving vehicle to exit the autonomous driving mode.

[0102] When there are multiple probabilities of preset environments, the maximum probability can be compared with the preset probability.

[0103] The preset probability is used to measure the confidence that the image to be identified contains the preset environment. When the probability is greater than the preset probability, it means that the confidence that the image to be identified contains the preset environment is high, and it can be confirmed that the current driving environment is not conducive to safe driving, that is, the autonomous driving vehicle can be controlled to exit the autonomous driving mode. Specifically, the autonomous driving vehicle can be controlled to perform parking operations and park the vehicle in a safe area, or remind passengers to drive manually. When the passenger's manual driving operation is detected, the autonomous driving mode is exited.

[0104] In an example of this embodiment, after controlling the autonomous driving vehicle to exit the autonomous driving mode, the method further includes: outputting the image to be recognized to a preset training image set. The training image set is used to update the environment recognition model.

[0105] After an embodiment of the present invention collects an image during the autonomous driving process, the probability that the image contains a preset environment is obtained by inputting the image into an environment recognition model, and when the probability is greater than the preset probability, the image is added to a preset training image set, and the training image set is used to update the environment recognition model. Since the environment recognition model obtains training data through migration, the data volume is rich, the environment recognition ability is strong, and new training images are collected during the autonomous driving process for model updating, the learning ability and recognition accuracy of the environment recognition model can be improved.

[0106] Among them, the environment recognition model is trained by the environment recognition model training method provided by Example 1 and / or Example 2 of the present invention, and has the beneficial effects corresponding to the environment recognition model training method.

[0107] Embodiment 4

[0108] Figure 6 A structural block diagram of an environment recognition model training device provided in Embodiment 4 of the present invention may specifically include the following modules:

[0109] A style image acquisition module 601 is used to acquire a style image containing a specified style, where the specified style is an image style of a preset environment for interrupting the automatic driving mode;

[0110] A first training image set acquisition module 602 is used to input the style image and the original image in the original training image set into an image style transfer network to transfer the specified style to the original image, thereby obtaining a first training image set including the first image;

[0111] A model training module 603 is used to train an environment recognition model using a first training image set;

[0112] A first environment recognition module 604 is used to input the image to be recognized into a pre-trained environment recognition model to obtain a probability that the image to be recognized contains a preset environment, where the preset environment is an environment that interrupts the automatic driving mode;

[0113] A second training image set acquisition module 605, configured to add the second image to the first training image set to obtain a second training image set when the first probability is greater than a preset probability;

[0114] The model updating module 606 is used to update the environment recognition model using the second training image set.

[0115] In an optional embodiment of the present invention, the first training image set acquisition module 602 includes:

[0116] The image input unit is used to input the style image containing the specified style and the original image in the original training image set into the image style transfer network for each specified style.

[0117] In an optional embodiment of the present invention, the first training image set acquisition module 602 further includes:

[0118] An image grouping unit, used for grouping a plurality of original images in an original training image set into a plurality of image groups;

[0119] A target image group determination unit, used for determining, for each specified style, a target image group corresponding to the specified style;

[0120] The first training image set acquisition unit is used to input the style image containing the specified style and the original image in the target image group into the image style transfer network to transfer the specified style to the original image, thereby obtaining a first training image set containing the first image.

[0121] In an optional embodiment of the present invention, the first training image set acquisition module 602 further includes:

[0122] An area of ​​interest clipping unit, used to clip an area of ​​interest associated with a preset environment from an original image in an original training image set to obtain a third image;

[0123] A fourth image acquisition unit, used for inputting the style image and the third image into an image style transfer network to obtain a fourth image;

[0124] The fourth image replacement unit is used to replace the fourth image into the region of interest of the original image to obtain the first image, and use the collection of the first images as the first training image set.

[0125] In an optional embodiment of the present invention, the model training module 603 includes:

[0126] A first image labeling unit, used to label a first image in a first training image set, wherein the label includes a preset environment included in the first image and a second probability of including the preset environment;

[0127] A model initialization unit, used to initialize the environment recognition model;

[0128] A first image input unit, used for randomly extracting the first image input environment recognition model to obtain a third probability that the first image contains a preset environment;

[0129] A loss rate calculation unit, used for calculating the loss rate using the second probability and the third probability;

[0130] A loss rate determination unit, used to determine whether the loss rate is less than a preset threshold value, and if so, to execute the content executed by the environment recognition model acquisition unit, and if not, to execute the content executed by the model parameter adjustment unit;

[0131] An environment recognition model acquisition unit, used to stop training the environment recognition model and obtain a trained environment recognition model;

[0132] The model parameter adjustment unit is used to adjust the model parameters of the environment recognition model by using the loss rate, and return the content executed by the first image input unit.

[0133] In an optional embodiment of the present invention, the second training image set acquisition module 605 includes:

[0134] A second image labeling unit, configured to label the second image when the first probability is greater than a preset probability, wherein the label includes a preset environment included in the second image and a probability of including the preset environment;

[0135] The second training image set acquisition unit is used to add the labeled second image to the first training image set to obtain the second training image set.

[0136] The environment recognition model training device provided in the embodiment of the present invention can execute the environment recognition model training method provided in the first embodiment and / or the second embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0137] Embodiment 5

[0138] Figure 7 This is a structural block diagram of a driving control device provided in Embodiment 5 of the present invention, which may specifically include the following modules:

[0139] The image acquisition module 701 is used to acquire an image to be recognized through a camera when the autonomous driving vehicle is driving in the autonomous driving mode;

[0140] The second environment recognition module 702 is used to input the image to be recognized into a pre-trained environment recognition model to obtain the category and probability that the image to be recognized contains a preset environment, where the preset environment is an environment that interrupts the automatic driving mode;

[0141] The vehicle control module 703 is used to control the autonomous driving vehicle to exit the autonomous driving mode when the probability is greater than a preset probability.

[0142] In an optional embodiment of the present invention, the driving control device further includes:

[0143] The image output module to be identified is used to output the image to be identified to a preset training image set, and the training image set is used to update the environment recognition model.

[0144] Among them, the environment recognition model is trained by the environment recognition model training method provided by Example 1 and / or Example 2 of the present invention, and has the beneficial effects corresponding to the environment recognition model training method.

[0145] The driving control device provided in the embodiment of the present invention can execute the driving control method provided in the third embodiment of the present invention, and also has the functional modules and beneficial effects corresponding to the execution method.

[0146] Embodiment 6

[0147] Figure 8A schematic diagram of the structure of an electronic device 40 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0148] like Figure 8 As shown, the electronic device 40 includes at least one processor 41, and a memory connected to the at least one processor 41, such as a read-only memory (ROM) 42, a random access memory (RAM) 43, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 41 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 42 or the computer program loaded from the storage unit 48 to the random access memory (RAM) 43. In the RAM 43, various programs and data required for the operation of the electronic device 40 can also be stored. The processor 41, the ROM 42, and the RAM 43 are connected to each other through a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.

[0149] A number of components in the electronic device 40 are connected to the I / O interface 45, including: an input unit 46, such as a keyboard, a mouse, etc.; an output unit 47, such as various types of displays, speakers, etc.; a storage unit 48, such as a disk, an optical disk, etc.; and a communication unit 49, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 49 allows the electronic device 40 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0150] The processor 41 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 41 performs the various methods and processes described above, such as the environment recognition model training method, and / or the driving control method.

[0151] In some embodiments, the environment recognition model training method, and / or the driving control method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 48. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 40 via the ROM 42 and / or the communication unit 49. When the computer program is loaded into the RAM 43 and executed by the processor 41, one or more steps of the review task assignment method described above may be performed. Alternatively, in other embodiments, the processor 41 may be configured to execute the above-mentioned environment recognition model training method, and / or the driving control method, in any other appropriate manner (e.g., by means of firmware).

[0152] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0153] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0154] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0155] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0156] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0157] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0158] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0159] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for training an environment recognition model, characterized in that: include: Acquire a style image including a specified style, where the specified style is an image style of a preset environment for interrupting the automatic driving mode; Inputting the style image and the original image in the original training image set into an image style transfer network to transfer the specified style to the original image, thereby obtaining a first training image set including the first image; Using the first training image set to train an environment recognition model; Inputting the collected second image into the environment recognition model to obtain a preset environment contained in the second image and a first probability of containing the preset environment, wherein the second image is an image collected in an actual driving scene; When the first probability is greater than a preset probability, adding the second image to the first training image set to obtain a second training image set; Using the second training image set to update the environment recognition model; Wherein, when the first probability is greater than a preset probability, adding the second image to the first training image set to obtain a second training image set comprises: When the first probability is greater than a preset probability, labeling the second image, the label including a preset environment included in the second image and a probability of including the preset environment; The labeled second images are added to the first training image set to obtain a second training image set, including: after the data amount of the second images reaches a preset data amount, each time a new second image is added, a corresponding number of the first images are randomly eliminated from the second training image set, and the first images are gradually replaced with real second images.

2. The method according to claim 1, characterized in that The step of inputting the style image and the original image in the original training image set into the image style transfer network comprises: For each of the specified styles, a style image containing the specified style and an original image in the original training image set are input into an image style transfer network.

3. The method according to claim 1, characterized in that The step of inputting the style image and the original image in the original training image set into an image style transfer network to transfer the specified style to the original image to obtain a first training image set including the first image comprises: Dividing a plurality of original images in the original training image set into a plurality of image groups; For each of the specified styles, determining a target image group corresponding to the specified style; The style image containing the specified style and the original image in the target image group are input into an image style transfer network to transfer the specified style to the original image, thereby obtaining a first training image set containing a first image.

4. The method according to claim 1, characterized in that: The step of inputting the style image and the original image in the original training image set into an image style transfer network to transfer the specified style to the original image to obtain a first training image set including the first image comprises: Cut out a region of interest associated with the preset environment from an original image in the original training image set to obtain a third image; Inputting the style image and the third image into an image style transfer network to obtain a fourth image; The fourth image is replaced into the region of interest of the original image to obtain a first image, and a collection of the first images is used as a first training image set.

5. The method according to claim 1, characterized in that The step of using the first training image set to train the environment recognition model includes: annotating a label for the first image in the first training image set, wherein the label includes a preset environment included in the first image and a second probability of including the preset environment; Initialize the environment recognition model; randomly extracting the first image and inputting it into the environment recognition model to obtain a third probability that the first image contains the preset environment; Calculating a loss rate using the second probability and the third probability; Determine whether the loss rate is less than a preset threshold; If yes, stop training the environment recognition model to obtain a trained environment recognition model; If not, the loss rate is used to adjust the model parameters of the environment recognition model, and the process returns to the step of randomly extracting the first image and inputting it into the environment recognition model.

6. A driving control method, characterized in that: include: When the autonomous driving vehicle is driving in the autonomous driving mode, an image is collected to obtain an image to be recognized; Inputting the image to be recognized into a pre-trained environment recognition model to obtain a probability that the image to be recognized contains a preset environment, where the preset environment is an environment that interrupts the automatic driving mode; When the probability is greater than a preset probability, controlling the autonomous driving vehicle to exit the autonomous driving mode; Wherein, the environment recognition model is trained by the environment recognition model training method described in any one of claims 1-5.

7. The driving control method according to claim 6, characterized in that: After controlling the autonomous driving vehicle to exit the autonomous driving mode, the method further includes: The image to be recognized is output to a preset training image set, and the training image set is used to update the environment recognition model.

8. An environment recognition model training device, characterized in that: include: A style image acquisition module, used to acquire a style image containing a specified style, where the specified style is an image style of a preset environment for interrupting the automatic driving mode; A first training image set acquisition module, used for inputting the style image and the original image in the original training image set into an image style transfer network to transfer the specified style to the original image, thereby obtaining a first training image set including the first image; A model training module, used for training an environment recognition model using the first training image set; A first environment recognition module, used for inputting a captured second image into the environment recognition model to obtain a preset environment contained in the second image and a first probability of containing the preset environment, wherein the second image is an image captured in an actual driving scene; A second training image set acquisition module, configured to add the second image to the first training image set to obtain a second training image set when the first probability is greater than a preset probability; A model updating module, configured to update the environment recognition model using the second training image set; Wherein, the second training image set acquisition module includes: A second image labeling unit, configured to label the second image when the first probability is greater than a preset probability, wherein the label includes a preset environment included in the second image and a probability of including the preset environment; A second training image set acquisition unit, used for adding the labeled second image to the first training image set to obtain a second training image set; The second training image set acquisition unit is specifically used to randomly eliminate a corresponding number of the first images in the second training image set each time a new second image is added after the data amount of the second image reaches a preset data amount, and gradually replace the first images with real second images.

9. A driving control device, characterized in that: include: An image acquisition module, used to acquire images to be recognized when the autonomous driving vehicle is driving in the autonomous driving mode; A second environment recognition module is used to input the image to be recognized into a pre-trained environment recognition model to obtain a probability that the image to be recognized contains a preset environment, where the preset environment is an environment that interrupts the automatic driving mode; A vehicle control module, configured to control the autonomous driving vehicle to exit the autonomous driving mode when the probability is greater than a preset probability; Wherein, the environment recognition model is trained by the environment recognition model training method described in any one of claims 1-5.

10. An electronic device, characterized in that: The electronic device comprises: one or more processors; a memory for storing one or more computer programs, When the one or more computer programs are executed by the one or more processors, the one or more processors implement the environment recognition model training method as described in any one of claims 1-5, and / or the driving control method as described in claims 6-7.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the environment recognition model training method as described in any one of claims 1-5, and / or the driving control method as described in any one of claims 6-7.

Images