Method, device, equipment and medium for training environmental perception ability of autonomous driving vehicles

Abstract

The embodiments of the present application provide a method, device, equipment and medium for training the environmental perception ability of an autonomous driving vehicle. It can be applied to scenes such as ports, mines, parks, urban transportation, or highways. The method includes: obtaining training data for training environmental perception ability based on pre-collected environmental data and a style model for style transfer of environmental data; inputting the training data into a perception algorithm model for environmental perception, training the perception algorithm model, and obtaining a trained perception algorithm model to realize autonomous driving vehicles, and perceive the environment based on the perception algorithm model. The embodiments of the present application solve the problem that the training data generated in the related technology is quite different from the actual scene and cannot meet the training requirements of the unmanned driving system, and ensure the perception accuracy of the vehicle under extreme weather conditions, thereby ensuring the safety of vehicle driving.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data processing technology, and in particular to a method, device, equipment and medium for training the environmental perception capability of an autonomous driving vehicle. Background Art

[0002] With the emergence of diversified transportation needs, technologies such as autonomous driving and unmanned driving are gradually being more widely used. In order to ensure the safety of unmanned vehicles, a large amount of environmental data in different scenarios is needed as training data to train the environmental perception capabilities of unmanned driving (such as visual perception, position perception, temperature perception, etc.) to ensure the driving safety of unmanned vehicles in different environments (these perception capabilities are usually implemented through corresponding perception algorithm models. The higher the recognition accuracy of the perception algorithm model, the stronger the corresponding perception capability). However, conventionally recorded environmental data is difficult to meet the needs under extreme weather conditions, resulting in actual vehicles experiencing reduced perception accuracy and insufficient safety under extreme weather conditions.

[0003] In the existing technology, training data that simulates different scenarios are generated through methods such as data fitting. However, the generated training data cannot guarantee that the scenario constraints are met (such as correctly representing environmental characteristics such as weather and temperature), and the data restoration degree of the training data is low, and it differs greatly from the actual scenario. As a result, the visual perception ability of the trained unmanned driving system is still poor and cannot meet the training requirements of the unmanned driving system. Summary of the invention

[0004] The embodiments of the present application provide a method, device, equipment and medium for training the environmental perception capability of an autonomous driving vehicle to solve the problem that the training data generated in the prior art is significantly different from the actual scene and cannot meet the training requirements of the unmanned driving system.

[0005] In a first aspect, an embodiment of the present application provides a method for training the environmental perception capability of an autonomous driving vehicle, and the method for training the environmental perception capability of an autonomous driving vehicle includes:

[0006] Based on the pre-collected environmental data and the style model used to transfer the style of the environmental data, training data for training environmental perception capability is obtained;

[0007] The training data is input into a perception algorithm model used for environmental perception, and the perception algorithm model is trained to obtain a trained perception algorithm model to realize autonomous driving vehicles and perceive and process the environment based on the perception algorithm model.

[0008] It can be seen that the pre-collected environmental data and the style model used for style transfer of environmental data are used as training data for training environmental perception capabilities to train the perception algorithm model used by the autonomous driving vehicle to perceive the environment. Therefore, the data used in the perception algorithm model is not generated by the data fitting method, but is obtained by style transfer based on the environmental data collected in the actual environment, which effectively ensures the fit between the obtained training data and the actual scene, and thus ensures the recognition accuracy of the trained perception algorithm model; at the same time, the style model is used to transfer the style of environmental data to achieve customization of the style model, and the same environmental data can be transferred to a variety of scene styles under extreme weather conditions, ensuring the perception accuracy of the vehicle under extreme weather conditions, and thus ensuring the safety of vehicle driving.

[0009] Optionally, the style model is trained in the following manner: obtaining style sample data, reverse style sample data and scene sample data for simulating scenes for training the style model, wherein style-related parameters of the reverse style sample data are opposite to corresponding parameters of the style sample data; inputting the style sample data and the scene sample data into a first adversarial neural network, and outputting target sample data for style transfer based on the style sample data; inputting the reverse style sample data and the target sample data into a second adversarial neural network, and outputting restored sample data corresponding to the target sample data; training the first adversarial neural network and the second neural network based on the deviation between the restored sample data and the scene sample data; and using the trained first adversarial neural network as the style model corresponding to the style sample data.

[0010] It can be seen that by combining the first adversarial neural network and the second adversarial neural network, the training process of the style model is repeatedly verified to ensure that the scene data after style transfer (i.e., target sample data) can be restored to the original state (i.e., restored sample data) to the maximum extent, thereby improving the scene restoration degree of the style model; at the same time, the scene data after style transfer can be restored to the original data, indicating that the data accuracy of the scene data after style transfer is retained, rather than there is serious data loss, thereby maximizing the constraint degree and data accuracy of the style model on the scene during the style transfer process.

[0011] Optionally, obtaining style sample data for training a style model, reverse style sample data, and scene sample data for simulating a scene includes: obtaining style sample data for training a style model and scene sample data for simulating a scene; and determining corresponding reverse style sample data based on the style sample data.

[0012] It can be seen that the first adversarial neural network and the second adversarial neural network are trained respectively through the style sample data and the reverse style sample data, so that the second adversarial neural network can restore the data obtained by the style transfer of the first pair of neural networks to the original state. At the same time, since the style sample data and the reverse style sample data correspond to each other, when training the first adversarial neural network and the second adversarial neural network, the generators in the two neural networks can be trained at the same time, or the verifiers in the two neural networks can be trained at the same time, thereby improving the training efficiency.

[0013] Optionally, the first adversarial neural network includes a first generator and a first verifier, and the style sample data and the scene sample data are input into the first adversarial neural network, and target sample data for style migration based on the style sample data is output, including: based on the scene sample data, extracting the corresponding first image feature data distributed along time; inputting the style sample data and the scene sample data into the first generator, and outputting the target sample data with the first image feature data distributed along time as a constraint condition.

[0014] It can be seen that by using the first image feature distributed along time as a constraint condition, it is possible to use scene sample data to constrain time and space in the style transfer process, ensuring that the target sample data after style transfer and the scene sample data before transfer have the same content features (such as the same object has the same features in the image before and after transfer). In the case where the scene sample data is video data of continuous frames, the scene constraint effect is significantly improved, excessive style transfer is avoided, and the recognition effect of the target sample data after style transfer is guaranteed, thereby ensuring the training effect of the style model and the perception algorithm model.

[0015] Optionally, the second adversarial neural network includes a second generator and a second verifier, which input the reverse style sample data and the target sample data into the second adversarial neural network and output restored sample data corresponding to the target sample data, including: extracting the corresponding second image feature data distributed along time based on the target sample data; inputting the reverse style sample data and the target sample data into the second generator, and outputting the restored sample data with the second image feature data distributed along time as a constraint condition.

[0016] It can be seen that by constraining the second adversarial neural network through the second image feature data distributed along time, similar to the first adversarial neural network, the constraint effect of the second neural network in the style transfer process can be guaranteed, and the recognition effect of the restored sample data obtained after style transfer can be guaranteed, thereby ensuring the training effect of the style model and the perception algorithm model.

[0017] Optionally, based on the deviation between the restored sample data and the scene sample data, the first adversarial neural network and the second neural network are trained, including: based on the target detection algorithm, extracting the features of the scene sample data to obtain the first sample features; based on the target detection algorithm, extracting the features of the restored sample data to obtain the second sample features; based on the degree of deviation between the first sample features and the second sample features, the first adversarial neural network and the second neural network are trained.

[0018] It can be seen that by extracting the features of scene sample data and restored sample data through the target detection algorithm, the process of the vehicle system obtaining environmental data based on sensors in the actual application scenario is simulated, and the similarity between the restored sample data and the scene sample data in the actual application scenario is effectively judged by comparing the extracted features. The higher the similarity, the better the restoration effect and the better the restorability of the target sample data. At the same time, since the target sample data can be accurately restored to the scene sample data, it means that the features that can be extracted by restoring the sample data in the actual application scenario are more consistent with the features of the environmental data extracted in the actual scene, and the restored sample data is obtained based on the style transfer of the target sample data. Therefore, the smaller the deviation between the restored sample data and the scene sample data, the higher the scene authenticity and clarity of the target sample data, which in turn means that the scene authenticity and clarity of the training data obtained by style model migration are higher.

[0019] Optionally, based on the degree of deviation between the first sample feature and the second sample feature, the first adversarial neural network and the second neural network are trained, including: if the degree of deviation between the first sample feature and the second sample feature is within a set range, determining that the first adversarial neural network has passed the training; if the degree of deviation between the first sample feature and the second sample feature is not within the set range, based on the degree of deviation, training the first generator and the second generator, or training the first verifier and the second verifier.

[0020] It can be seen that since the first adversarial neural network and the second neural network are trained by corresponding style sample data and reverse style sample data respectively, their internal parameters are also corresponding. Therefore, during the training process, the first generator and the second generator can be trained at the same time, or the first verifier and the second verifier can be trained at the same time, which can significantly improve the training efficiency while ensuring the training effect.

[0021] In a second aspect, an embodiment of the present application provides an autonomous driving vehicle environment perception ability training device, the autonomous driving vehicle environment perception ability training device comprising:

[0022] A determination module, for obtaining training data for training environmental perception capability based on pre-collected environmental data and a style model for performing style transfer on the environmental data;

[0023] The output module is used to train the training data using a perception algorithm model for environmental perception to obtain a trained perception algorithm model, so as to enable the autonomous driving vehicle to perceive the environment based on the perception algorithm model.

[0024] Optionally, the determination module is specifically used to train a style model in the following manner: obtaining style sample data, reverse style sample data and scene sample data for simulating scenes for training the style model, wherein style-related parameters of the reverse style sample data are opposite to corresponding parameters of the style sample data; inputting the style sample data and the scene sample data into a first adversarial neural network, and outputting target sample data for style migration based on the style sample data; inputting the reverse style sample data and the target sample data into a second adversarial neural network, and outputting restored sample data corresponding to the target sample data; training the first adversarial neural network and the second neural network based on the deviation between the restored sample data and the scene sample data; and using the trained first adversarial neural network as the style model corresponding to the style sample data.

[0025] Optionally, the determination module is specifically used to obtain style sample data for training the style model and scene sample data for simulating scenes; and determine corresponding reverse style sample data based on the style sample data.

[0026] Optionally, the determination module is specifically used to extract corresponding first image feature data distributed along time based on scene sample data; input the style sample data and the scene sample data into the first generator, and output the target sample data with the first image feature data distributed along time as a constraint condition.

[0027] Optionally, the determination module is specifically used to extract corresponding second image feature data distributed along time based on the target sample data; input the reverse style sample data and the target sample data into the second generator, and output the restored sample data with the second image feature data distributed along time as a constraint condition.

[0028] Optionally, the determination module is specifically used to extract features of scene sample data based on a target detection algorithm to obtain a first sample feature; extract features of restored sample data based on a target detection algorithm to obtain a second sample feature; and train a first adversarial neural network and a second neural network based on the degree of deviation between the first sample feature and the second sample feature.

[0029] Optionally, the determination module is specifically used to determine whether the first adversarial neural network has passed training if the degree of deviation between the first sample feature and the second sample feature is within a set range; if the degree of deviation between the first sample feature and the second sample feature is not within the set range, based on the degree of deviation, train the first generator and the second generator, or train the first verifier and the second verifier.

[0030] In a third aspect, an embodiment of the present application further provides a control device, the control device comprising:

[0031] at least one processor;

[0032] and a memory communicatively coupled to the at least one processor;

[0033] Among them, the memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor to enable the control device to execute an autonomous driving vehicle environmental perception capability training method corresponding to any embodiment in the first aspect of the embodiments of the present application.

[0034] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, in which computer execution instructions are stored. When the computer execution instructions are executed by a processor, they are used to implement any autonomous driving vehicle environmental perception capability training method as described in the first aspect of the embodiment of the present application.

[0035] In the fifth aspect, the embodiments of the present application also provide a computer program product, which includes computer execution instructions. When the computer execution instructions are executed by a processor, they are used to implement the environmental perception capability training method of an autonomous driving vehicle of any embodiment corresponding to the first aspect of the embodiments of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 An application scenario diagram of the method for training the environmental perception capability of an autonomous driving vehicle provided in an embodiment of the present application;

[0037] Figure 2 A flowchart of a method for training the environmental perception capability of an autonomous driving vehicle provided in accordance with one embodiment of the present application;

[0038] Figure 3a A flowchart of a method for training the environmental perception capability of an autonomous driving vehicle provided in yet another embodiment of the present application;

[0039] Figure 3b for Figure 3a A flow chart of a method for obtaining scene sample data and reverse style sample data provided in the illustrated embodiment;

[0040] Figure 3c for Figure 3aA flowchart of a specific method for generating target sample data provided in the illustrated embodiment;

[0041] Figure 3d for Figure 3a A flowchart of a specific method for generating restored sample data provided in the illustrated embodiment;

[0042] Figure 3e for Figure 3a A flowchart of a training method for a first adversarial neural network and a second adversarial neural network provided in the illustrated embodiment;

[0043] Figure 3f for Figure 3a A flowchart of training an adversarial neural network based on sample features provided in the illustrated embodiment;

[0044] Figure 4 A schematic diagram of the structure of a device for training the environmental perception ability of an autonomous driving vehicle provided in yet another embodiment of the present application;

[0045] Figure 5 A schematic diagram of the structure of a control device provided in yet another embodiment of the present application. DETAILED DESCRIPTION

[0046] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the appended claims.

[0047] The following specific embodiments are used to describe in detail the technical solutions of the embodiments of the present application and how the technical solutions of the embodiments of the present application solve the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0048] With the emergence of diversified transportation needs, technologies such as autonomous driving and unmanned driving are gradually being more widely used. Unmanned vehicles will detect the surrounding environmental data through various sensors while driving, and process these environmental data through pre-trained perception algorithm models to achieve the positioning of themselves and surrounding objects and the judgment of environmental conditions (such as weather, road conditions, etc.), and then adjust their own driving status to achieve safe driving.

[0049] Therefore, in order to ensure the safety of driverless vehicles, a large amount of environmental data in different scenarios is needed as training data to train the environmental perception capabilities of driverless vehicles (such as visual perception, position perception, temperature perception, etc.) to ensure the driving safety of driverless vehicles in different environments (these perception capabilities are usually implemented through corresponding perception algorithm models. The higher the recognition accuracy of the perception algorithm model, the stronger the corresponding perception capability). However, conventionally recorded environmental data can hardly meet the needs under extreme weather conditions, resulting in reduced perception accuracy and insufficient safety of actual vehicles under extreme weather conditions.

[0050] In the prior art, training data simulating different scenarios are generated through methods such as data fitting. However, the generated training data cannot guarantee that the scenario constraints are met (such as correctly representing environmental features such as weather and temperature, or the shape of the same object corresponding to different frames of video data should not change suddenly). In addition, the training data has a low degree of data restoration (indicating that the clarity is poor and too much information is lost during the generation process compared to the original data), which is quite different from the actual scenario. As a result, the visual perception ability of the trained unmanned driving system is still poor and cannot meet the training requirements of the unmanned driving system.

[0051] In order to solve the above problems, an embodiment of the present application provides a method for training the environmental perception capability of an autonomous driving vehicle, which uses a pre-trained style model to transfer the style of actually collected environmental data to obtain training data, so as to train a perception algorithm model, thereby ensuring the scene authenticity and accuracy of the training data, and further ensuring the reliability of the trained perception algorithm model and the safety of the unmanned vehicle.

[0052] Figure 1 This is an application scenario diagram of the method for training the environmental perception capability of an autonomous driving vehicle provided in an embodiment of the present application. Figure 1 As shown, in the process of training the environmental perception capability of an autonomous driving vehicle, the server 100 generates training data 140 based on the environmental data 120 pre-collected by the sensor 110, combined with the required style data 121 and the pre-trained style model 130, and inputs the training data into the perception model 150 for training, thereby obtaining a trained perception model.

[0053] It should be noted that Figure 1 In the scenario shown, only one server, sensor, environmental data, style data, style model, training data, and perception model is used as an example for illustration, but the embodiments of the present application are not limited to this. That is, the number of servers, sensors, environmental data, style data, style model, training data, and perception models can be arbitrary.

[0054] The following describes in detail the autonomous driving vehicle environment perception ability training method provided by the present application through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0055] Figure 2 This is a flow chart of a method for training the environmental perception capability of an autonomous driving vehicle provided in one embodiment of the present application. Figure 2 As shown, including but not limited to the following steps:

[0056] Step S201: obtaining training data for training environmental perception capability based on pre-collected environmental data and a style model for performing style transfer on the environmental data.

[0057] Specifically, the pre-collected environmental data is data collected in a real environment in advance by sensors, rather than simulated data generated by 3D engine simulation rendering, simulation tool fitting, etc., thereby effectively avoiding the problem of large differences between simulated data and real data in the prior art.

[0058] The style model is an adversarial neural network for style transfer that is trained based on time and space constraints to ensure that when style transfer is performed through the style model, the training data after transfer and the environment data before transfer meet the time and space constraints. The time and space constraints can be that the number of frames (or duration) corresponding to the training data and the environment data are the same, the image structure features of the corresponding object in each frame do not mutate, and the color difference change of adjacent areas in the same frame does not exceed the set threshold.

[0059] There are usually multiple style models at the same time. Each style model corresponds to a scene style according to the different scene styles that need to be generated. By using style models corresponding to different scene styles to transfer the style of environmental data, it is fully guaranteed that the training data can cover different types of extreme weather environments, and the processing ability of the perception algorithm model in extreme weather environments is maximized.

[0060] The style model can be used to transfer the style of environmental data to obtain target environmental data of the corresponding style. At the same time, data migration is performed using environmental data collected in the real environment. Since the accuracy, scene authenticity, and clarity of the environmental data obtained in this way can be optimized, the accuracy, scene authenticity, and clarity of the migrated training data can be maximized to ensure the training effect when training the perception algorithm model.

[0061] Step S202: input the training data into a perception algorithm model for environmental perception, train the perception algorithm model, and obtain a trained perception algorithm model to enable an autonomous driving vehicle to perceive the environment based on the perception algorithm model.

[0062] Specifically, by transferring the training data from the environmental data style, it is possible to train the perception algorithm model to have recognition and perception capabilities in different extreme weather environments, so that autonomous driving vehicles can ensure perception accuracy in different extreme weather environments, thereby effectively ensuring the driving safety of autonomous driving vehicles in different environments.

[0063] In some embodiments, the autonomous driving vehicle environment perception capability training method described in step S201 to step S202 can be deployed to a cloud server and connected to a server platform corresponding to the autonomous driving vehicle. At this time, the mass production of the training data set can be achieved, thereby efficiently completing the training of the perception model. It can also achieve customized development of the data set, thereby forming a fully automatic production line.

[0064] The method for training the environmental perception capability of an autonomous driving vehicle provided in an embodiment of the present application uses pre-collected environmental data and a style model for style transfer of environmental data as training data for training environmental perception capability, so as to train the perception algorithm model used by the autonomous driving vehicle for perceiving and processing the environment. As a result, the data used in the perception algorithm model is not generated by a data fitting method, but is obtained by style transfer based on environmental data collected in the actual environment, which effectively ensures the fit between the obtained training data and the actual scene, thereby ensuring the recognition accuracy of the trained perception algorithm model; at the same time, by transferring the style of environmental data through a style model, the style model can be customized, and the same environmental data can be transferred to a variety of scene styles under extreme weather conditions, thereby ensuring the perception accuracy of the vehicle under extreme weather conditions, thereby ensuring the safety of vehicle driving.

[0065] Figure 3a A flowchart of a method for training the environmental perception capability of an autonomous driving vehicle provided in an embodiment of the present application. Figure 3a As shown, the autonomous driving vehicle environment perception capability training provided in this embodiment includes the following steps:

[0066] Step S301: Acquire style sample data for training a style model, reverse style sample data, and scene sample data for simulating a scene.

[0067] The style-related parameters of the reverse style sample data are opposite to the corresponding parameters of the style sample data.

[0068] Specifically, the style model is obtained based on an adversarial neural network. The adversarial neural network includes both a generator and a verifier. After the generator completes the style transfer, the verifier is used to ensure that the style-transferred data meets the requirements, and then the style-transferred data is output.

[0069] Although the verifier can be used to verify the effect of style transfer, a single adversarial neural network cannot guarantee the accuracy and clarity of the output data after style transfer, because the accuracy and clarity corresponding to different input data are different. Therefore, in this scheme, on the basis of a single adversarial neural network, an adversarial neural network for style restoration (i.e., the second adversarial neural network) corresponding to the adversarial neural network used for style transfer (i.e., the first adversarial neural network) is set up. If the data after style transfer can be restored to the data of the style before transfer through the second adversarial neural network, and the accuracy and clarity of the restored data are as close as possible to the original data (ideally, the accuracy and clarity of the restored data will be the same as the original data), then it can be considered that the accuracy and clarity of the data after style transfer can be guaranteed to the maximum extent.

[0070] Since the first adversarial neural network is used to transfer the style of data, and the second adversarial neural network is used to restore the style of the style-transferred data in the opposite direction, the styles of the two are relative, so it is necessary to prepare in advance the style sample data for training the first adversarial neural network and the reverse style sample data for training the second adversarial neural network to train the style transfer features of the two adversarial neural networks.

[0071] At the same time, environmental sample data is also needed for training and testing to verify the effects of style transfer and style restoration.

[0072] In some embodiments, such as Figure 3b As shown, it is a flow chart of a method for obtaining scene sample data and reverse style sample data, which includes the following steps:

[0073] Step S3011: Acquire style sample data for training a style model and scene sample data for simulating a scene.

[0074] Specifically, the style sample data may be selected from existing style transfer training data, and usually may also be style transfer training data of style features corresponding to the extreme weather environment to be applied, to ensure that the obtained style model can achieve style transfer of style features corresponding to the extreme weather environment.

[0075] The scene sample data can be selected from the scene data in the actual driving environment collected in advance to maximize the fit with the actual situation.

[0076] Step S3012: Determine corresponding reverse style sample data based on the style sample data.

[0077] Specifically, since the second adversarial neural network corresponding to the inverse style sample data needs to restore the style of the data after the style transfer of the first adversarial neural network based on the style sample data, the inverse style sample data needs to correspond to the style sample data and be obtained by reverse adjustment.

[0078] The specific adjustment method needs to be determined according to the type of style sample data. For example, if the style sample data is a 20-unit increase in color temperature, then the corresponding inverse style sample data is a 20-unit decrease in color temperature. Or if the style sample data is a 10% decrease in contrast, then the corresponding inverse style sample data is an effect of removing the 10% decrease in contrast (because the original contrast will not be obtained if the contrast is increased by 10% after a 10% decrease in contrast. Therefore, here, the description is simply to remove the decrease effect. In actual applications, different implementation methods in existing image processing technologies can be used, and this is not limited here).

[0079] Step S302: input the style sample data and the scene sample data into a first adversarial neural network, and output target sample data for style transfer based on the style sample data.

[0080] Among them, the first adversarial neural network includes a first generator and a first verifier.

[0081] Specifically, the data that need to be input into the corresponding neural network include style data corresponding to the style expected by style transfer (i.e., style sample data) and scene data that need to be style transferred (i.e., scene sample data), and the generator outputs the migrated scene data after style transfer (i.e., target sample data), and the verifier verifies the migrated scene data based on the style data and scene data. If it is judged that the style features of the migrated scene data (such as hue, contrast, brightness, etc.) correspond to the style data and scene features, it can be considered that the migrated scene data obtained by this style transfer is usable.

[0082] In some embodiments, such as Figure 3c As shown, it is a flow chart of a specific method for generating target sample data, which includes the following steps:

[0083] Step S3021: based on the scene sample data, extract the corresponding first image feature data distributed along time.

[0084] Specifically, in addition to using style sample data and scene sample data to verify the migrated scene data, this solution also needs to extract features from the scene sample data as constraints to further constrain the results of style migration, so as to avoid the loss of accuracy and clarity of the environmental data during the style migration process.

[0085] The features extracted from the scene sample data, i.e., the first image features distributed along time, include the time features of the scene sample data, such as the total duration (number of frames), the image content structure features in consecutive frames (such as the shape structure, area, coordinates, and color difference with the surrounding area of ​​certain objects in the corresponding image), and the changing trends of these image content structure features (such as the coordinate changing trend and area changing trend of certain structures in the corresponding image).

[0086] Step S3022: input the style sample data and the scene sample data into the first generator, and output the target sample data with the first image feature data distributed along time as a constraint condition.

[0087] Specifically, since the image content structural features in consecutive frames of the collected scene sample data should remain fixed or change continuously, rather than undergo sudden changes (such as the structure in the middle of an image usually does not suddenly disappear or suddenly appear in two adjacent frames), the corresponding image content structural features after style transfer should also maintain the corresponding trend (i.e., remain fixed or change continuously). By extracting these first image feature data distributed along time as constraints, the data generated by the first generator (i.e., the target sample data before verification by the first verifier) ​​is constrained, thereby effectively ensuring the stability of the image content during the style transfer process and avoiding problems such as data loss, precision loss, and clarity loss.

[0088] Step S303: input the reverse style sample data and the target sample data into the second adversarial neural network, and output the restored sample data corresponding to the target sample data.

[0089] The second adversarial neural network includes a second generator and a second verifier.

[0090] Specifically, similar to the first adversarial neural network, the second adversarial neural network also needs to input corresponding style data (reverse style sample data) and scene data (target sample data) to obtain scene data after style transfer (restored sample data).

[0091] Since the second adversarial neural network performs style restoration processing after the first adversarial neural network performs style transfer on the scene sample data, the second adversarial neural network will directly use the style transfer result (i.e., target sample data) output by the first adversarial neural network as input scene data, and use predetermined reverse style sample data as style data to ensure that the output restored sample data can be restored to the scene sample data to the greatest extent (i.e., to ensure the restorability of the output target sample data) and to ensure the accuracy of its restored sample data.

[0092] In some embodiments, such as Figure 3d As shown, it is a flowchart of a specific method for generating restored sample data, which includes the following steps:

[0093] Step S3031: based on the target sample data, extract the corresponding second image feature data distributed along time.

[0094] Specifically, unlike extracting the first image feature data distributed along time from the scene sample data, the constraints in the second adversarial neural network are constraints on the style restoration process from the target sample data to the restored sample data. Therefore, it is necessary to extract the second image feature data distributed along time based on the target sample data. The type of the second image feature data is the same as that of the first image feature data to ensure the correspondence of the style transfer and style restoration processes, thereby maximizing the restoration degree of the restored sample data.

[0095] Step S3032: input the reverse style sample data and the target sample data into the second generator, and output the restored sample data with the second image feature data distributed along time as a constraint condition.

[0096] Specifically, the process of performing style restoration on the target sample data based on the reverse style sample data by the second generator corresponds to the process in step S3022, which will not be described in detail here.

[0097] Step S304: training the first adversarial neural network and the second neural network based on the deviation between the restored sample data and the scene sample data.

[0098] Specifically, the goal of training the first adversarial neural network and the second adversarial neural network is to minimize the deviation between the restored sample data and the scene sample data, that is, the target sample data obtained by style transfer using the first adversarial neural network can be restored to the original state of the scene sample data to the greatest extent. Ideally, the deviation between the restored sample data and the scene sample data is 0, and at this time, there will be no loss of accuracy and clarity in the process of style transfer using the first adversarial neural network, thus maximizing the reliability of the training of the perception algorithm model.

[0099] In some embodiments, Figure 3e As shown, it is a flow chart of a training method for a first adversarial neural network and a second adversarial neural network, which includes the following steps:

[0100] Step S3041: Based on the target detection algorithm, extract the features of the scene sample data to obtain the first sample features.

[0101] Specifically, in actual applications, the perception algorithm model configured in the vehicle control unit of the autonomous vehicle will be used to extract features from the surrounding environment data (that is, scene data) obtained by the sensor to control the positioning and driving status of the autonomous vehicle. Different types of scene data use different perception algorithm models. When the scene data is continuously collected video / image data, or distance detection data (such as infrared detection, radar sensor detection data), the perception algorithm model can be implemented using a target detection algorithm, so the target detection algorithm is used for description here.

[0102] During the training phase, the target detection algorithm is also directly used to perform target detection on the scene sample data and the restored sample data to extract the detected target features (ie, the first sample features and the second sample features), and the deviation between the scene sample data and the restored sample data is evaluated accordingly.

[0103] Step S3042: Based on the target detection algorithm, extract and restore the features of the sample data to obtain the second sample features.

[0104] Specifically, the process of obtaining the second sample feature is the same as that of the first sample feature, which will not be repeated here.

[0105] Step S3043: training the first adversarial neural network and the second neural network based on the degree of deviation between the first sample feature and the second sample feature.

[0106] Specifically, by comparing the difference between the first sample feature and the second sample feature, the first adversarial neural network and the second neural network can be trained with the goal of minimizing the difference.

[0107] The difference between the first sample feature and the second sample feature may be the difference in features such as the number, coordinates, and size of targets detected by the target detection algorithm. These differences may be used to evaluate the degree of deviation by corresponding comparison or difference.

[0108] Furthermore, if Figure 3f As shown in FIG. 1 , it is a flowchart of training an adversarial neural network based on sample features, and the steps specifically include:

[0109] Step A1: If the degree of deviation between the first sample feature and the second sample feature is within a set range, it is determined that the first adversarial neural network has passed the training.

[0110] Specifically, when the degree of deviation between the first sample feature and the second sample feature is within a set range, it means that the similarity between the restored sample data and the scene sample data meets the requirements. The target sample data after style transfer through the first adversarial neural network can achieve maximum style restoration, and ensure that data loss in the process of style transfer and style restoration is minimized, and data accuracy and restoration are maximized.

[0111] At this point, it can be determined that the first adversarial neural network and the second adversarial neural network meet the requirements. Since only the first adversarial neural network is actually needed for style transfer, it can be directly determined that the first adversarial neural network can pass the training.

[0112] Step A2: If the degree of deviation between the first sample feature and the second sample feature is not within a set range, based on the degree of deviation, the first generator and the second generator are trained, or the first verifier and the second verifier are trained.

[0113] Specifically, when the degree of deviation is not within the set range, it is necessary to train the first adversarial neural network and the second adversarial neural network. During the training process, since the first adversarial neural network and the second adversarial neural network are trained by corresponding style sample data and reverse style sample data, the generators in the first adversarial neural network and the second adversarial neural network can be trained at the same time (the verifier is fixed at this time), or the verifier is trained (the generator is fixed at this time), instead of only fixing the generator or verifier in one adversarial neural network, thereby improving the training efficiency and ensuring the training effect.

[0114] Step S305: Use the trained first adversarial neural network as the style model corresponding to the style sample data.

[0115] Specifically, since only the style transfer function is needed in practical applications, and the style restoration function is not needed, the style model of style transfer in practical applications is the trained first adversarial neural network. The second adversarial neural network is only used for training and is no longer retained after training.

[0116] Step S306: input the training data into a perception algorithm model used for environmental perception, train the perception algorithm model, and obtain a trained perception algorithm model to realize an autonomous driving vehicle and perceive and process the environment based on the perception algorithm model.

[0117] Specifically, this step is Figure 2 The content of step S202 in the illustrated embodiment is the same and will not be repeated here.

[0118] The method for training the environmental perception capability of an autonomous driving vehicle provided in the embodiment of the present application obtains style sample data, reverse style sample data and scene sample data for training a style model, then inputs the style sample data and scene sample data into a first adversarial neural network, outputs target sample data for style transfer based on the style sample data, then inputs the reverse style sample data and the target sample data into a second adversarial neural network, outputs restored sample data corresponding to the target sample data, and then trains the first adversarial neural network and the second neural network based on the deviation between the restored sample data and the scene sample data, and finally uses the trained first adversarial neural network as the style model corresponding to the style sample data. Thus, by improving the network structure of the model used for style transfer, the clarity of the generated target sample data is improved, the target restoration degree of the target sample data is improved, and the harshness of the scene sample data is reduced. At the same time, customization can be achieved according to the selected style sample data, and the customized style model is saved according to the style type, so as to provide the maximum constraint of the scene when training and evaluating the perception model, and more scientifically and effectively evaluate the robustness of the perception algorithm in different scenarios. By adding image feature data distributed along time as constraints, the physical environment scene is effectively guaranteed to remain basically unchanged during the style transfer process. The effect is better than not taking continuous frames to constrain the time and space of the data, and the effect is more obvious for continuous frame data. At the same time, by matching with the second adversarial neural network, the restoration and clarity of the style transferred data are significantly improved.

[0119] Figure 4 A schematic diagram of the structure of an autonomous driving vehicle environment perception ability training device provided in an embodiment of the present application. Figure 4 As shown, the autonomous driving vehicle environment perception ability training device 400 includes: a determination module 410 and an output module 420. Among them:

[0120] A determination module 410, configured to obtain training data for training environmental perception capability based on pre-collected environmental data and a style model for performing style transfer on the environmental data;

[0121] The output module 420 is used to adopt the perception algorithm model for environmental perception to train the training data to obtain a trained perception algorithm model, so as to realize the autonomous driving vehicle to perceive the environment based on the perception algorithm model.

[0122] Optionally, the determination module 410 is specifically used to train a style model in the following manner: obtain style sample data, reverse style sample data and scene sample data for simulating scenes for training the style model, wherein style-related parameters of the reverse style sample data are opposite to corresponding parameters of the style sample data; input the style sample data and the scene sample data into a first adversarial neural network, and output target sample data for style transfer based on the style sample data; input the reverse style sample data and the target sample data into a second adversarial neural network, and output restored sample data corresponding to the target sample data; train the first adversarial neural network and the second neural network based on the deviation between the restored sample data and the scene sample data; and use the trained first adversarial neural network as the style model corresponding to the style sample data.

[0123] Optionally, the determination module 410 is specifically configured to obtain style sample data for training a style model and scene sample data for simulating a scene; and determine corresponding reverse style sample data based on the style sample data.

[0124] Optionally, the determination module 410 is specifically used to extract corresponding first image feature data distributed along time based on the scene sample data; input the style sample data and the scene sample data into the first generator, and output the target sample data with the first image feature data distributed along time as a constraint condition.

[0125] Optionally, the determination module 410 is specifically used to extract corresponding second image feature data distributed along time based on the target sample data; input the reverse style sample data and the target sample data into the second generator, and output the restored sample data with the second image feature data distributed along time as a constraint condition.

[0126] Optionally, the determination module 410 is specifically used to extract features of scene sample data based on a target detection algorithm to obtain a first sample feature; extract features of restored sample data based on a target detection algorithm to obtain a second sample feature; and train a first adversarial neural network and a second neural network based on the degree of deviation between the first sample feature and the second sample feature.

[0127] Optionally, the determination module 410 is specifically used to determine whether the first adversarial neural network has passed the training if the degree of deviation between the first sample feature and the second sample feature is within a set range; if the degree of deviation between the first sample feature and the second sample feature is not within the set range, based on the degree of deviation, train the first generator and the second generator, or train the first verifier and the second verifier.

[0128] In this embodiment, the autonomous driving vehicle environmental perception ability training device can solve the problem that the training data generated in the prior art is quite different from the actual scene and cannot meet the training requirements of the unmanned driving system through the combination of various modules, thereby ensuring the vehicle's perception accuracy under extreme weather conditions and further ensuring the safety of vehicle driving.

[0129] Figure 5 A schematic diagram of a control device provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, the control device 500 includes: a memory 510 and a processor 520 .

[0130] The memory 510 stores a computer program that can be executed by at least one processor 520. The computer program is executed by at least one processor 520 to enable the control device to implement the method for training the environmental perception capability of an autonomous driving vehicle provided in any of the above embodiments.

[0131] The memory 510 and the processor 520 may be connected via a bus 530 .

[0132] The relevant instructions can be understood by referring to the relevant descriptions and effects corresponding to the method embodiments, which will not be repeated here.

[0133] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and the computer program is executed by a processor to implement the following Figures 2 to 3a A method for training the environmental perception capability of an autonomous driving vehicle according to any corresponding embodiment.

[0134] Among them, the computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0135] An embodiment of the present application provides a computer program product, which includes computer-executable instructions, which are used to implement the following when the computer-executable instructions are executed by a processor: Figures 2 to 3a A method for training the environmental perception capability of an autonomous driving vehicle according to any corresponding embodiment.

[0136] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0137] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the disclosure disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope of the present application is indicated by the claims.

[0138] It should be understood that the present application is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.

Claims

1. A method for training the environmental perception ability of an autonomous driving vehicle, characterized in that: include: Based on the pre-collected environmental data and the style model used to transfer the style of the environmental data, training data for training environmental perception capability is obtained; Inputting the training data into a perception algorithm model for environmental perception, training the perception algorithm model, and obtaining a trained perception algorithm model, so as to enable the autonomous driving vehicle to perceive the environment based on the perception algorithm model; The style model is trained in the following way: Acquire style sample data for training a style model, reverse style sample data, and scene sample data for simulating scenes, wherein style-related parameters of the reverse style sample data are opposite to corresponding parameters of the style sample data; Inputting the style sample data and the scene sample data into a first adversarial neural network, and outputting target sample data for style migration based on the style sample data; Inputting the reverse style sample data and the target sample data into a second adversarial neural network, and outputting restored sample data corresponding to the target sample data; Based on the deviation between the restored sample data and the scene sample data, training the first adversarial neural network and the second neural network; The trained first adversarial neural network is used as the style model corresponding to the style sample data.

2. The method for training the environmental perception ability of an autonomous driving vehicle according to claim 1, characterized in that: The acquiring of style sample data for training a style model, reverse style sample data, and scene sample data for simulating scenes includes: Acquire style sample data for training a style model and scene sample data for simulating scenes; Corresponding inverse style sample data is determined based on the style sample data.

3. The method for training the environmental perception ability of an autonomous driving vehicle according to claim 1, characterized in that: The first adversarial neural network includes a first generator and a first verifier, The step of inputting the style sample data and the scene sample data into a first adversarial neural network and outputting target sample data for style migration based on the style sample data comprises: Based on the scene sample data, extracting corresponding first image feature data distributed along time; The style sample data and the scene sample data are input into the first generator, and the target sample data is output with the first image feature data distributed along time as a constraint condition.

4. The method for training the environmental perception ability of an autonomous driving vehicle according to claim 3, characterized in that: The second adversarial neural network includes a second generator and a second verifier, The step of inputting the reverse style sample data and the target sample data into a second adversarial neural network and outputting restored sample data corresponding to the target sample data comprises: Based on the target sample data, extracting corresponding second image feature data distributed along time; The reverse style sample data and the target sample data are input into the second generator, and the restored sample data is output with the second image feature data distributed along time as a constraint condition.

5. The method for training the environmental perception ability of an autonomous driving vehicle according to claim 4, characterized in that: The training of the first adversarial neural network and the second neural network based on the deviation between the restored sample data and the scene sample data includes: Based on the target detection algorithm, extract the features of the scene sample data to obtain a first sample feature; Based on the target detection algorithm, extract the features of the restored sample data to obtain a second sample feature; The first adversarial neural network and the second neural network are trained based on the degree of deviation between the first sample feature and the second sample feature.

6. The method for training the environmental perception ability of an autonomous driving vehicle according to claim 5, characterized in that: The training of the first adversarial neural network and the second neural network based on the degree of deviation between the first sample feature and the second sample feature includes: If the degree of deviation between the first sample feature and the second sample feature is within a set range, determining that the first adversarial neural network passes the training; If the degree of deviation between the first sample feature and the second sample feature is not within a set range, based on the degree of deviation, the first generator and the second generator are trained, or the first verifier and the second verifier are trained.

7. A device for training the environmental perception ability of an autonomous driving vehicle, characterized in that: include: A determination module, for obtaining training data for training environmental perception capability based on pre-collected environmental data and a style model for performing style transfer on the environmental data; An output module, used to input the training data into a perception algorithm model for environmental perception, train the perception algorithm model, and obtain a trained perception algorithm model, so as to enable the autonomous driving vehicle to perceive the environment based on the perception algorithm model; The style model is trained in the following way: Acquire style sample data for training a style model, reverse style sample data, and scene sample data for simulating scenes, wherein style-related parameters of the reverse style sample data are opposite to corresponding parameters of the style sample data; Inputting the style sample data and the scene sample data into a first adversarial neural network, and outputting target sample data for style migration based on the style sample data; Inputting the reverse style sample data and the target sample data into a second adversarial neural network, and outputting restored sample data corresponding to the target sample data; Based on the deviation between the restored sample data and the scene sample data, training the first adversarial neural network and the second neural network; The trained first adversarial neural network is used as the style model corresponding to the style sample data.

8. A control device, characterized in that: include: at least one processor; and a memory communicatively coupled to the at least one processor; In which, the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the control device executes the autonomous driving vehicle environmental perception capability training method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the environmental perception capability training method for an autonomous driving vehicle as described in any one of claims 1 to 6.

Images