Scene generation method, model testing method, and model training method

Abstract

The application discloses a scene generation method, a model test method and a model training method. An original training scene is selected as an initial scene from an original training scene set corresponding to a target model, a first loss function of the initial scene under the target model is determined, a first gradient of the first loss function to the original initial scene is determined, a first scene is determined according to the first gradient and the initial scene, a second loss function of the first scene under the target model is determined, and then the first scene is determined as a target scene when the second loss function meets a preset first condition. According to the embodiment of the application, a new target scene can be generated based on the original training scene set of the model, and scene sampling from an actual environment is not needed, so that the time required for obtaining the scene is shortened, the consumption of manpower and material resources is reduced, and the gradient information and the loss function information in the target model can be used to more specifically generate new scene data.

Description

Technical Field

[0001] This application belongs to the field of autonomous driving, and in particular relates to a scene generation method, a model testing method, and a model training method. Background Technology

[0002] Typically, autonomous driving systems consist of a localization module, a mapping module, a perception module, and a control module. The perception module primarily utilizes deep learning models to perceive and recognize images transmitted from high-definition cameras within the autonomous driving system. In industry, several mainstream deep learning models (such as PoseNet, Faster R-CNN, YOLO, and VGG) have been used in the perception modules of autonomous driving systems for tasks such as pose perception, image semantic segmentation, traffic light detection, and detection of environmental objects (pedestrians / bicycles / cars / trucks / buses / other objects).

[0003] Currently, when training the above models, it is usually necessary to perform large-scale scene sampling from the real environment, and then use the sampled scenes to train the model.

[0004] However, large-scale scene sampling in real-world environments involves collaborative work among multiple parties, requiring advance preparation of sampling vehicles, road test permits, and other resources. This inevitably prolongs sampling time and consumes significant manpower and resources. Therefore, a new method for acquiring scene data is urgently needed. Summary of the Invention

[0005] This application provides a scene generation method, a model testing method, and a model training method. New target scenes can be generated based on the original training scene set of the model, eliminating the need to sample scenes from the actual environment, thus shortening the time required to obtain scenes and reducing the consumption of manpower and material resources.

[0006] In a first aspect, embodiments of this application provide a scene generation method, including:

[0007] Select an original training scene from the original training scene set of the target model as the initial scene, where the target model is the model trained based on the original training scene set;

[0008] Determine the first loss function for the initial scene under the target model;

[0009] Determine the first gradient of the first loss function with respect to the initial scene;

[0010] Based on the first gradient and the initial scene, generate the first scene corresponding to the initial scene;

[0011] Determine the second loss function for the first scenario under the target model;

[0012] Determine whether the second loss function satisfies the preset first condition;

[0013] If the second loss function is determined to satisfy the first condition, then the first scenario is determined as the target scenario.

[0014] As one possible implementation, after determining the first scene as the target scene, the method further includes:

[0015] Determine the first time interval from selecting the original training scene as the initial scene to determining the first scene as the target scene;

[0016] The first duration is taken as the generation duration corresponding to the target scene.

[0017] As one possible implementation, after determining the first scene as the target scene, the method further includes:

[0018] Return to the step of selecting an original training scene from the original training scene set of the target model as the initial scene.

[0019] As one possible implementation, the method further includes:

[0020] If it is determined that the second loss function does not satisfy the first condition, then the first scenario is taken as the initial scenario, and the process returns to the step of determining the first loss function of the initial scenario under the target model.

[0021] As one possible implementation, based on the first gradient and the initial scene, a first scene corresponding to the initial scene is generated, including:

[0022] Calculate the product of the first gradient and the first coefficient;

[0023] The sum of the initial scene and the product above is taken as the first scene corresponding to the initial scene.

[0024] As one possible implementation, determining whether the second loss function satisfies a preset first condition includes:

[0025] Calculate the difference between the second loss function and the first loss function;

[0026] Compare the difference with the first threshold;

[0027] If the difference is found to be greater than or equal to the first threshold, then the second loss function is determined to satisfy the preset first condition;

[0028] If the difference is less than the first threshold, then the second loss function is determined not to meet the first condition.

[0029] As one possible implementation, after selecting the original training scene from the original training scene set of the target model as the initial scene, the method also includes:

[0030] Remove the selected original training scene from the original training scene set;

[0031] Before the step of selecting an original training scene as the initial scene from the original training scene set of the target model, the method further includes:

[0032] Determine whether the original training scene set is an empty set;

[0033] Select original training scenes as initial scenes from the original training scene set of the target model, including:

[0034] If the original training scene set is not empty, select an original training scene from the original training scene set of the target model as the initial scene.

[0035] As one possible implementation, after determining that the second loss function does not satisfy the first condition, and before using the first scene as the initial scene, the method further includes:

[0036] Increment the number of iterations corresponding to the selected original training scene by 1, and determine the second time from selecting the original training scene as the initial scene to determining that the second loss function does not satisfy the first condition;

[0037] Determine if the number of iterations is less than the maximum number of iterations;

[0038] Determine if the second duration is less than the maximum timing time;

[0039] Using the first scene as the initial scene, including:

[0040] If the number of iterations is less than the maximum number of iterations and the second duration is less than the maximum timing time, the first scene will be used as the initial scene.

[0041] Secondly, embodiments of this application also provide a model testing method, including:

[0042] Generate a set of target scenarios and a set of scenario time corresponding to the target model, wherein the set of target scenarios includes target scenarios generated based on the scenario generation method of any one of the first aspects, and the set of scenario time includes the generation time corresponding to the target scenarios;

[0043] The target model is tested using the target scene set and the scene duration set to obtain the test results corresponding to the target model.

[0044] As one possible implementation, the target model is tested using a set of target scenarios and a set of scenario durations to obtain the test results corresponding to the target model, including:

[0045] Based on the target scenario set and scenario duration set, calculate the test indicators corresponding to the target model according to the preset test indicator calculation formula;

[0046] Use the test metrics as the test results for the target model;

[0047] The formula for calculating the test index is:

[0048]

[0049] Where P represents the test metric, s represents the target scenario, and t represents the test metric. s This indicates the generation time corresponding to the target scene s. Represents the set of target scenarios. T represents the number of target scenes contained in the target scene set. max This represents the set of times taken for a scene. The original training scenario set contains the number of original training scenarios, α represents the time importance, β represents the success rate importance, α and β are constants, and α+β=1, α>0, β>0, ∈ is a minimal constant.

[0050] Thirdly, embodiments of this application also provide a model training method, including:

[0051] Based on the scene generation method described in any one of the first aspects, generate the target scene corresponding to the target model;

[0052] Add the target scene to the training set;

[0053] Using the training set, the target model is trained by gradient descent until the loss function of the target model converges on the preset validation set.

[0054] Fourthly, embodiments of this application also provide a scene generation apparatus, the apparatus comprising:

[0055] The selection unit is used to select an original training scene as an initial scene from the original training scene set of the target model, wherein the target model is a model trained based on the original training scene set;

[0056] The first computational unit is used to determine the first loss function of the initial scene under the target model;

[0057] The second calculation unit is used to determine the first gradient of the first loss function with respect to the initial scene;

[0058] The scene generation unit is used to determine the first scene corresponding to the initial scene based on the first gradient and the initial scene;

[0059] The third computational unit is used to determine the second loss function of the first scenario under the target model;

[0060] The first judgment unit is used to determine whether the second loss function satisfies the preset first condition;

[0061] The scene determination unit is used to determine the first scene as the target scene if the second loss function satisfies the first condition.

[0062] Fifthly, embodiments of this application also provide a model testing apparatus, comprising:

[0063] The acquisition unit is used to generate a target scene set and a scene time set corresponding to the target model, wherein the target scene set includes target scenes generated based on the scene generation method described in any one of the first aspects, and the scene time set includes the generation time corresponding to the target scene;

[0064] The test unit is used to test the target model using the target scene set and the scene time set, and obtain the test results corresponding to the target model.

[0065] Sixthly, embodiments of this application also provide a model training apparatus, comprising:

[0066] A scene generation unit is used to generate a target scene corresponding to the target model based on the scene generation method described in any one of the first aspects.

[0067] Training set construction unit, used to add the target scene to the training set;

[0068] The training unit is used to train the target model using the training set and gradient descent until the loss function of the target model converges on the preset validation set.

[0069] In a seventh aspect, embodiments of this application also provide an apparatus, characterized in that the apparatus includes: a processor and a memory storing computer program instructions;

[0070] When the processor executes the computer program instructions, it implements the scene generation method as described in any one of the first aspects, the model testing method as described in any one of the second aspects, or the model training method as described in the third aspect.

[0071] Eighthly, embodiments of this application also provide a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the scene generation method as described in any one of the first aspects, the model testing method as described in any one of the second aspects, or the model training method as described in the third aspect.

[0072] In a ninth aspect, embodiments of this application also provide a computer program product, wherein instructions in the computer program product, when executed by a processor of an electronic device, cause the electronic device to perform the scene generation method as described in any one of the first aspects, the model testing method as described in any one of the second aspects, or the model training method as described in the third aspect.

[0073] The scene generation method of this application embodiment selects an original training scene from the original training scene set corresponding to the target model as the initial scene, determines the first loss function of the initial scene under the target model, determines the first gradient of the first loss function with respect to the original initial scene, determines the first scene based on the first gradient and the initial scene, and determines the second loss function of the first scene under the target model. Then, when it is determined that the second loss function meets a preset first condition, the first scene is determined as the target scene. According to the embodiments of this application, new target scenes can be generated based on the original training scene set of the model, without the need for scene sampling from the actual environment. This shortens the time required to obtain scenes, reduces the consumption of manpower and material resources, and utilizes the gradient information and loss function information within the target model to generate new scene data more specifically. Attached Figure Description

[0074] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0075] Figure 1 This is a flowchart illustrating a scene generation method provided in one embodiment of this application;

[0076] Figure 2 This is a flowchart illustrating a scene generation method provided in another embodiment of this application;

[0077] Figure 3 This is a flowchart illustrating a scene generation method provided in another embodiment of this application;

[0078] Figure 4 This is a flowchart illustrating a scene generation method provided in yet another embodiment of this application;

[0079] Figure 5This is a schematic flowchart of a model training method provided in one embodiment of this application;

[0080] Figure 6 This is a flowchart illustrating a model testing method provided in one embodiment of this application;

[0081] Figure 7 This is a schematic diagram of model convergence provided in one embodiment of this application;

[0082] Figure 8 This is a schematic diagram of the structure of a scene generation device provided in one embodiment of this application;

[0083] Figure 9 This is a schematic diagram of the structure of a model training device provided in one embodiment of this application;

[0084] Figure 10 This is a schematic diagram of the structure of a model testing device provided in one embodiment of this application;

[0085] Figure 11 This is a schematic diagram of the structure of a device provided in another embodiment of this application. Detailed Implementation

[0086] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0087] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0088] To address the problems of existing technologies, embodiments of this application provide a scene generation method, a model testing method, and a model training method. The scene generation method provided in this application is mainly used for generating scene library samples at the micro-pixel level.

[0089] The scene generation method provided in the embodiments of this application will be introduced first below.

[0090] Figure 1 A flowchart illustrating a scene generation method provided in one embodiment of this application is shown. Figure 1 As shown, the scene generation method provided by Shili8 in this application may include the following steps:

[0091] S11. Select an original training scene from the original training scene set of the target model as the initial scene, where the target model is a model trained based on the original training scene set.

[0092] In the embodiments of this application, "scene" refers to a scene image.

[0093] S12. Determine the first loss function for the initial scene under the target model.

[0094] S13. Determine the first gradient of the first loss function with respect to the initial scene.

[0095] S14. Generate a first scene corresponding to the initial scene based on the first gradient and the initial scene.

[0096] S15. Determine the second loss function for the first scenario under the target model.

[0097] S16. Determine whether the second loss function meets the preset first condition. If it is determined that the second loss function meets the first condition, then execute S17.

[0098] S17. Determine the first scene as the target scene.

[0099] The scene generation method of this application embodiment selects an original training scene from the original training scene set corresponding to the target model as the initial scene, determines the first loss function of the initial scene under the target model, determines the first gradient of the first loss function with respect to the original initial scene, generates a first scene based on the first gradient and the initial scene, and determines the second loss function of the first scene under the target model. Then, when it is determined that the second loss function meets a preset first condition, the first scene is determined as the target scene. According to the embodiments of this application, new target scenes can be generated based on the original training scene set of the model, without the need for scene sampling from the actual environment. This shortens the time required to acquire scenes, reduces the consumption of manpower and resources, and utilizes the gradient information and loss function information within the target model to generate new scene data more specifically.

[0100] The specific implementation methods for each of the above steps are described below.

[0101] In some embodiments, the training process of the target model in S11 may include the following steps:

[0102] Step 1: Select the model m to be trained from the set training model library M, where m∈M.

[0103] This application's embodiments can be used to train the perception model of the perception module in an autonomous driving system. The perception module typically contains multiple perception models, which may be used to implement different functions. For example, they may include models for pose perception, image segmentation, traffic light detection, and / or environmental object detection. The models to be trained or tested in the perception module can be collected and organized into a training model library M. Based on this, during model training, the user can select a suitable model from the training model library M according to actual needs. For example, the user can select the PoseNet model for pose perception as model m from the training model library M.

[0104] Step 2: Construct an initial training scene set for model m

[0105] The training scenarios for different models may vary. For example, a model for traffic light detection (such as the YOLO model) is trained on a traffic light scene, while a model for image semantic segmentation (such as the Faster R-CNN model) is trained on a scene of all objects within a 360-degree radius around an autonomous vehicle. Therefore, it is usually necessary to prepare an initial training scene set for each model. Among them, the initial training scene set The scenes can be extracted from a historical scene database.

[0106] A historical scene database can be a database composed of scenes previously acquired through hardware devices or web searches. By extracting scenes from this historical scene database to form the initial training scene set, there is no need to collect scenes from the actual environment, which shortens the scene acquisition time, reduces the consumption of manpower and resources, and improves the efficiency of model training.

[0107] Step 3: Utilize the initial training scene set The model m is trained to obtain the trained model m, which is also the target model.

[0108] Using the initial training scenario set Training model m can be divided into the following steps:

[0109] Step 3.1: Initial training scene set Image processing is performed on the scenes to ensure that each scene meets the input requirements of model m.

[0110] Initial training scenario set The scene in the text refers to the scene image. Because... It typically includes multiple scenes, and these different scenes may have been collected using different hardware devices. The size and dimensions may vary depending on the scene. To ensure the model's performance, it is necessary to... Image processing is performed on the scene.

[0111] Image processing may include cropping the scene to make its dimensions meet requirements. It may also include normalizing the pixel values ​​of the scene.

[0112] Assume that the input dimensions of model m are W×H×3, where W and H are the standard length and width of the model's output scene. The dimensions of scene s are W. s ×H s ×3, then each layer s in scene s k The center point is When performing image processing on scene s, s can be cropped. k The rectangle in the diagram, where the coordinates of the bottom left corner of the rectangle are... The coordinates of the top right corner are After cropping, the pixel values ​​of each layer are normalized according to the following formula.

[0113]

[0114]

[0115]

[0116] in, This represents the average pixel value of all pixels in the k-th layer of scene s, where k is 1, 2, or 3, representing the first, second, or third layer of the color image, respectively. This represents the pixel value of a pixel in the k-th layer of scene s. Let represent the pixel variance of all pixels in the k-th layer of scene s.

[0117] Step 3.2: Initial training scene set Divided into training set and verification set

[0118] The partitioning ratio can be preset, and then the initial training scene set can be divided according to the partitioning ratio. Divided into training set and verification set The partition ratio can be the ratio of the training set to the validation set, which can be set according to actual needs. For example, the partition ratio can be set to 3:1, and the initial training scene set will be based on this partition ratio. When dividing, you can Divide the data into four equal parts, and use three of these parts to form the training set. Use the remaining set as the validation set.

[0119] Step 3.3: Utilizing the training set Model m is trained using gradient descent until it is validated on the validation set. The loss function on the surface converges.

[0120] The perceptual model is trained using gradient descent, and there are many common types of gradient descent, such as stochastic gradient descent, Adam, and RMSprop. The loss function for scene s on model m is... During the training process, It will continue to decrease along the direction of gradient descent.

[0121] This completes the training of model m, resulting in the target model.

[0122] As can be seen from the above, the target model utilizes the training set. Therefore, the training set is obtained through training. This refers to the original set of training scenarios for the target model; the training set. The scene in the image is the original training scene.

[0123] In some embodiments, in S11, when selecting an initial training scene from the original training scene set of the target model, any initial training scene can be arbitrarily selected from the original training scene set. That is, from... Select any scene As the initial scene.

[0124] In some embodiments, in S12, when determining the first loss function of the initial scene under the target model, the initial scene can be input into the target model to obtain the loss function of the target model, and the obtained loss function can be used as the first loss function of the initial scene under the target model.

[0125] In some embodiments, in S13, the first gradient of the first loss function with respect to the initial scene can be calculated based on the following gradient calculation formula:

[0126]

[0127] in, Represents the first loss function The first gradient of the initial scene s.

[0128] because Since it is a high-dimensional matrix, the above calculation formula can be solved by a computer to improve computational efficiency. The size and dimensions are consistent with the initial scene s.

[0129] In some embodiments, in S14, the first scene corresponding to the initial scene can be generated in the following manner:

[0130] Calculate the product of the first gradient and the first coefficient, and then use the sum of the initial scene and the above product as the first scene corresponding to the initial scene.

[0131] In one example, the first scenario can be calculated using the following formula:

[0132]

[0133] Where s′ represents the first scene corresponding to the initial scene s, and η is the first coefficient, which is a constant.

[0134] Using the above method, after performing gradient testing on the target model with the initial scene, the gradient is applied back to the initial scene to generate a new first scene.

[0135] In some embodiments, when determining the second loss function of the first scene under the target model in S15, the first scene can be input into the target model to obtain the loss function of the target model, and then the obtained loss function can be used as the second loss function of the first scene under the target model.

[0136] In some embodiments, the Taylor second-order expansion is shown below:

[0137]

[0138] Among them, Δs=s′-s,o(Δs)→0, This is the second loss function for the first scene s′ under the target model. According to the above Taylor second-order expansion, Given the decision condition, if Where δ is a threshold constant characterizing the change in the loss function, and is a pre-given value. This indicates that the effect of changing the loss function is not significant enough, and the newly generated first scene s′ cannot be used to retrain the perception model; further generation is required. This indicates that the change in the loss function is large enough, and the newly generated first scene s′ can reveal the defects of the model. Using the newly generated first scene s′, the perception model can be retrained, thereby improving the model performance.

[0139] Based on this, in S16, the first condition can be set as follows: Here, δ is the first threshold. When determining whether the second loss function meets the preset first condition, the difference between the second loss function and the first loss function can be calculated, and the obtained difference can be compared with the first threshold. If the difference is greater than or equal to the first threshold, the second loss function is determined to meet the preset first condition. If the difference is less than the first threshold, the second loss function is determined not to meet the first condition.

[0140] In some embodiments, in S17, if the first scene is determined as the target scene, the first scene can be added to the target scene set. In order to ensure It contains target scenes generated from the original training scene set of the target model, which can be initialized before the first execution of S11.

[0141] As another implementation of this application, in order to generate target scenes based on each original training scene in the original training scene set, such as Figure 2 As shown, S17 may also include: returning to execute S11.

[0142] Based on the above method, after generating a new target scene based on an original scene, return to S11, and continue to select original training scenes from the original training scene set as initial scenes for generating initial scenes. Based on this, multiple target scenes can be generated.

[0143] As another implementation of this application, in order to more accurately determine whether the target scene can be generated based on the original training scene, such as Figure 3 As shown, if the second loss function does not satisfy the first condition, the scene generation method may further include the following steps:

[0144] S18. Use the first scene as the initial scene and return to execute S12.

[0145] By using the above method, when it is determined that the second loss function does not meet the first condition, compared to directly determining that the selected original training scene cannot generate the target scene and discarding it, by using the first scene as the initial scene for iteration, the probability of generating the target scene is increased, and the number of generated target scenes is also increased.

[0146] In some embodiments, considering the efficiency of generating the target scene, for each original training scene, timing can begin from the moment the original training scene is selected as the initial scene, and the number of iterations corresponding to the original training scene can be recorded. Specifically, a timer can be used for timing, and a counter can be used to record the number of iterations. Furthermore, when the original training scene is first selected as the initial scene, both the timer and the counter are reset to 0.

[0147] Then, before executing S18, the iteration count corresponding to the original training scene selected in S11 can be incremented by 1, and the second duration from selecting this original training scene as the initial scene to determining that the second loss function does not satisfy the first condition can be determined. It is also determined whether the iteration count corresponding to this original training scene is less than the maximum iteration count, and whether the second duration is less than the maximum timing time. If it is determined that the iteration count is less than the maximum iteration count and the second duration is less than the maximum timing time, then S18 is executed. If it is determined that the iteration count is greater than or equal to the maximum iteration count, or that the second duration is greater than or equal to the maximum timing time, then it is determined that the original target scene selected in S11 failed to generate the target scene. Then, S11 can be returned to select another original target scene as the initial scene for target scene generation. Wherein, the maximum timing time T... max And the maximum number of iterations Itr max It can be a fixed value given before the first execution of S11.

[0148] As another implementation of this application, since repeatedly selecting the same scene as the initial scene for target scene generation may generate duplicate target scenes, and duplicate scenes are usually invalid scenes, in order to avoid generating invalid scenes, after selecting the original training scene as the initial scene in S11, it can be deleted from the original training scene set. In this way, when selecting the initial scene again in the future, the original training scene will not be selected again, thus avoiding the repeated selection of the original training scene and thus avoiding the generation of duplicate invalid scenes.

[0149] Furthermore, since the original training scene set contains a finite number of original training scenes, deleting an original training scene from the original training scene set may result in the original training scene set becoming an empty set, meaning that all original training scenes in the original training scene set are deleted. In this case, it will be impossible to select an original training scene as the initial scene from the original training scene set. Therefore, to avoid invalid operations, before executing S11, it can be determined whether the original training scene set is an empty set. If it is determined that the original training scene set is not an empty set, then S11 is executed. If it is determined that the original training scene set is an empty set, it is determined that all original training scenes in the original training scene set have been generated, thereby stopping the execution of the scene generation step.

[0150] As another implementation of this application, such as Figure 4 As shown, S17 may also include:

[0151] S19. Determine the first duration from selecting the original training scene as the initial scene to determining the first scene as the target scene, and then use the first duration as the generation duration corresponding to the target scene.

[0152] The production time corresponding to the target scenario is the time taken to generate the target scenario.

[0153] In one example, the generation time corresponding to the target scene can also be added to the scene time set T. NEW In order to ensure T NEW The data includes the generation time of the target scene based on the original training scene set of the target model. T can be initialized before the first execution of S11. NEW =φ.

[0154] Furthermore, Target scenario and T NEW The time taken to generate the target scene must correspond one-to-one.

[0155] Therefore, in order to further improve the performance of the target model, based on the scene generation method provided in the above embodiments, this application also provides a specific implementation of the model training method.

[0156] like Figure 5 As shown, the model training method provided in this application embodiment may include the following steps:

[0157] S51. Based on the scene generation method provided in the above embodiments, generate the target scene corresponding to the target model.

[0158] S52. Add the target scene to the training set.

[0159] S53. Using the training set, train the target model using gradient descent until the loss function of the target model converges on the preset validation set.

[0160] The preset validation set can be the validation set used when training the target model based on the original training scene set.

[0161] The model training method provided in this application selects an original training scene from the original training scene set corresponding to the target model as the initial scene, determines the first loss function of the initial scene under the target model, determines the first gradient of the first loss function with respect to the original initial scene, determines the first scene based on the first gradient and the initial scene, and determines the second loss function of the first scene under the target model. Then, when the second loss function meets a preset first condition, the first scene is determined as the target scene. According to this application embodiment, a new target scene can be generated based on the original training scene set of the model, without the need for scene sampling from the actual environment. This shortens the time required to obtain scenes, reduces the consumption of manpower and resources, and utilizes the gradient information and loss function information inside the target model to generate new scene data more specifically. Retraining the target model based on the target scene can improve the model's recognition effect, robustness, and generalization.

[0162] Currently, the standard practice for performance testing of models in the perception module is for testers to conduct road tests on autonomous vehicles, collecting and organizing scenarios where misjudgments, stalls, sudden braking, or jerking occur to form a scenario library. Then, the model's pass rate on this scenario library is calculated using a test pass rate formula to characterize the model's performance.

[0163] The formula for the test pass rate is as follows:

[0164]

[0165] Where P is a percentage, representing the pass rate, and N... P To determine the number of scenarios that pass the test, N T This represents the total number of scenes.

[0166] However, the aforementioned model performance testing method requires extensive road testing, which incurs significant time and financial costs. Furthermore, the accuracy of the metrics data provided by this method needs further improvement.

[0167] To address the aforementioned technical problems, based on the scene generation method provided in the above embodiments, this application also provides a specific implementation of a model testing method.

[0168] See Figure 6 The model testing method provided in this application embodiment may include the following steps:

[0169] S61. Generate a target scene set and a scene time set corresponding to the target model, wherein the target scene set includes target scenes generated based on the scene generation method provided in the above embodiments, and the scene time set includes the generation time corresponding to the target scenes determined based on the scene generation method provided in the above embodiments.

[0170] In one example, the scene generation method provided in the above embodiments can be used to generate a target scene, which is the target scene corresponding to the target model. The generated target scenes are then grouped into a target scene set, the generation time of each generated target scene is determined, and the generation times are combined into a scene time set.

[0171] S62. Test the target model using the target scene set and the scene time set to obtain the test results corresponding to the target model.

[0172] The model testing method provided in this application selects an original training scene from the original training scene set corresponding to the target model as the initial scene, determines the first loss function of the initial scene under the target model, determines the first gradient of the first loss function with respect to the original initial scene, determines the first scene based on the first gradient and the initial scene, and determines the second loss function of the first scene under the target model. Then, when the second loss function meets a preset first condition, the first scene is determined as the target scene. According to this application embodiment, new target scenes can be generated based on the original training scene set of the model, eliminating the need for scene sampling from the actual environment. This shortens the time required to acquire scenes, reduces the consumption of manpower and resources, and allows for more targeted generation of new scene data by utilizing the gradient information and loss function information within the target model. Testing the target model using the target scene set and the scene time set eliminates the need for road testing, thereby reducing the time and financial costs required for model testing.

[0173] In one example, when testing a target model using a set of target scenarios and a set of scenario durations, the test metrics corresponding to the target model can be calculated according to a preset test metric calculation formula based on the set of target scenarios and the set of scenario durations. The calculated test metrics are then used as the test results of the target model.

[0174] The formula for calculating the test index is as follows:

[0175]

[0176] Where P represents the test metric, s represents the target scenario, and t represents the test metric. s This indicates the generation time corresponding to the target scene s. Represents the set of target scenarios. T represents the number of target scenes contained in the target scene set. max This represents the set of times taken for a scene. The original training scenario set contains the number of original training scenarios. α represents the importance of time and β represents the importance of success rate. Both α and β are constants, and α+β=1, α>0, β>0. ∈ is a minimal constant, mainly used to handle the case where the denominator is equal to 0.

[0177] In the formula for calculating test metrics,

[0178]

[0179]

[0180] ≤α+β=1

[0181] Therefore, the P-index ranges from [0,1]. In the formula, for a set of target scenes, the shorter the generation time, the more defects the target model has, the worse its performance, and the smaller the P-value. Conversely, for a set of target scenes, the more target scenes generated, the more scenes the target model does not cover, the worse its performance, and the smaller the P-value. Therefore, using the P-index to measure the robustness of a target model is generally meaningful; a higher P-value indicates stronger robustness, more complete training, and a greater ability to avoid attacks from unknown scene data.

[0182] The model training and testing methods provided in this application embodiment are verified through an example below:

[0183] Taking the PoseNet autonomous driving perception model as an example for training and testing, the comma2k19 video dataset was used as experimental data. This dataset contains multiple segments of raw scene data. A portion of the raw scene data was selected to validate the training and testing methods of the model. A training scene set and a validation scene set were constructed based on the selected raw scene data. The training scene set contained 3000 training scene data points, and the validation scene set contained 500 validation scene data points. After multiple training iterations, the model's loss function finally converged on the validation scene set. The convergence graph is shown below. Figure 7 As shown. Then, the scene generation method provided in the above embodiment is used to generate new scenes from 3000 training scene data, ultimately generating 748 target scene data, with a maximum iteration time T. max=20s, the total time to generate the 748 target scene data was 11668s. Setting α=β=0.5, ε=0.01, the final P-index is P=0.765. When the model is retrained using the target scene data, the model converges on the validation scene set, proving that the high-dimensional function distribution of the target scene data is similar to that of the original scene data, satisfying the assumption that the training and validation data of the perceptual model have the same distribution.

[0184] Based on the scene generation method provided in the above embodiments, this application also provides specific implementations of the scene generation apparatus. Please refer to the following embodiments.

[0185] See Figure 8 The scene generation apparatus provided in this application embodiment includes the following units:

[0186] The selection unit 801 is used to select an original training scene as the initial scene from the original training scene set of the target model, wherein the target model is a model trained based on the original training scene set.

[0187] The first calculation unit 802 is used to determine the first loss function of the initial scene under the target model.

[0188] The second calculation unit 803 is used to determine the first gradient of the first loss function with respect to the initial scene.

[0189] The scene generation unit 804 is used to generate the first scene corresponding to the initial scene based on the first gradient and the initial scene.

[0190] The third calculation unit 805 is used to determine the second loss function of the first scene under the target model.

[0191] The first judgment unit 806 is used to determine whether the second loss function satisfies the preset first condition.

[0192] The scene determination unit 807 is used to determine the first scene as the target scene if it is determined that the second loss function satisfies the first condition.

[0193] The scene generation apparatus of this application selects an original training scene from the original training scene set corresponding to the target model as an initial scene, determines a first loss function of the initial scene under the target model, determines a first gradient of the first loss function with respect to the original initial scene, determines a first scene based on the first gradient and the initial scene, and determines a second loss function of the first scene under the target model. Then, when the second loss function meets a preset first condition, the first scene is determined as the target scene. According to this application embodiment, a new target scene can be generated based on the original training scene set of the model, eliminating the need for scene sampling from the actual environment. This shortens the time required to acquire scenes, reduces the consumption of manpower and resources, and allows for more targeted generation of new scene data by utilizing the gradient information and loss function information within the target model.

[0194] As one possible implementation, the above-mentioned device may further include:

[0195] The timing unit is used to determine the first duration from selecting the original training scene as the initial scene to determining the first scene as the target scene after the first scene has been identified as the target scene;

[0196] The first duration is taken as the generation duration corresponding to the target scene.

[0197] As one possible implementation, the above-mentioned device may further include:

[0198] The first return unit is used to return to the step of selecting the original training scene from the original training scene set of the target model as the initial scene after the first scene is determined as the target scene.

[0199] As one possible implementation, the above-mentioned device may further include:

[0200] An iterative unit is used to determine, if it is determined that the second loss function does not satisfy the first condition, the first scene is taken as the initial scene, and the step of determining the first loss function of the initial scene under the target model is returned.

[0201] As one possible implementation, the scene generation unit 804 is used for:

[0202] Calculate the product of the first gradient and the first coefficient;

[0203] The sum of the initial scene and the product above is taken as the first scene corresponding to the initial scene.

[0204] As one possible implementation, the decision unit 806 is used for:

[0205] Calculate the difference between the second loss function and the first loss function;

[0206] Compare the difference with the first threshold;

[0207] If the difference is found to be greater than or equal to the first threshold, then the second loss function is determined to satisfy the preset first condition.

[0208] If the difference is less than the first threshold, then the second loss function is determined not to meet the first condition.

[0209] As one possible implementation, the above-mentioned device may further include:

[0210] The deletion unit is used to select an original training scene from the original training scene set of the target model as the initial scene, and then delete the selected original training scene from the original training scene set.

[0211] The above-mentioned device may further include a second judgment unit, used to determine whether the original training scene set is an empty set before the step of selecting the original training scene as the initial scene from the original training scene set of the target model.

[0212] Selecting unit 801 is used for:

[0213] If the original training scene set is not empty, select an original training scene from the original training scene set of the target model as the initial scene.

[0214] As one possible implementation, the above-mentioned device further includes:

[0215] The parameter update unit is used to increment the number of iterations corresponding to the selected original training scene by 1 after determining that the second loss function does not meet the first condition and before taking the first scene as the initial scene, and to determine the second time from selecting the original training scene as the initial scene to determining that the second loss function does not meet the first condition.

[0216] The third judgment unit is used to determine whether the number of iterations is less than the maximum number of iterations;

[0217] The fourth judgment unit is used to determine whether the second duration is less than the maximum timing time;

[0218] Iteration units are used for:

[0219] If the number of iterations is less than the maximum number of iterations and the second duration is less than the maximum timing time, the first scene will be used as the initial scene.

[0220] Based on the model training method provided in the above embodiments, this application also provides specific implementations of the model training apparatus. Please refer to the following embodiments.

[0221] See Figure 9 The model training apparatus provided in this application includes the following units:

[0222] The scene generation unit 901 is used to generate a target scene corresponding to the target model based on the scene generation method provided in the above embodiments.

[0223] Training set construction unit 902 is used to add the target scene to the training set.

[0224] Training unit 903 is used to train the target model using gradient descent on the training set until the loss function of the target model converges on the preset validation set.

[0225] The model training apparatus provided in this application selects an original training scene as the initial scene from the original training scene set corresponding to the target model, determines the first loss function of the initial scene under the target model, determines the first gradient of the first loss function with respect to the original initial scene, determines the first scene based on the first gradient and the initial scene, and determines the second loss function of the first scene under the target model. Then, when the second loss function meets a preset first condition, the first scene is determined as the target scene. According to this application embodiment, a new target scene can be generated based on the original training scene set of the model, without the need for scene sampling from the actual environment. This shortens the time required to acquire scenes, reduces the consumption of manpower and resources, and utilizes the gradient information and loss function information within the target model to generate new scene data more specifically. Retraining the target model based on the target scene can improve the model's recognition effect, robustness, and generalization.

[0226] Based on the model testing method provided in the above embodiments, this application also provides specific implementation methods of the scene generation device. Please refer to the following embodiments.

[0227] See Figure 10 The model testing apparatus provided in this application embodiment may include the following units:

[0228] The acquisition unit 101 is used to generate a target scene set and a scene time set corresponding to the target model. The target scene set includes target scenes generated based on the scene generation method provided in the above embodiments, and the scene time set includes the generation time corresponding to the target scene.

[0229] Test unit 102 is used to test the target model using the target scene set and the scene time set, and obtain the test results corresponding to the target model.

[0230] The model testing method provided in this application selects an original training scene from the original training scene set corresponding to the target model as the initial scene, determines the first loss function of the initial scene under the target model, determines the first gradient of the first loss function with respect to the original initial scene, determines the first scene based on the first gradient and the initial scene, and determines the second loss function of the first scene under the target model. Then, when the second loss function meets a preset first condition, the first scene is determined as the target scene. According to this application embodiment, new target scenes can be generated based on the original training scene set of the model, eliminating the need for scene sampling from the actual environment. This shortens the time required to acquire scenes, reduces the consumption of manpower and resources, and allows for more targeted generation of new scene data by utilizing the gradient information and loss function information within the target model. Testing the target model using the target scene set and the scene time set eliminates the need for road testing, thereby reducing the time and financial costs required for model testing.

[0231] As one possible implementation, test unit 102 is used for:

[0232] The test indicators corresponding to the target model are calculated according to the target scenario set and the scenario time set using a preset test indicator calculation formula.

[0233] The test metrics are used as the test results for the target model;

[0234] The formula for calculating the test index is as follows:

[0235]

[0236] Where P represents the test metric, s represents the target scenario, and t represents the test metric. s This indicates the generation time corresponding to the target scene s. Represents the set of target scenarios. T represents the number of target scenes contained in the target scene set. max This represents the set of times taken for a scene. The original training scenario set contains the number of original training scenarios, α represents the time importance, β represents the success rate importance, α and β are constants, and α+β=1, α>0, β>0, ∈ is a minimal constant.

[0237] Figure 11 A schematic diagram of the hardware structure of the device provided in an embodiment of this application is shown.

[0238] The device may include a processor 1101 and a memory 1102 storing computer program instructions.

[0239] Specifically, the processor 1101 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0240] Memory 1102 may include mass storage for data or instructions. For example, and not limitingly, memory 1102 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 1102 may include removable or non-removable (or fixed) media. Where appropriate, memory 1102 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 1102 is non-volatile solid-state memory.

[0241] Memory 1102 may include read-only memory (ROM), random access memory (RAM), disk storage media device, optical storage media device, flash memory device, electrical, optical, or other physical / tangible memory storage device. Therefore, typically, memory 1102 includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the methods according to any aspect of this disclosure.

[0242] The processor 1101 reads and executes computer program instructions stored in the memory 1102 to implement any of the scene generation methods, model training methods, or model testing methods in the above embodiments.

[0243] In one example, the device may also include a communication interface 1103 and a bus 1110. Wherein, as... Figure 11 As shown, the processor 1101, memory 1102, and communication interface 1103 are connected through bus 1110 and complete communication with each other.

[0244] The communication interface 1103 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0245] Bus 1110 includes hardware, software, or both, that couples components of an online data traffic metering device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 1110 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, any suitable bus or interconnect is contemplated herein.

[0246] Furthermore, in conjunction with the scene generation method, model training method, or model testing method in the above embodiments, this application embodiment can provide a computer storage medium for implementation. This computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any one of the scene generation method, model training method, or model testing method in the above embodiments.

[0247] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0248] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0249] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0250] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0251] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A scene generation method characterized by, The method comprises the following steps: selecting an original training scene as an initial scene from a set of original training scenes of a target model, wherein the target model is a model trained based on the set of original training scenes; a scene refers to a scene image; determining a first loss function of the initial scene under the target model, wherein the first loss function is a loss function of the target model obtained by inputting the initial scene into the target model; determining a first gradient of the first loss function with respect to the initial scene; generating a first scene corresponding to the initial scene according to the first gradient and the initial scene; determining a second loss function of the first scene under the target model, wherein the second loss function is a loss function of the target model obtained by inputting the first scene into the target model; judging whether the second loss function satisfies a preset first condition; if it is determined that the second loss function satisfies the first condition, determining the first scene as a target scene; the generating of the first scene corresponding to the initial scene according to the first gradient and the initial scene comprises: calculating a product of the first gradient and a first coefficient; taking a sum of the initial scene and the product as the first scene corresponding to the initial scene; the judging of whether the second loss function satisfies the preset first condition comprises: calculating a difference between the second loss function and the first loss function; comparing the difference with a first threshold value; if the comparison shows that the difference is greater than or equal to the first threshold value, it is determined that the second loss function satisfies the preset first condition; if the comparison shows that the difference is less than the first threshold value, it is determined that the second loss function does not satisfy the first condition.

2. The method of claim 1, wherein, after the first scene is determined as the target scene, the method further comprises: determining a first time length used from the selection of the original training scene as the initial scene to the determination of the first scene as the target scene; taking the first time length as a generation time length corresponding to the target scene.

3. The method of claim 1, wherein, after the first scene is determined as the target scene, the method further comprises: returning to the step of selecting an original training scene as an initial scene from a set of original training scenes of a target model.

4. The method of claim 1, wherein, the method further comprises: if it is determined that the second loss function does not satisfy the first condition, taking the first scene as an initial scene and returning to the step of determining a first loss function of the initial scene under the target model.

5. The method of claim 1, wherein, after the original training scene is selected as the initial scene from the set of original training scenes of the target model, the method further comprises: deleting the selected original training scene from the set of original training scenes; before the step of selecting an original training scene as an initial scene from a set of original training scenes of a target model, the method further comprises: judging whether the set of original training scenes is an empty set; the step of selecting an original training scene as an initial scene from a set of original training scenes of a target model comprises: under the condition that it is determined that the set of original training scenes is not an empty set, selecting an original training scene as an initial scene from a set of original training scenes of a target model.

6. The method of claim 3, wherein, Before the first scene is taken as the initial scene after the determination that the second loss function does not satisfy the first condition, the method further comprises: adding 1 to the iteration number corresponding to the selected original training scene, and determining a second time length used from the selection of the original training scene as the initial scene to the determination that the second loss function does not satisfy the first condition; determining whether the iteration number is less than a maximum iteration number; determining whether the second time length is less than a maximum timing time; the taking of the first scene as the initial scene comprises: in the case where it is determined that the iteration number is less than the maximum iteration number and the second time length is less than the maximum timing time, taking the first scene as the initial scene.

7. A model testing method characterized by, comprises: generating a target scene set corresponding to a target model and a scene time set, wherein the target scene set contains target scenes generated based on the scene generation method of any one of claims 1-6, and the scene time set contains generation times corresponding to the target scenes; testing the target model by using the target scene set and the scene time set, to obtain a test result corresponding to the target model.

8. The method of claim 7, wherein, the testing of the target model by using the target scene set and the scene time set to obtain the test result corresponding to the target model comprises: calculating a test index corresponding to the target model according to the target scene set and the scene time set according to a preset test index calculation formula; taking the test index as the test result of the target model; the test index calculation formula is: wherein, denotes a test indicator, denotes a target scenario, denotes a target scenario corresponding generation time, denotes a target scenario set, denotes a number of target scenarios contained in the target scenario set; denotes a scenario time set; denotes a number of original training scenarios contained in the original training scenario set, denotes a time importance, denotes a success rate importance, and are constants, and , is a very small constant.

9. A model training method, comprising: comprises: generating a target scene corresponding to a target model based on the scene generation method of any one of claims 1-6; adding the target scene to a training set; training the target model by using the training set in a gradient descent manner until a loss function of the target model on a preset verification set converges.

10. A scene generation apparatus, characterized by comprising: the device comprises: a selection unit configured to select an original training scene as an initial scene from an original training scene set of a target model, wherein the target model is a model trained based on the original training scene set; a scene refers to a scene image; a first calculation unit configured to determine a first loss function of the initial scene under the target model, the first loss function being a loss function of the target model obtained by inputting the initial scene into the target model; a second calculation unit configured to determine a first gradient of the first loss function with respect to the initial scene; a scene generation unit configured to generate a first scene corresponding to the initial scene according to the first gradient and the initial scene; a third calculation unit configured to determine a second loss function of the first scene under the target model, the second loss function being a loss function of the target model obtained by inputting the first scene into the target model; a first determination unit configured to determine whether the second loss function satisfies a preset first condition; a scene determination unit configured to determine the first scene as a target scene if it is determined that the second loss function satisfies the first condition; the scene generation unit is configured to: calculating a product of the first gradient and a first coefficient; adding the product to the initial scene to obtain a first scene corresponding to the initial scene; the first judging unit is configured to: calculating a difference between the second loss function and the first loss function; comparing the difference with a first threshold value; if the comparison result shows that the difference is greater than or equal to the first threshold value, determining that the second loss function satisfies a preset first condition; if the comparison result shows that the difference is less than the first threshold value, determining that the second loss function does not satisfy the first condition.

11. A model test apparatus characterized by comprising: comprise: an obtaining unit configured to generate a target scene set corresponding to a target model and a scene time set, wherein the target scene set comprises target scenes generated based on the scene generation method according to any one of claims 1-6, and the scene time set comprises generation times corresponding to the target scenes; a testing unit configured to test the target model by using the target scene set and the scene time set, and obtain a test result corresponding to the target model.

12. A model training apparatus, comprising: comprise: a scene generation unit configured to generate target scenes corresponding to a target model based on the scene generation method according to any one of claims 1-6; a training set construction unit configured to add the target scenes to a training set; a training unit configured to train the target model by using the training set in a gradient descent manner until a loss function of the target model on a preset verification set converges.

Images