A perception algorithm data closed loop method and related apparatus

Abstract

The application discloses a perception algorithm data closed loop method and related device, and relates to the virtual reality technical field.In the application, the vehicle feature set of a to-be-tested vehicle and the scene identifier of a test scene are acquired from intelligent driving algorithm model training and test requests; then, based on each feature attribute contained in the vehicle feature set, a virtual vehicle satisfying a preset vehicle similarity condition with the to-be-tested vehicle is constructed; further, a target virtual scene matching the scene identifier is selected from a preset candidate virtual scene set; finally, the virtual vehicle is driven to travel in the target virtual scene, and the perception algorithm model of the virtual vehicle is trained and tested based on the environmental perception information of the virtual vehicle. In this way, based on the constructed virtual vehicle and the preset candidate virtual scene set, the test cost of extreme working conditions and the like is reduced, the long tail effect data problem of automatic driving is solved, and the reliability and comprehensiveness of the perception algorithm model are improved.

Description

Technical Field

[0001] This application relates to the field of virtual reality technology, and in particular to a perception algorithm data closed-loop method and related apparatus. Background Technology

[0002] Autonomous driving is a cutting-edge technology and focus in the current transportation sector. Furthermore, with the approaching mass production of Level 3 (L3) and above autonomous driving technologies, the requirements for the maturity and reliability of autonomous driving technologies are also increasing dramatically.

[0003] Currently, in order to ensure the maturity and reliability of autonomous driving technology, intelligent driving vision perception systems are usually used to train and test massive amounts of data. As a result, many problems need to be solved, such as various marginal and dangerous working conditions caused by different weather changes, complex road conditions and uncertainties in the behavior of traffic participants, the high cost and long cycle brought about by massive road test mileage data, and the high cost of manual data annotation.

[0004] See related technologies. Figure 1 As shown, the intelligent driving vision perception data system collects a large amount of road data through real vehicle road tests, then processes the obtained road data (such as cleaning and filtering the data), and manually annotates the useful data. Next, based on the manually annotated data, it trains the perception algorithm model and evaluates the perception algorithm model. Furthermore, it saves the algorithm model to the algorithm model library, and finally, it deploys the algorithm model library to real vehicles for vehicle performance testing.

[0005] However, the above-mentioned perception algorithm model training method requires the collection of road data through real vehicle road tests. This means that in some road scenarios where road data collection is very difficult, it is impossible to directly collect the corresponding road data to train and test the perception algorithm model, thus failing to guarantee the reliability and comprehensiveness of the trained perception algorithm model.

[0006] Therefore, using the above method results in a vehicle perception algorithm model with low reliability and comprehensiveness. Summary of the Invention

[0007] This application provides a data closed-loop method and related apparatus for a perception algorithm, which can improve the reliability and comprehensiveness of the perception algorithm model.

[0008] In a first aspect, embodiments of this application provide a method for training a vehicle perception algorithm model, the method comprising:

[0009] From the intelligent driving algorithm model training and testing requests sent by the target terminal, obtain the vehicle feature set of the vehicle under test and the scene identifier of the test scenario;

[0010] Based on the various feature attributes contained in the vehicle feature set, a virtual vehicle that meets the preset vehicle similarity conditions with the vehicle under test is constructed.

[0011] From a pre-defined set of candidate virtual scenes, target virtual scenes that match the scene identifier are selected; wherein, the target virtual scene and the test scene meet the pre-defined scene similarity conditions;

[0012] The system drives virtual vehicles in a target virtual scene and trains and tests the perception algorithm model of the virtual vehicles based on their environmental perception information.

[0013] Secondly, embodiments of this application also provide a data closed-loop device for a sensing algorithm, the device comprising:

[0014] The acquisition module is used to obtain the vehicle feature set of the vehicle under test and the scene identifier of the test scenario from the intelligent driving algorithm model training and testing request sent by the target terminal.

[0015] The construction module is used to construct a virtual vehicle that meets the preset vehicle similarity conditions with the vehicle under test based on the various feature attributes contained in the vehicle feature set.

[0016] The filtering module is used to filter out target virtual scenes that match scene identifiers from a preset set of candidate virtual scenes; wherein, the target virtual scene and the test scene meet preset scene similarity conditions;

[0017] The driving module is used to drive the virtual vehicle in the target virtual scene and to train and test the perception algorithm model of the virtual vehicle based on the environmental perception information of the virtual vehicle.

[0018] In one optional embodiment, the preset set of candidate virtual scenes includes candidate virtual scenes of one or more test scenes; wherein, the candidate virtual scenes of the first test scene are obtained in the following manner:

[0019] Based on the scene information of the first test scene collected by the scene information acquisition device, the inherent scene features contained in the first test scene are obtained.

[0020] Based on the inherent features of each scenario and the dynamic feature set corresponding to the training and testing requests of the intelligent driving algorithm model, a candidate virtual scenario for the first test scenario is constructed. The first test scenario is any one of one or more test scenarios.

[0021] In an optional embodiment, during the process of constructing candidate virtual scenes for the first test scenario, the construction module is further configured to:

[0022] Determine the inherent set type of the inherent feature set to which each inherent scene feature belongs, and the dynamic set type of the dynamic feature set;

[0023] Based on the inherent collection type and the dynamic collection type, a scene identifier for the first test scenario is generated.

[0024] In one alternative embodiment, the various feature attributes include visual perception feature attributes of the vehicle under test.

[0025] Based on the various feature attributes contained in the vehicle feature set, a virtual vehicle that meets the preset vehicle similarity conditions with the vehicle to be tested is constructed. The construction module is specifically used for:

[0026] Based on visual perception feature attributes, a visual perception device for virtual vehicles is constructed.

[0027] In one optional embodiment, when driving the virtual vehicle to move within the target virtual scene, the driving module is specifically used for:

[0028] Using the visual perception device of the virtual vehicle, road condition information within a set range in front of the virtual vehicle in the target virtual scene is obtained;

[0029] The vehicle driving mode is set based on the corresponding road condition information, which drives the virtual vehicle to drive in the target virtual scene.

[0030] Thirdly, embodiments of this application also propose an electronic device, which includes a processor and a memory, wherein the memory stores program code, and when the program code is executed by the processor, the processor performs the steps of the perception algorithm data loop closure method described in the first aspect.

[0031] Fourthly, embodiments of this application also propose a computer-readable storage medium including program code, which, when run on an electronic device, causes the electronic device to perform the steps of the perception algorithm data loop closure method described in the first aspect.

[0032] Fifthly, embodiments of this application also provide a computer program product, which, when invoked by a computer, causes the computer to execute the data closure method steps of the perception algorithm as described in the first aspect.

[0033] The beneficial effects of this application are as follows:

[0034] In the perception algorithm data closed-loop method provided in this application, the vehicle feature set of the vehicle under test and the scene identifier of the test scenario are obtained from the intelligent driving algorithm model training and testing request sent by the target terminal. Then, based on the various feature attributes contained in the vehicle feature set, a virtual vehicle that meets the preset vehicle similarity conditions with the vehicle under test is constructed. Further, a target virtual scene that matches the scene identifier is selected from the preset candidate virtual scene set. The target virtual scene and the test scenario meet the preset scene similarity conditions. Finally, the virtual vehicle is driven to drive in the target virtual scene, and the perception algorithm model of the virtual vehicle is trained and tested based on the environmental perception information of the virtual vehicle.

[0035] This approach, based on constructing virtual vehicles and a pre-set set of candidate virtual scenarios containing target virtual scenarios, reduces testing costs for extreme conditions through autonomous driving virtual simulation applications. It constructs a highly realistic simulation environment and various extreme scenarios for autonomous driving, builds a traffic database based on real data, solves the long-tail effect data problem in autonomous driving, and avoids the technical drawbacks of related technologies that require real-vehicle road tests to collect road data. This results in the inability to directly collect the corresponding road data in some road scenarios where road data collection is very difficult, making it impossible to train and test the perception algorithm model and thus failing to guarantee the reliability and comprehensiveness of the trained perception algorithm model. Therefore, it improves the reliability and comprehensiveness of the perception algorithm model.

[0036] Furthermore, other features and advantages of this application will be set forth in the following description and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0038] Figure 1 A logical schematic diagram of a conventional visual perception data loop closure method provided in an embodiment of this application;

[0039] Figure 2 This is a schematic diagram of an optional system structure applicable to the embodiments of this application;

[0040] Figure 3This application provides a schematic flowchart of a method for constructing a candidate virtual scene.

[0041] Figure 4 A logical diagram illustrating the generation of scene identifiers provided in an embodiment of this application;

[0042] Figure 5 A schematic diagram illustrating the implementation process of a data loop closure method for a perception algorithm provided in this application embodiment;

[0043] Figure 6 A schematic diagram illustrating a virtual vehicle driving in a target virtual scene, provided as an embodiment of this application;

[0044] Figure 7 A method based on the embodiments of this application is provided. Figure 5 Schematic diagram of specific application scenarios;

[0045] Figure 8 A method based on the embodiments of this application is provided. Figure 5 A logical diagram;

[0046] Figure 9 A schematic diagram of the structure of a perception algorithm data closed-loop device provided in an embodiment of this application;

[0047] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0049] It should be noted that in the description of this application, "multiple" is understood as "at least two". "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. A connected to B can represent: A and B directly connected, or A and B connected through C. Furthermore, in the description of this application, terms such as "first" and "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance or order.

[0050] Before introducing the perception algorithm data loop closure method provided in the embodiments of this application, for ease of understanding, the design concept of the embodiments of this application will be briefly introduced below:

[0051] In related technologies, the visual perception data closed-loop method of intelligent driving visual perception data system (such as...) Figure 1 As shown, it requires massive amounts of data, therefore, it has many limitations, as follows:

[0052] 1. The data acquisition cycle of the actual vehicle road vision sensor is long, and it is difficult or impossible to collect data in many extreme scenarios, resulting in low efficiency;

[0053] 2. The cost of collecting and testing real-vehicle road vision sensor data is high, requiring dozens of vehicles to collect data and a large amount of manual data processing and annotation work;

[0054] 3. The data collected by the real vehicle road vision sensor is from a simple scenario, which makes it impossible to comprehensively test various scenarios. This results in the training algorithm having great safety risks and making it impossible to implement.

[0055] It is evident that current closed-loop methods for visual perception data suffer from problems such as long testing cycles, high costs, simplistic scenarios, and an inability to comprehensively test various scenarios. These issues pose numerous safety risks to perception algorithms, hindering the mass production and deployment of autonomous driving technology, and ultimately failing to ensure the maturity and reliability of autonomous driving technology.

[0056] However, applying Virtual Reality (VR) simulation technology to the field of autonomous driving—that is, building a physical world model in virtual space based on game rendering engines and real-world data—recreates the microscopic behavior of dynamic elements such as motor vehicles, non-motor vehicles, and pedestrians in real-world road scenes and traffic flows, physically reproducing details of the real environment such as weather and lighting, constructing a virtual reality simulation environment, and applying essential simulation elements such as physical-level perception sensor models, will replace real vehicle data collection and manual annotation. This will allow for the comprehensive and efficient synthesis of massive amounts of perfectly labeled data, enabling visual perception algorithm training and data closure, and reducing testing costs such as extreme conditions. Therefore, it can greatly improve the reliability and comprehensiveness of perception algorithm models. Furthermore, by constructing highly realistic simulation environments and various extreme scenarios for autonomous driving, and building a traffic database based on real data, the long-tail effect data problem in autonomous driving can be effectively solved. This will save on the development costs of autonomous driving data collection and shorten the cycle of algorithm deployment and verification, and to some extent, improve the efficiency of training and testing perception algorithm models.

[0057] In view of this, in order to improve the reliability and comprehensiveness of the perception algorithm model, a perception algorithm data closed-loop method is proposed in this embodiment. Specifically, it includes: obtaining the vehicle feature set of the vehicle under test and the scene identifier of the test scenario from the intelligent driving algorithm model training and testing request sent by the target terminal; then, constructing a virtual vehicle that meets a preset vehicle similarity condition with the vehicle under test based on the various feature attributes contained in the vehicle feature set; further, selecting a target virtual scene that matches the scene identifier from a preset set of candidate virtual scenes; wherein the target virtual scene and the test scenario meet a preset scene similarity condition; finally, driving the virtual vehicle in the target virtual scene, and training and testing the perception algorithm model of the virtual vehicle based on the environmental perception information of the virtual vehicle.

[0058] In particular, the preferred embodiments of this application will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application. Furthermore, the embodiments of this application and the features in the embodiments can be combined with each other without conflict.

[0059] See Figure 2 As shown, this is a schematic diagram of a system architecture provided in an embodiment of this application. The system architecture includes a target terminal (201a, 201b) and a server 202. The target terminal (201a, 201b) and the server 202 can interact with each other through a communication network. The communication network can use wireless communication or wired communication.

[0060] For example, the target terminal (201a, 201b) can access the network and communicate with the server 202 through cellular mobile communication technology, wherein the cellular mobile communication technology includes, for example, 5th generation mobile network (5G) technology.

[0061] Optionally, the target terminal (201a, 201b) can access the network and communicate with the server 202 via short-range wireless communication, wherein the short-range wireless communication method includes, for example, Wireless Fidelity (Wi-Fi) technology.

[0062] This application embodiment does not impose any limitation on the number of communication devices involved in the above system architecture. For example, there may be more target terminals, or no target terminals, or other network devices may be included, such as... Figure 2 As shown, only the target terminals (201a, 201b) and server 202 are described as examples. The following is a brief introduction to each of the above devices and their respective functions:

[0063] The target terminal (201a, 201b) is a device that can provide voice and / or data connectivity to a user, and may be a device that supports wired and / or wireless connection methods.

[0064] For example, target terminals (201a, 201b) include, but are not limited to: mobile phones, tablets, laptops, handheld computers, mobile internet devices (MID), wearable devices, virtual reality devices, augmented reality (AR) devices, wireless terminal devices in industrial control, wireless terminal devices in autonomous driving, wireless terminal devices in smart grids, wireless terminal devices in transportation safety, wireless terminal devices in smart cities, or wireless terminal devices in smart homes, etc.

[0065] In addition, the target terminal (201a, 201b) may have a related client installed. The client may be software, such as an application (APP), browser, short video software, or webpage, mini-program, etc.

[0066] It should also be noted that, in the embodiments of this application, the target terminal (201a, 201b) can be used to send a training and testing request for an intelligent driving algorithm model carrying the vehicle feature set of the vehicle under test and the scene identifier of the test scene to the server 202.

[0067] Server 202 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0068] It is worth mentioning that, in the embodiments of this application, the server 202 is used to obtain the vehicle feature set of the vehicle under test and the scene identifier of the test scene from the intelligent driving algorithm model training and testing request sent by the target terminal, and then construct a virtual vehicle that meets the preset vehicle similarity conditions with the vehicle under test based on the various feature attributes contained in the vehicle feature set. In this way, the server selects the target virtual scene that matches the scene identifier from the preset candidate virtual scene set, and drives the virtual vehicle to drive in the target virtual scene. Based on the environmental perception information of the virtual vehicle, the server trains and tests the perception algorithm model of the virtual vehicle.

[0069] The following describes the perception algorithm data loop closure method provided by the exemplary embodiments of this application in conjunction with the above system architecture and with reference to the accompanying drawings. It should be noted that the above system architecture is only shown to facilitate understanding of the spirit and principles of this application, and the embodiments of this application are not limited in any way in this respect.

[0070] It should be noted that before the server receives the intelligent driving algorithm model training and testing request sent by the target terminal, the server needs to pre-build one or more candidate virtual scenarios from a preset set of candidate virtual scenarios, so as to perform the following operations: based on the scene identifier of the obtained test scenario, obtain the relevant operations of the target virtual scenario that matches the corresponding scene identifier.

[0071] Optionally, when constructing candidate virtual scenarios for one or more of the above test scenarios, refer to Figure 3 As shown, taking the server as an example, the execution subject performs the following operations for any scenario in one or more of the above test scenarios, i.e., the first test scenario:

[0072] S301: Based on the scene information of the first test scene collected by the scene information collection device, obtain the inherent scene features contained in the first test scene.

[0073] The scene information collection device can be a drone and / or a collection vehicle, which is an information collection device capable of collecting scene information in various complex scenes. The inherent scene features can also be called static scene features or time-invariant scene features, which are scene features that will not change within a set time range, such as lane lines, traffic signs, latitude and longitude coordinates, road boundary lines, buildings and other scene features.

[0074] S302: Based on the inherent scene features and the dynamic feature set corresponding to the training and testing requests of the intelligent driving algorithm model, construct a candidate virtual scene for the first test scene.

[0075] Among them, the dynamic feature set corresponding to the training and testing requests of intelligent driving algorithm models contains one or more dynamic scene features, also known as time-varying scene features, which are scene features that will not change within a set time range, such as weather conditions, traffic lights, traffic flow, pedestrians, and other scene features.

[0076] For example, based on the method steps S301 to S302 above, the server can construct a virtual reality simulation world, i.e., a corresponding candidate virtual scene. For example, using drones and data collection vehicles, scene information is collected for the first test scene, and a preliminary 3D model is automatically generated. Then, the obtained 3D model is imported into 3D modeling software (such as 3ds Max or AutoCAD) for refinement, thereby obtaining a processed 3D model. Finally, the parts that need to be isolated are extracted, i.e., high-precision maps are extracted from the 3D model, and the information that needs to be semantically layered in the high-precision map is semantically layered. For example, information such as lane lines, traffic signs, latitude and longitude coordinates, and road boundary lines are layered and reflected in the high-precision map dataset.

[0077] Furthermore, based on the 3D model obtained from the above methods and steps, a traffic participant model library can be established, including but not limited to: a building library, a vegetation library, a road surface library, a traffic light library, a traffic sign library, and a material library. It is evident that establishing such a traffic participant model library enhances the adaptability to generating various complex virtual (simulation) scenarios. In addition, by calling the already constructed traffic participant model library, the goal of quickly constructing virtual scenarios can be achieved. Moreover, when performing large-scale virtual road expansion simulations, artificial intelligence (AI) algorithms can be applied to automatically call the already constructed traffic participant model library to generate virtual scenarios, thereby efficiently and cost-effectively recreating virtual scenarios that encompass real-world environmental information. These virtual scenarios include: roads, vegetation, vehicles, buildings, road signs, traffic lights, streetlights, and the lighting logic of buildings and streetlights, etc.

[0078] Optionally, in the process of constructing the virtual scene, physical ray tracing technology can also be used to physically reproduce real environmental details such as weather and lighting, and accurately simulate weather changes and special scenarios.

[0079] In one alternative implementation, the server can also simulate the behavior control of traffic participants such as vehicles and pedestrians in the real world, achieving refined and parameterized traffic participant behavior simulation, and maximally reproducing the actual vehicle test conditions. Thus, by having a data-driven traffic participant control model, it can meet the simulation requirements of autonomous driving and dangerous driving control of participating vehicles.

[0080] In one optional implementation, during the construction of the virtual reality simulation world, the server can generate and render simulated roads based on road surface information (including road surface, lanes, shoulders, etc.). Simultaneously, it generates green vegetation, road traffic facilities, and buildings within a specified area, randomly generating objects such as traffic cones, trash cans, and obstacles, simulating the realistic effect of vegetation shedding leaves like in the real world. It can also generate traffic flow based on route maps. Furthermore, it allows for rich customization of environmental variables, such as various traffic participants, buildings with different regional characteristics, environmental weather, environmental lighting, and water accumulation. The rich customization of scenarios allows for a wider and more comprehensive range of test scenarios, some of which are even difficult or impossible to collect in the real world. Through realistic scenario simulation, a database of intelligent driving perception scenarios can be quickly built.

[0081] It should also be noted that during the execution of step S302, refer to Figure 4 As shown, the server can determine the inherent set type (e.g., Inher.Type1) of the inherent feature set (e.g., Inher.Feat.Set1) to which each inherent scene feature (e.g., lanes, buildings, traffic signs, etc.) belongs, and the dynamic set type (e.g., Dynam.Type1) of the dynamic feature set (e.g., Dynam.Feat.Set1, where the dynamic feature set Dynam.Feat.Set1 includes: traffic lights, weather conditions, pedestrians, traffic flow, etc.). Based on the inherent set type Inher.Type1 and the dynamic set type Dynam.Type1, and the preset scene identifier generation rules Scena.ID.Gener.Rules, the server generates the scene identifier of the first test scene, such as Scena.ID1.

[0082] For example, taking two inherent set types and two dynamic set types as examples, the generated scene identifiers based on the preset identifier generation rules are shown in Table 1:

[0083] Table 1

[0084] Inher.Type1 Dynam.Type1 Scena.ID1 Inher.Type1 Dynam.Type2 Scena.ID2 Inher.Type2 Dynam.Type1 Scena.ID3 Inher.Type2 Dynam.Type2 Scena.ID4

[0085] Furthermore, after obtaining the candidate virtual scenes of the first test scenario, the server can save them to a preset set of candidate virtual scenes.

[0086] Clearly, based on the aforementioned method for constructing candidate virtual scenes and the method for generating scene identifiers for test scenes, after obtaining the candidate virtual scenes and scene identifiers corresponding to various test environments, the server can obtain a preset set of candidate virtual scenes, thereby enabling the subsequent training of the perception algorithm model for virtual vehicles. (See [reference needed]). Figure 5The diagram shown is a flowchart of a data loop closure method for a perception algorithm provided in this application. The specific implementation process of this method is as follows:

[0087] S501: Obtain the vehicle feature set of the vehicle under test and the scene identifier of the test scenario from the intelligent driving algorithm model training and testing request sent by the target terminal.

[0088] Specifically, during step S501, after receiving the intelligent driving algorithm model training and testing request sent by the target terminal, the server can parse the obtained intelligent driving algorithm model training and testing request, and thus obtain the vehicle feature set of the vehicle under test and the scene identifier of the test scenario from the intelligent driving algorithm model training and testing request.

[0089] The vehicle feature set includes at least one feature attribute, which can be a feature of the vehicle body system, braking system, steering system, suspension system, tires, lighting system, cabin, etc., or a visual perception feature attribute of the vehicle under test, i.e., a feature of the visual perception device.

[0090] S502: Based on the various feature attributes contained in the vehicle feature set, construct a virtual vehicle that meets the preset vehicle similarity conditions with the vehicle under test.

[0091] It should be noted that in autonomous driving virtual reality simulation, in addition to the extremely high requirements for the realism of the scene, there are also extremely high requirements for the dynamic performance of the vehicle, including the effect of the road on the chassis, the aerodynamic influence, etc., which require the establishment of a vehicle dynamic model with high degree of freedom and high fidelity.

[0092] Among these methods, vehicle dynamics can be applied to simulate a vehicle's response to driver, road surface, and aerodynamic inputs. This is primarily used to predict and simulate the handling stability, braking performance, ride comfort, power, and fuel economy of a complete vehicle. Through parametric modeling, which defines the vehicle's dimensions, sprung mass, aerodynamics, steering, braking, suspension, tires, etc., a more realistic vehicle dynamic performance can be obtained, accurately reproducing the vehicle's motion behavior. Simultaneously, relying on data-driven modeling technology, vehicle models can be built, for example, to highly realistically reproduce vehicle lighting rendering, instruments, control screens, etc.

[0093] Therefore, based on the method steps of S502 above, after the server obtains the vehicle feature set of the vehicle to be tested and the scene identifier of the test scene, it can construct a virtual vehicle that meets the preset vehicle similarity conditions with the vehicle to be tested based on the various feature attributes contained in the vehicle feature set. For example, the preset vehicle similarity conditions can be specifically: the similarity of various performance parameters (i.e., various feature attributes) between the vehicle to be tested and the constructed virtual vehicle is greater than 99%.

[0094] In one alternative implementation, since the aforementioned feature attributes include the visual perception feature attributes of the vehicle under test, the server can also construct a visual perception device for the virtual vehicle based on the visual perception feature attributes, in the process of constructing a virtual vehicle that meets the preset vehicle similarity conditions with the vehicle under test based on the various feature attributes contained in the vehicle feature set. This allows the constructed virtual vehicle to perceive environmental information like the vehicle under test during subsequent vehicle perception algorithm model training and testing.

[0095] The visual perception device of the virtual vehicle is a virtual sensor, which can be called a visual sensor or a visual image acquisition device. Therefore, in this embodiment, its name or naming is not limited.

[0096] For example, visual perception device simulation constructs a virtual 3D model of the real world and adds color and optical properties to the 3D model based on the real material and texture of the object; it uses physically based rendering and physically based lighting to ensure the conservation of light energy and synthesize realistic image data, while also generalizing and post-processing the image; the visual perception device simulation can adjust the extrinsic, intrinsic, and distortion parameters of the camera, such as: position and attitude, field of view (FOV), resolution, sampling frame rate, resolution distortion, radial distortion, tangential distortion, principal point coordinates, focal length, etc.; it can also output target-level signals, outputting the target's position, attitude, velocity, and other state information, where targets include but are not limited to: pedestrians, any vehicle in traffic, and animals; in addition, the visual perception device also has the ability to adjust dynamic range, blur and sharpen, halos, noise, and other feature attributes.

[0097] S503: Select the target virtual scene that matches the scene identifier from the preset set of candidate virtual scenes.

[0098] In view of the virtual scene construction method in S301 to S302 above, the test scene and its corresponding candidate virtual scene meet the preset scene similarity. Therefore, the target virtual scene and the test scene also meet the preset scene similarity condition. For example, the preset scene similarity condition can be specifically: the scene similarity between the target virtual scene and the test scene is 96%.

[0099] For example, when performing step S503, it is assumed that the preset candidate virtual scene set contains candidate virtual scenes for each of the five test scenes (e.g., scenic area A, park B, street C, urban area D, and mountainous area E), and the candidate virtual scenes and scene identifiers for each of the five test scenes are shown in Table 2:

[0100] Table 2

[0101]

[0102]

[0103] Based on the test scenarios recorded in the table above, as well as their respective candidate virtual scenarios and scenario identifiers, if the server obtains the scenario identifier of the scenario to be tested as Scena.ID3, then based on the above correspondence, the corresponding target virtual scenario can be determined as the candidate virtual scenario Vir.Scen3.

[0104] S504: Drive the virtual vehicle to drive in the target virtual scene, and train and test the perception algorithm model of the virtual vehicle based on the environmental perception information of the virtual vehicle.

[0105] In one alternative implementation, see [link to relevant documentation]. Figure 6 As shown, when executing step S504, the server can obtain road condition information within a set range in front of the virtual vehicle in the target virtual scene through the visual perception device of the virtual vehicle. Then, based on the preset mapping relationship between road condition information and vehicle driving mode, the server can obtain the vehicle driving mode set according to the corresponding road condition information. Based on the vehicle driving mode set according to the corresponding road condition information, the server can drive the virtual vehicle in the target virtual scene. The set range can be set according to the actual situation (e.g., the high-precision visual perception range of the vehicle under test).

[0106] Based on the perception algorithm data closed-loop method described in S501 to S504 above, please refer to... Figure 7 As shown, the server obtains the vehicle feature set Car.Feat.Set and the scene identifier Scena.ID3 of the test scene Test.Scen3 (e.g., XX Park) from the intelligent driving algorithm model training and testing request Algo.Mode.Tra.Request sent by the target terminal. Next, based on the various feature attributes contained in the vehicle feature set Car.Feat.Set1, a virtual vehicle Vir.Car that meets the preset vehicle similarity condition Veh.Sim.Con with the vehicle under test is constructed. Further, from the preset candidate virtual scene set Vir.Scen.Set, the target virtual scene Vir.Scen3 that matches the scene identifier Scena.ID3 is selected. Finally, the virtual vehicle Vir.Car is driven in the target virtual scene Vir.Scen3, and the perception algorithm model of the virtual vehicle Vir.Car is trained and tested based on the environmental perception information Env.Awa.Inform of the virtual vehicle Vir.Car.

[0107] It should be noted that, for reference Figure 8As shown, based on the aforementioned perception algorithm data closed-loop method, the server can efficiently construct a highly realistic simulation environment and various extreme scenarios for autonomous driving within a virtual reality intelligent driving simulation platform. This is achieved by leveraging a real-world traffic database, combining scene elements, weather, and traffic participants. It can also rapidly build a rare dataset spanning tens of thousands of kilometers, comprehensively constructing intelligent driving visual perception scenarios. Furthermore, based on the virtual reality simulation platform, it can replace real-vehicle data collection and annotation, synthesizing the training set required for (visual) perception algorithms and efficiently synthesizing massive amounts of perception annotation data. The relevant data from the visual perception devices mainly includes: images and corresponding semantic segmentation, instance segmentation, depth annotation, object detection annotation, and direct training of (visual) perception algorithm models. The trained (visual) perception algorithm model is then deployed to the virtual reality simulation world for testing, providing real-time feedback on model testing issues, efficiently achieving data closed-loop, and improving the efficiency and security of data collection.

[0108] It should also be noted that, based on the training requirements of the (visual) algorithm perception model, appropriate datasets can be selected from the virtual reality intelligent driving simulation platform for training and testing. Optionally, the data in the virtual reality intelligent driving simulation platform needs to be converted into a data format suitable for the training and testing of the (visual) algorithm perception model.

[0109] In summary, in the perception algorithm data closed-loop method provided in this application embodiment, the vehicle feature set of the vehicle under test and the scene identifier of the test scenario are obtained from the intelligent driving algorithm model training and testing request sent by the target terminal; then, based on the various feature attributes contained in the vehicle feature set, a virtual vehicle that meets the preset vehicle similarity conditions with the vehicle under test is constructed; further, a target virtual scene that matches the scene identifier is selected from the preset candidate virtual scene set; wherein, the target virtual scene and the test scenario meet the preset scene similarity conditions; finally, the virtual vehicle is driven to drive in the target virtual scene, and the perception algorithm model of the virtual vehicle is trained and tested based on the environmental perception information of the virtual vehicle.

[0110] This approach, based on constructing virtual vehicles and a pre-set set of candidate virtual scenarios containing target virtual scenarios, reduces testing costs for extreme conditions through autonomous driving virtual simulation applications. It constructs a highly realistic simulation environment and various extreme scenarios for autonomous driving, builds a traffic database based on real data, solves the long-tail effect data problem in autonomous driving, and avoids the technical drawbacks of related technologies that require real-vehicle road tests to collect road data. This can lead to situations where road data collection is difficult, making it impossible to directly collect the corresponding road data for training and testing perception algorithm models, thus compromising the reliability and comprehensiveness of the trained perception algorithm models. Therefore, it improves the reliability and comprehensiveness of perception algorithm models and, to a certain extent, improves the efficiency of perception algorithm model training and testing.

[0111] Furthermore, based on the same technical concept, embodiments of this application also provide a perception algorithm data closed-loop device, which is used to implement the above-described perception algorithm data closed-loop method flow of embodiments of this application. See also... Figure 9 As shown, the perception algorithm data closed-loop device includes: an acquisition module 901, a construction module 902, a filtering module 903, and a driving module 904, wherein:

[0112] The acquisition module 901 is used to acquire the vehicle feature set of the vehicle under test and the scene identifier of the test scenario from the intelligent driving algorithm model training and testing request sent by the target terminal.

[0113] The construction module 902 is used to construct a virtual vehicle that meets the preset vehicle similarity conditions with the vehicle under test based on the various feature attributes contained in the vehicle feature set.

[0114] The filtering module 903 is used to filter out target virtual scenes that match scene identifiers from a preset set of candidate virtual scenes; wherein the target virtual scene and the test scene meet preset scene similarity conditions.

[0115] The driving module 904 is used to drive the virtual vehicle in the target virtual scene and to train and test the perception algorithm model of the virtual vehicle based on the environmental perception information of the virtual vehicle.

[0116] In one optional embodiment, the preset set of candidate virtual scenes includes candidate virtual scenes of one or more test scenes; wherein, the candidate virtual scenes of the first test scene are obtained in the following manner:

[0117] Based on the scene information of the first test scene collected by the scene information acquisition device, the inherent scene features contained in the first test scene are obtained.

[0118] Based on the inherent features of each scenario and the dynamic feature set corresponding to the training and testing requests of the intelligent driving algorithm model, a candidate virtual scenario for the first test scenario is constructed. The first test scenario is any one of one or more test scenarios.

[0119] In an optional embodiment, during the process of constructing candidate virtual scenes for the first test scene, the construction module 902 is further configured to:

[0120] Determine the inherent set type of the inherent feature set to which each inherent scene feature belongs, and the dynamic set type of the dynamic feature set;

[0121] Based on the inherent collection type and the dynamic collection type, a scene identifier for the first test scenario is generated.

[0122] In one alternative embodiment, the various feature attributes include visual perception feature attributes of the vehicle under test.

[0123] Based on the various feature attributes contained in the vehicle feature set, a virtual vehicle that meets the preset vehicle similarity conditions with the vehicle to be tested is constructed. The construction module 902 is specifically used for:

[0124] Based on visual perception feature attributes, a visual perception device for virtual vehicles is constructed.

[0125] In an optional embodiment, when driving the virtual vehicle in the target virtual scene, the driving module 904 is specifically used for:

[0126] Using the visual perception device of the virtual vehicle, road condition information within a set range in front of the virtual vehicle in the target virtual scene is obtained;

[0127] The vehicle driving mode is set based on the corresponding road condition information, which drives the virtual vehicle to drive in the target virtual scene.

[0128] Based on the same technical concept, embodiments of this application also provide an electronic device that can implement the perception algorithm data closed-loop method flow provided in the above embodiments of this application. In one embodiment, the electronic device can be a server, a terminal device, or other electronic devices. Figure 10 As shown, the electronic device may include:

[0129] At least one processor 1001 and a memory 1002 connected to at least one processor 1001. In this embodiment, the specific connection medium between the processor 1001 and the memory 1002 is not limited. Figure 10 The example shown is the connection between processor 1001 and memory 1002 via bus 1000. Bus 1000 is... Figure 10The connections between other components are indicated by thick lines and are for illustrative purposes only, not as limiting information. The Bus 1000 can be divided into address bus, data bus, control bus, etc., for ease of representation. Figure 10 The term 1001 is represented by a single thick line, but this does not imply that there is only one bus or one type of bus. Alternatively, the processor 1001 can also be called a controller; there are no restrictions on the name.

[0130] In this embodiment, memory 1002 stores instructions executable by at least one processor 1001. By executing the instructions stored in memory 1002, at least one processor 1001 can execute a perception algorithm data loop closure method discussed above. Processor 1001 can implement... Figure 9 The functions of each module in the device shown.

[0131] The processor 1001 is the control center of the device. It can connect to various parts of the control device through various interfaces and lines. By running or executing instructions stored in memory 1002 and calling data stored in memory 1002, the processor can perform various functions and process data, thereby monitoring the device as a whole.

[0132] In one possible design, processor 1001 may include one or more processing units. Processor 1001 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into processor 1001. In some embodiments, processor 1001 and memory 1002 may be implemented on the same chip; in some embodiments, they may also be implemented on separate chips.

[0133] The processor 1001 can be a general-purpose processor, such as a CPU, digital signal processor, application-specific integrated circuit, field-programmable gate array or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the perception algorithm data loop closing method disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0134] Memory 1002, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory 1002 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic memory, magnetic disk, optical disk, etc. Memory 1002 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. In the embodiments of this application, memory 1002 can also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0135] By designing and programming the processor 1001, the code corresponding to the perception algorithm data loop closure method described in the foregoing embodiments can be embedded into the chip, thereby enabling the chip to execute the code during operation. Figure 5 The illustrated embodiment presents the steps of a perception algorithm data loop closure method. How to design and program the processor 1001 is a technique well-known to those skilled in the art and will not be described further here.

[0136] Based on the same inventive concept, embodiments of this application also provide a storage medium storing computer instructions that, when executed on a computer, cause the computer to perform a perception algorithm data closed-loop method as described above.

[0137] In some possible implementations, various aspects of the perception algorithm data closed-loop method provided in this application can also be implemented in the form of a program product, which includes program code. When the program product is run on a device, the program code is used to cause the control device to perform the steps in the perception algorithm data closed-loop method according to various exemplary embodiments of this application described above.

[0138] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0139] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0140] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0141] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A data closure method for a sensing algorithm, characterized in that, include: From the intelligent driving algorithm model training and testing requests sent by the target terminal, obtain the vehicle feature set of the vehicle under test and the scene identifier of the test scenario; Based on the various feature attributes contained in the vehicle feature set, a virtual vehicle that meets the preset vehicle similarity conditions with the vehicle under test is constructed. From a preset set of candidate virtual scenes, a target virtual scene that matches the scene identifier is selected; wherein the target virtual scene and the test scene satisfy a preset scene similarity condition; The virtual vehicle is driven to drive in the target virtual scene, and the perception algorithm model of the virtual vehicle is trained and tested based on the environmental perception information of the virtual vehicle. The preset set of candidate virtual scenes includes candidate virtual scenes for one or more test scenarios; wherein the candidate virtual scenes for the first test scenario are obtained in the following manner: Based on the scene information of the first test scene collected by the scene information collection device, the inherent scene features contained in the first test scene are obtained. Based on the inherent scene features and the dynamic feature set corresponding to the training and testing requests of the intelligent driving algorithm model, a candidate virtual scene of the first test scene is constructed, wherein the first test scene is any one of the one or more test scenes; The process of constructing the candidate virtual scene for the first test scene further includes: Determine the inherent set type of the inherent feature set to which each inherent scene feature belongs, and the dynamic set type of the dynamic feature set; Based on the inherent set type and the dynamic set type, a scene identifier for the first test scenario is generated.

2. The method as described in claim 1, characterized in that, The various feature attributes include the visual perception feature attributes of the vehicle under test; The step of constructing a virtual vehicle that meets preset vehicle similarity conditions with the vehicle under test based on the various feature attributes included in the vehicle feature set includes: Based on the aforementioned visual perception feature attributes, a visual perception device for the virtual vehicle is constructed.

3. The method as described in claim 2, characterized in that, The process of driving the virtual vehicle in the target virtual scene includes: The road condition information within a set range in front of the virtual vehicle in the target virtual scene is obtained through the visual perception device of the virtual vehicle. The vehicle driving mode is set according to the corresponding road condition information, which drives the virtual vehicle to drive in the target virtual scene.

4. A data closed-loop device for a sensing algorithm, characterized in that, include: The acquisition module is used to obtain the vehicle feature set of the vehicle under test and the scene identifier of the test scenario from the intelligent driving algorithm model training and testing request sent by the target terminal. The construction module is used to construct a virtual vehicle that meets the preset vehicle similarity conditions with the vehicle under test based on the various feature attributes contained in the vehicle feature set. The filtering module is used to filter out target virtual scenes that match the scene identifier from a preset set of candidate virtual scenes; wherein the target virtual scene and the test scene meet a preset scene similarity condition; The driving module is used to drive the virtual vehicle in the target virtual scene and to train and test the perception algorithm model of the virtual vehicle based on the environmental perception information of the virtual vehicle. The preset set of candidate virtual scenes includes candidate virtual scenes for one or more test scenarios; wherein the candidate virtual scenes for the first test scenario are obtained in the following manner: Based on the scene information of the first test scene collected by the scene information collection device, the inherent scene features contained in the first test scene are obtained. Based on the inherent scene features and the dynamic feature set corresponding to the training and testing requests of the intelligent driving algorithm model, a candidate virtual scene of the first test scene is constructed, wherein the first test scene is any one of the one or more test scenes; In the process of constructing the candidate virtual scene of the first test scene, the construction module is further configured to: Determine the inherent set type of the inherent feature set to which each inherent scene feature belongs, and the dynamic set type of the dynamic feature set; Based on the inherent set type and the dynamic set type, a scene identifier for the first test scenario is generated.

5. The apparatus as described in claim 4, characterized in that, The various feature attributes include the visual perception feature attributes of the vehicle under test; The virtual vehicle that meets the preset vehicle similarity conditions with the vehicle under test is constructed based on the various feature attributes included in the vehicle feature set. The construction module is specifically used for: Based on the aforementioned visual perception feature attributes, a visual perception device for the virtual vehicle is constructed.

6. The apparatus as claimed in claim 5, characterized in that, When driving the virtual vehicle in the target virtual scene, the driving module is specifically used for: The road condition information within a set range in front of the virtual vehicle in the target virtual scene is obtained through the visual perception device of the virtual vehicle. The vehicle driving mode is set according to the corresponding road condition information, which drives the virtual vehicle to drive in the target virtual scene.

7. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1-3.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-3.

Images