Scene data storage method, device, equipment and medium

Abstract

The present invention discloses a scene data storage method, device, equipment and medium, involving multiple data processing threads and multiple scene simulation models, each data processing thread is associated with a message queue, and the method includes: when receiving multiple initial scene data output by any scene simulation model, calculating the queue hash value corresponding to each initial scene data; sending each initial scene data to the message queue corresponding to the queue hash value; obtaining the multiple initial scene data from the message queue through the data processing thread and integrating the data to obtain target scene data; writing the target scene data to the target database through the data processing thread, thereby improving the writing efficiency of fragmented data such as scene data.

Description

Technical Field

[0001] The present invention relates to the technical field of scene data transmission, and in particular to a scene data storage method, device, equipment and medium. Background Art

[0002] With the development of society and the continuous improvement of various technologies, people's requirements for autonomous driving are also increasing. For autonomous driving related technologies, to ensure safety, extensive testing is required to verify the feasibility of this technology. However, road testing is costly and cannot be carried out in the early stages of technology development. Therefore, simulation testing has undoubtedly become the most common and practical testing method.

[0003] In the existing simulation test process, it is usually necessary to simulate and test multiple simulation scenarios in each scenario. During the test process of each simulation scenario, a large amount of fragmented data will be generated for subsequent analysis.

[0004] To this end, existing solutions usually write data to the database as it is generated. However, due to the high degree of data fragmentation and the large amount of data, the database writing efficiency is limited. Summary of the Invention

[0005] The present invention provides a scene data storage method, device, equipment and medium, which solves the technical problem that existing solutions usually write data to the database when generating data, but the database writing efficiency is limited due to the high degree of data fragmentation and large amount of data.

[0006] A first aspect of the present invention provides a scene data storage method, involving multiple data processing threads and multiple scene simulation models, each of the data processing threads is associated with a message queue, and the method includes:

[0007] When receiving a plurality of initial scene data output by any of the scene simulation models, calculating a queue hash value corresponding to each of the initial scene data;

[0008] Sending each of the initial scene data to the message queue corresponding to the queue hash value;

[0009] Acquiring the plurality of initial scene data from the message queue through the data processing thread and integrating the data to obtain target scene data;

[0010] The target scene data is written into a target database through the data processing thread.

[0011] Optionally, when receiving a plurality of initial scene data output by any of the scene simulation models, the step of calculating a queue hash value corresponding to each of the initial scene data includes:

[0012] When receiving a plurality of initial scene data output by any of the scene simulation models, obtaining an algorithm identifier corresponding to each initial scene data;

[0013] Selecting a first target hash function from a plurality of preset hash functions according to the algorithm identifier;

[0014] The first target hash function is used to calculate the queue hash value corresponding to each of the initial scene data.

[0015] Optionally, it also involves a message queue to be grouped and a data grouping component; when receiving multiple initial scenario data output by any of the scenario simulation models, the step of calculating the queue hash value corresponding to each of the initial scenario data includes:

[0016] When receiving a plurality of initial scene data output by any of the scene simulation models, sending all of the initial scene data to the message queue to be grouped;

[0017] Obtaining the initial scene data from the to-be-grouped message queue by the data grouping component according to a preset first time interval;

[0018] Grouping the initial scene data by the data grouping component to obtain a plurality of new initial scene data and marking corresponding algorithm identifiers;

[0019] selecting, by the data grouping component, a second target hash function from a plurality of preset hash functions according to the algorithm identifier;

[0020] The queue hash value corresponding to each of the new initial scene data is calculated by the data grouping component using the second target hash function.

[0021] Optionally, the step of obtaining the plurality of initial scene data from the message queue through the data processing thread and integrating the data to obtain the target scene data includes:

[0022] Acquiring, by the data processing thread, a plurality of the initial scene data and corresponding partition identifiers from the message queue according to a preset second time interval;

[0023] Clustering the initial scene data according to each of the partition identifiers to obtain at least one clustered data cluster;

[0024] The initial scene data in the clustered data clusters are integrated to obtain target scene data.

[0025] Optionally, the target database includes multiple data partitions; the step of writing the target scene data to the target database through the data processing thread includes:

[0026] Writing the target scene data into a target database according to the partition identifier by the data processing thread;

[0027] The target database is used to receive the target scene data and merge the target scene data into each of the data partitions according to the partition identifiers.

[0028] Optionally, the method further includes:

[0029] When receiving the updated scenario data sent by the updated scenario simulation model, a preset updated hash function is used to calculate the updated hash value corresponding to the updated scenario data;

[0030] Traversing all the message queues to determine whether there is a message queue associated with the updated hash value;

[0031] If so, sending the updated scene data to the message queue associated with the updated hash value;

[0032] If not, selecting the associated updated data partition from the target database according to the updated hash function, and generating a new partition identifier corresponding to the updated data partition;

[0033] An update message queue and a corresponding update data processing thread associated with the update hash value are created, and the update scene data is marked with the new partition identifier and then sent to the update message queue.

[0034] Optionally, the method further includes:

[0035] When receiving a plurality of specific scene data respectively output by a plurality of the scene simulation models, sending all of the specific scene data to the data processing thread;

[0036] Grouping all of the specific scene data by the data processing thread to obtain a plurality of specific scene data groups corresponding to the scene simulation model;

[0037] Each of the specific scenario data groups is written into the target database through the data processing thread.

[0038] Optionally, the method further includes:

[0039] When a data query request is received, obtaining a target algorithm identifier corresponding to the data query request;

[0040] The target database is searched according to the target algorithm identifier to obtain all target scene data carrying the target algorithm identifier.

[0041] A second aspect of the present invention provides a scene data storage device, involving multiple data processing threads and multiple scene simulation models, each of the data processing threads is associated with a message queue, and the device includes:

[0042] A queue hash value calculation module, configured to calculate a queue hash value corresponding to each of the initial scene data upon receiving a plurality of initial scene data output by any of the scene simulation models;

[0043] A queue cache module, configured to send each of the initial scene data to a message queue corresponding to the queue hash value;

[0044] A data integration module, configured to obtain the plurality of initial scene data from the message queue through the data processing thread and perform data integration to obtain target scene data;

[0045] A data writing module is used to write the target scene data into a target database through the data processing thread.

[0046] The third aspect of the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the scene data storage method as described in any one of the first aspects of the present invention.

[0047] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed, implements the scene data storage method as described in any one of the first aspects of the present invention.

[0048] It can be seen from the above technical solutions that the present invention has the following advantages:

[0049] When the present invention receives multiple initial scenario data output by any scenario simulation model, it calculates the queue hash value corresponding to each initial scenario data to determine the corresponding message queue. After sending each initial scenario data to its corresponding message queue, the data processing thread further obtains multiple initial scenario data from its associated message queue according to a preset time interval, integrates the obtained multiple initial scenario data, and obtains a set of target scenario data. Finally, the data processing thread writes the target scenario data to the corresponding data partition in the target database. This improves the writing efficiency of fragmented data such as scenario data. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0051] Figure 1 A flowchart of a scene data storage method provided in the first embodiment of the present invention;

[0052] Figure 2 A flowchart of a scene data storage method provided in the second embodiment of the present invention;

[0053] Figure 3 A schematic diagram of a flow chart of a scene data storage method provided by an embodiment of the present invention;

[0054] Figure 4 This is a structural block diagram of a scene data storage device provided in Example 3 of the present invention. DETAILED DESCRIPTION

[0055] Embodiments of the present invention provide a scene data storage method, apparatus, device and medium for solving the technical problem that existing solutions generally write data to a database while generating data, but the database writing efficiency is limited due to the high degree of data fragmentation and large amount of data.

[0056] In order to make the purpose, features, and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0057] See also Figure 1 , Figure 1 This is a flowchart of the steps of a scene data storage method provided in Example 1 of the present invention.

[0058] The present invention provides a scene data storage method, involving multiple data processing threads and multiple scene simulation models, each data processing thread is associated with a message queue, and the method includes:

[0059] Step 101: When receiving multiple initial scene data output by any scene simulation model, calculate the queue hash value corresponding to each initial scene data;

[0060] A scenario simulation model refers to a model that pre-stores multiple operating scenarios and can simulate driving or simulation experiments on the control algorithm or simulation program of the autonomous driving system. The operating scenarios can be various vehicle driving conditions, such as abnormal scenarios such as battery power loss and system freeze, and can also be applied to various driving conditions such as turns, traffic jams or complex environments such as urban villages. Specifically, it can be applied to situations such as pre-road testing of the autonomous driving system of an autonomous driving vehicle. For example, through multiple scenario simulation models, multiple speed planning algorithms and path planning algorithms for driving are simulated in different operating scenarios. In each scenario simulation model, multiple operating scenarios can be simulated in parallel based on the same set of algorithms to obtain multiple initial scenario data.

[0061] The initial scenario data refers to the fragmented data generated by the scenario simulation model when simulating the operation of the algorithm in the autonomous driving system under different operating scenarios. Each fragmented data can carry a corresponding algorithm identifier, and each fragmented data can correspond to a different simulation scenario.

[0062] In an embodiment of the present invention, the same scenario simulation model can simultaneously simulate the operation of the same algorithm in different scenarios to generate multiple initial scenario data. When multiple initial scenario data output by any scenario simulation model are received, the corresponding hash function can be selected according to the source of each initial scenario data, and the hash function can be used to calculate the queue hash value corresponding to each initial scenario data.

[0063] It should be noted that the queue hash value refers to the hash value obtained by calculating the initial scene data using a hash function. The hash value corresponds to the message queue one-to-one. Hash functions can include but are not limited to addition hash, bitwise hash, multiplication hash, division hash, table lookup hash or hybrid hash function algorithms.

[0064] Step 102: Send each initial scene data to the message queue corresponding to the queue hash value;

[0065] The message queue refers to a linked list that can store messages, and is used to store the messages in the order in which the initial scene data is received.

[0066] After calculating the queue hash value corresponding to each initial scene data, in order to improve the processing efficiency of the initial scene data, each initial scene data can be sent to the message queue according to the queue hash value distribution.

[0067] Step 103: Obtain multiple initial scene data from the message queue through the data processing thread and integrate the data to obtain target scene data;

[0068] The data processing thread refers to one of the threads in the data processor, which is used to integrate the initial scene data in the associated message queue.

[0069] After the initial scene data is sent to each message queue according to the queue hash value distribution, due to the small amount of data and the large number of initial scene data, writing it directly to the database will cause the database to need to merge partitions multiple times, increasing the database load. To solve this problem, a data processing thread can obtain multiple initial scene data from the associated message queue according to a preset time interval, and after integrating the multiple initial scene data, obtain the target scene data.

[0070] Step 104: Write the target scene data into the target database through the data processing thread.

[0071] After the data processing thread integrates multiple initial scene data to obtain the target scene data, the target scene data is written to the corresponding data partition of the target database, thereby ensuring that multiple scene data corresponding to the same algorithm are stored in the same data partition, while completing the efficient storage of scene data and reducing the load of database merging partitions.

[0072] In an embodiment of the present invention, upon receiving multiple initial scenario data outputted by any scenario simulation model, the queue hash value corresponding to each initial scenario data is calculated to determine the corresponding message queue. After each initial scenario data is sent to its corresponding message queue, a data processing thread further retrieves multiple initial scenario data from its associated message queue according to a preset time interval. The retrieved multiple initial scenario data are then integrated to obtain a set of target scenario data. Finally, the data processing thread writes the target scenario data to the corresponding data partition within the target database. This improves the writing efficiency of fragmented data such as scenario data.

[0073] See also Figure 2 , Figure 2 This is a flowchart of the steps of a scene data storage method provided in Example 2 of the present invention.

[0074] The present invention provides a scene data storage method, involving multiple data processing threads and multiple scene simulation models, each data processing thread is associated with a message queue, and the method includes:

[0075] Step 201: When receiving multiple initial scene data output by any scene simulation model, calculate the queue hash value corresponding to each initial scene data;

[0076] In one example of the present invention, step 201 may include the following sub-steps:

[0077] When receiving multiple initial scene data output by any scene simulation model, obtaining the algorithm identifier corresponding to each initial scene data;

[0078] Selecting a first target hash function from a plurality of preset hash functions according to the algorithm identifier;

[0079] The first target hash function is used to calculate the queue hash value corresponding to each initial scene data.

[0080] In specific implementation, in order to reduce the testing cost of the autonomous driving system, the algorithms or programs used by the autonomous driving system can be simulated before road testing. Different models of autonomous driving vehicles can be created in the scenario simulation model, and the corresponding algorithms or programs can be loaded. The operation of the algorithm in different driving scenarios and operating environments can be simulated, and the initial scenario data under the corresponding scenario can be output.

[0081] When receiving multiple initial scenario data output by any scenario simulation model, the algorithm identifier carried by each initial scenario data can be obtained to determine the source of the initial scenario data, and then multiple preset hash functions can be retrieved according to the algorithm identifier to determine the first target hash function corresponding to each algorithm identifier; and then the queue hash value corresponding to the initial scenario data is calculated using the corresponding first target hash function to determine the message queue that will serve as the subsequent data sending target.

[0082] It should be noted that since the same scenario simulation model can run multiple simulation programs or execute multiple algorithms simultaneously in multiple different scenarios, and each initial scenario data can be marked with the corresponding algorithm identifier when output, each scenario simulation model can be associated with a hash function respectively, and the hash function can be associated with a data partition of the database, and the hash value calculated by the hash function is associated with the message queue.

[0083] In another example of the present invention, the method further involves a message queue to be grouped and a data grouping component, and step 201 may further include the following sub-steps:

[0084] When receiving multiple initial scene data output by any scene simulation model, all the initial scene data are sent to the message queue to be grouped;

[0085] Obtaining initial scene data from a queue of messages to be grouped according to a preset first time interval by a data grouping component;

[0086] The initial scene data is grouped by a data grouping component to obtain a plurality of new initial scene data and marked with corresponding algorithm identifiers;

[0087] Selecting, by the data grouping component, a second target hash function from a plurality of preset hash functions according to the algorithm identifier;

[0088] The queue hash value corresponding to each new initial scene data is calculated by using the second target hash function through the data grouping component.

[0089] This embodiment also involves a message queue to be grouped and a data grouping component. For a scenario simulation model that cannot perform algorithm identification marking when outputting initial scenario data, when receiving multiple initial scenario data output by the scenario simulation model, all initial scenario data can be sent to a separate message queue to be grouped. The data grouping component obtains multiple initial scenario data from the message queue to be grouped according to a preset first time interval. The data grouping component further groups the initial scenario data according to predetermined rules to obtain multiple new initial scenario data and marks each new initial scenario data with an algorithm identification. Finally, the data grouping component selects a second target hash function from multiple preset hash functions according to the algorithm identification, and the data grouping component uses the second target hash function to calculate the queue hash value corresponding to each new initial scenario data.

[0090] The predetermined rule may be grouping according to the source of the initial scene data, or grouping several initial scene data according to the order in which the initial scene data are obtained, etc., which is not limited in the embodiment of the present invention.

[0091] Step 202: Send each initial scene data to the message queue corresponding to the queue hash value;

[0092] After calculating and obtaining the queue hash values ​​corresponding to each piece of initial scene data, the initial scene data may be sent to the corresponding message queue according to each queue hash value.

[0093] Step 203: obtaining a plurality of initial scene data and corresponding partition identifiers from a message queue according to a preset second time interval through a data processing thread;

[0094] After the initial scene data is sent to the corresponding message queues, since each message queue can be associated with its corresponding queue hash value, and the hash function corresponding to the queue hash value can be associated with the corresponding data partition, further processing can be performed through the data processing thread on the data processor to obtain multiple initial scene data and corresponding partition identifiers from the associated message queue according to the preset second time interval, thereby determining the data partition corresponding to the initial scene data.

[0095] In which, multiple initial scenario data in different data partitions can exist simultaneously in the same message queue. For example, the database partition is divided according to every 100 simulation experiments, and 10 experiments with the same last number are regarded as a partition. For example, the simulation experiment data numbered 001, 011, and 021 are in one partition. Then the hash function at this time is hash(x)=x mod10, ensuring that all data in the same partition will be divided into the same queue. At the same time, because there is a lot of simulation data, there will only be data from two different partitions in the same message queue (for example: 002 and 102 belong to different partitions, but will be in the same queue). This also ensures that the message queue can be reused.

[0096] It should be noted that the first time interval or the second time in the present invention can be implemented in the form of a time window.

[0097] Step 204: clustering the initial scene data according to each partition identifier to obtain at least one clustered data cluster;

[0098] In this embodiment, after obtaining each partition identifier, since initial scene data belonging to multiple data partitions may exist on the same message queue, in order to achieve subsequent data integration, the initial scene data can be clustered according to each partition identifier to obtain clustered data clusters corresponding to the partition identifier.

[0099] Step 205: Integrate the initial scene data in the clustered data cluster to obtain target scene data;

[0100] After obtaining multiple clustered data clusters, the data processing thread can respectively integrate the initial scene data in each clustered data cluster to obtain a single target scene data.

[0101] Step 206: Write the target scene data into the target database through the data processing thread.

[0102] Optionally, the target database includes multiple data partitions, and step 206 may include the following sub-steps:

[0103] Write the target scene data to the target database according to the partition identifier through the data processing thread;

[0104] The target database is used to receive target scene data and merge the target scene data into each data partition according to the partition identifier.

[0105] In an embodiment of the present invention, after the data processing thread integrates the initial scene data to obtain the target scene data, it can write the target scene data to the target database according to the partition identifier. At the same time, after receiving the target scene data, the target database merges the target scene data into each data partition according to the partition identifier, thereby completing the storage of the scene data of the scene simulation model.

[0106] The target database in this embodiment refers to a column-based database, such as ClickHouse.

[0107] Furthermore, the method may further include the following steps S11-S15:

[0108] S11. When receiving the updated scenario data sent by the updated scenario simulation model, using a preset updated hash function to calculate an updated hash value corresponding to the updated scenario data;

[0109] S12. Traverse all message queues to determine whether there is a message queue associated with the updated hash value;

[0110] S13. If so, send the updated scene data to the message queue associated with the updated hash value;

[0111] In one example of the present invention, if a new scenario simulation model is added, it can be used as the updated scenario simulation model. When updated scenario data sent by the updated scenario simulation model is received, a preset update hash function can be used to calculate the update hash value corresponding to the updated scenario data. By traversing all existing message queues, it is determined whether there is a message queue associated with the update hash value. Specifically, this can be determined by comparing the queue hash value with the update hash value to see whether it is equal.

[0112] If they are equal, it indicates that there is a message queue associated with the updated hash value. At this time, the updated scene data can be sent to the message queue associated with the updated hash value.

[0113] S14. If it does not exist, select the associated updated data partition from the target database according to the update hash function, and generate a new partition identifier corresponding to the updated data partition;

[0114] S15. Create an update message queue associated with the updated hash value and a corresponding update data processing thread, mark the updated scene data with a new partition identifier, and send it to the update message queue.

[0115] In another example of the present invention, if the update hash value is not equal to any queue hash value, it indicates that there is no message queue associated with it at this time. The associated update data partition can be further determined from the target database according to the update hash function, and a corresponding new partition identifier can be generated; an update message queue associated with the update hash value and a corresponding update data processing thread are created, and the update scene data is marked with the new partition identifier, and the update scene data is sent to the update message queue, so that the subsequent update data processing thread can obtain and integrate the data from the update message queue and write it to the update data partition of the target database.

[0116] The updated hash function may be selected from a plurality of preset hash functions or may be specified by a technician, and this embodiment does not impose any limitation on this.

[0117] Optionally, the method may further include the following steps S21-S23:

[0118] S21. When receiving multiple specific scene data outputted by multiple scene simulation models respectively, sending all the specific scene data to the data processing thread;

[0119] S22, grouping all specific scene data through a data processing thread to obtain multiple specific scene data groups corresponding to the scene simulation model;

[0120] S23. Write each specific scenario data group into the target database through the data processing thread.

[0121] In an optional embodiment of the present invention, when multiple specific scenario data output by all or multiple scenario simulation models are received, all specific scenario data are sent to the data processing thread, and the data processing thread groups all specific scenario data according to preset rules to obtain multiple specific scenario data groups corresponding to the scenario simulation models, and then the data processing thread directly writes each specific scenario data group to the target database.

[0122] It should be noted that, when the amount of data is small, the above-mentioned method of directly writing into the database can be adopted to reduce resource consumption.

[0123] In another example of the present invention, the method further includes the following steps S31-S32:

[0124] S31. When a data query request is received, obtain a target algorithm identifier corresponding to the data query request;

[0125] S32: Search the target database according to the target algorithm identifier to obtain all target scene data carrying the target algorithm identifier.

[0126] In an embodiment of the present invention, if a user needs to perform a data query, they can send a data query request to the corresponding virtual device of the present invention. Upon receiving the data query request, the target algorithm identifier corresponding to the data query request can be obtained, and the target database can be searched according to the target algorithm identifier, thereby obtaining all target scene data corresponding to the target algorithm identifier from the same partition.

[0127] See also Figure 3 , Figure 3 A schematic flow chart of a scene data storage method provided in an embodiment of the present invention.

[0128] like Figure 3 As shown, the scenario simulation model of the present invention is a simulation experiment program. The simulation experiment program produces a total of 11 sets of data. These 11 sets of data belong to five partitions of the same size. If they are written directly to the database during production, then:

[0129] For the simulation experiment, it is necessary to process 11 pieces of data and interact with the database 11 times, which requires a total of 11 times of time to process the data and 11 times of time to interact with the database.

[0130] For the database, each insertion requires a merge operation on the data partition, which requires 11 database partition merges and a total of 11 partition merge times.

[0131] If we use the message queue model we proposed, then:

[0132] For the simulation experiment, each set of data needs to interact with the message queue once, and a total of 11 interactions with the message queue are required.

[0133] For the data processing end, it is necessary to obtain and process 11 pieces of data from the message queue and interact with the database 5 times at the same time, which takes a total of 11 times of interaction with the message queue, 11 times of data processing, and 5 times of interaction with the database.

[0134] For the database, each insertion requires a merge operation on the data partition, which requires five database partition merges and a total of five partition merge times.

[0135] In production scenarios, however, the primary time consumption comes from data processing and database partition merging. The time spent interacting with the database and the message queue is orders of magnitude smaller than these two times. Therefore, with the original solution, the simulation experiment would take 11 data processing times and the database would take 11 partition merging times. With the new solution, the simulation experiment takes virtually no time, while the database would take only five partition merging times. This significantly reduces simulation time and the database load by reducing the number of partition merges. Regarding resource consumption, the introduction of the message queue and data processor consumes essentially no resources compared to the computing resources consumed by the simulation. In fact, in production simulations, over a thousand data items are generated per second. Using this solution significantly reduces the number of database partition merges and the time it takes to process data in the simulation.

[0136] In an embodiment of the present invention, upon receiving multiple initial scenario data outputted by any scenario simulation model, the queue hash value corresponding to each initial scenario data is calculated to determine the corresponding message queue. After each initial scenario data is sent to its corresponding message queue, a data processing thread further retrieves multiple initial scenario data from its associated message queue according to a preset time interval. The retrieved multiple initial scenario data are then integrated to obtain a set of target scenario data. Finally, the data processing thread writes the target scenario data to the corresponding data partition within the target database. This improves the writing efficiency of fragmented data such as scenario data.

[0137] See also Figure 4 , Figure 4 This is a structural block diagram of a scene data storage device provided in Example 3 of the present invention.

[0138] An embodiment of the present invention provides a scene data storage device, involving multiple data processing threads and multiple scene simulation models, each data processing thread is associated with a message queue, and the device includes:

[0139] The queue hash value calculation module 401 is used to calculate the queue hash value corresponding to each initial scene data when receiving multiple initial scene data output by any scene simulation model;

[0140] The queue cache module 402 is used to send each initial scene data to the message queue corresponding to the queue hash value;

[0141] The data integration module 403 is used to obtain multiple initial scene data from the message queue through the data processing thread and integrate the data to obtain the target scene data;

[0142] The data writing module 404 is configured to write the target scene data into the target database through the data processing thread.

[0143] Optionally, the queue hash value calculation module 401 is specifically configured to:

[0144] When receiving multiple initial scene data output by any scene simulation model, obtaining the algorithm identifier corresponding to each initial scene data;

[0145] Selecting a first target hash function from a plurality of preset hash functions according to the algorithm identifier;

[0146] The first target hash function is used to calculate the queue hash value corresponding to each initial scene data.

[0147] Optionally, it also involves a message queue to be grouped and a data grouping component; the queue hash value calculation module 401 is specifically used to:

[0148] When receiving multiple initial scene data output by any scene simulation model, all the initial scene data are sent to the message queue to be grouped;

[0149] Obtaining initial scene data from a queue of messages to be grouped according to a preset first time interval by a data grouping component;

[0150] The initial scene data is grouped by a data grouping component to obtain a plurality of new initial scene data and marked with corresponding algorithm identifiers;

[0151] Selecting, by the data grouping component, a second target hash function from a plurality of preset hash functions according to the algorithm identifier;

[0152] The queue hash value corresponding to each new initial scene data is calculated by using the second target hash function through the data grouping component.

[0153] Optionally, the data integration module 403 is specifically configured to:

[0154] Obtaining, by the data processing thread, a plurality of initial scene data and corresponding partition identifiers from the message queue according to a preset second time interval;

[0155] Clustering the initial scene data according to each partition identifier to obtain at least one clustered data cluster;

[0156] The initial scene data in the clustered data cluster is integrated to obtain the target scene data.

[0157] Optionally, the target database includes multiple data partitions; the data writing module 404 is specifically used to:

[0158] Write the target scene data to the target database according to the partition identifier through the data processing thread;

[0159] The target database is used to receive target scene data and merge the target scene data into each data partition according to the partition identifier.

[0160] Optionally, the device further comprises:

[0161] An update hash value calculation module is used to calculate an update hash value corresponding to the update scenario data using a preset update hash function when receiving the update scenario data sent by the update scenario simulation model;

[0162] The queue traversal module is used to traverse all message queues to determine whether there is a message queue associated with the updated hash value;

[0163] An update scene data sending module, configured to send the update scene data to a message queue associated with the update hash value if the update scene data exists;

[0164] An update data partition determination module is used to select the associated update data partition from the target database according to the update hash function if it does not exist, and generate a new partition identifier corresponding to the update data partition;

[0165] The update message queue creation and sending module is used to create an update message queue associated with the update hash value and the corresponding update data processing thread, and use the new partition identifier to mark the updated scene data and send it to the update message queue.

[0166] Optionally, the device further comprises:

[0167] A specific scene data sending module is used to send all the specific scene data to the data processing thread when receiving multiple specific scene data output by multiple scene simulation models respectively;

[0168] A data grouping module is used to group all specific scene data through a data processing thread to obtain multiple specific scene data groups corresponding to the scene simulation model;

[0169] The scenario data group writing module is used to write each specific scenario data group to the target database through the data processing thread.

[0170] Optionally, the device further comprises:

[0171] A target algorithm identifier acquisition module is used to obtain the target algorithm identifier corresponding to the data query request when a data query request is received;

[0172] The target scene data query module is used to search the target database according to the target algorithm identifier to obtain all target scene data carrying the target algorithm identifier.

[0173] An embodiment of the present invention further provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the scene data storage method as described in any embodiment of the present invention.

[0174] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed, the scene data storage method as described in any embodiment of the present invention is implemented.

[0175] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0176] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0177] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0178] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0179] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0180] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A scene data storage method, characterized in that: Involving multiple data processing threads and multiple scenario simulation models, each of the data processing threads is associated with a message queue, and the method includes: Upon receiving a plurality of initial scenario data output by any of the scenario simulation models, calculating a queue hash value corresponding to each of the initial scenario data; the initial scenario data refers to fragmented data generated by the scenario simulation model when simulating the operation of the algorithm within the autonomous driving system under different operating scenarios; Sending each of the initial scene data to the message queue corresponding to the queue hash value; Acquiring the plurality of initial scene data from the message queue through the data processing thread and integrating the data to obtain target scene data; Writing the target scene data into a target database through the data processing thread; If the initial scene data has been marked with the algorithm identifier corresponding to the scene simulation model, then when receiving multiple initial scene data output by any of the scene simulation models, the step of calculating the queue hash value corresponding to each of the initial scene data includes: When receiving a plurality of initial scene data output by any of the scene simulation models, obtaining an algorithm identifier corresponding to each initial scene data; Selecting a first target hash function from a plurality of preset hash functions according to the algorithm identifier; Calculating the queue hash value corresponding to each of the initial scene data using the first target hash function; The method further involves a message queue to be grouped and a data grouping component; if the initial scenario data is not marked with the algorithm identifier corresponding to the scenario simulation model, then when receiving multiple initial scenario data output by any of the scenario simulation models, the step of calculating the queue hash value corresponding to each of the initial scenario data includes: When receiving a plurality of initial scene data output by any of the scene simulation models, sending all of the initial scene data to the message queue to be grouped; Obtaining the initial scene data from the to-be-grouped message queue by the data grouping component according to a preset first time interval; Grouping the initial scene data by the data grouping component to obtain a plurality of new initial scene data and marking corresponding algorithm identifiers; selecting, by the data grouping component, a second target hash function from a plurality of preset hash functions according to the algorithm identifier; The queue hash value corresponding to each of the new initial scene data is calculated by the data grouping component using the second target hash function.

2. The method according to claim 1, characterized in that The step of obtaining the plurality of initial scene data from the message queue and integrating the data to obtain target scene data through the data processing thread includes: Acquiring, by the data processing thread, a plurality of the initial scene data and corresponding partition identifiers from the message queue according to a preset second time interval; Clustering the initial scene data according to each of the partition identifiers to obtain at least one clustered data cluster; The initial scene data in the clustered data clusters are integrated to obtain target scene data.

3. The method according to claim 2, characterized in that The target database includes multiple data partitions; the step of writing the target scene data into the target database through the data processing thread includes: Writing the target scene data into a target database according to the partition identifier by the data processing thread; The target database is used to receive the target scene data and merge the target scene data into each of the data partitions according to the partition identifiers.

4. The method according to any one of claims 1 to 3, characterized in that The method further comprises: When receiving the updated scenario data sent by the updated scenario simulation model, a preset updated hash function is used to calculate the updated hash value corresponding to the updated scenario data; Traversing all the message queues to determine whether there is a message queue associated with the updated hash value; If so, sending the updated scene data to the message queue associated with the updated hash value; If not, selecting the associated updated data partition from the target database according to the updated hash function, and generating a new partition identifier corresponding to the updated data partition; An update message queue and a corresponding update data processing thread associated with the update hash value are created, and the update scene data is marked with the new partition identifier and then sent to the update message queue.

5. The method according to claim 1, wherein The method further comprises: When receiving a plurality of specific scene data respectively output by a plurality of the scene simulation models, sending all of the specific scene data to the data processing thread; Grouping all of the specific scene data by the data processing thread to obtain a plurality of specific scene data groups corresponding to the scene simulation model; Each of the specific scenario data groups is written into the target database through the data processing thread.

6. The method according to claim 1, characterized in that The method further comprises: When a data query request is received, obtaining a target algorithm identifier corresponding to the data query request; The target database is searched according to the target algorithm identifier to obtain all target scene data carrying the target algorithm identifier.

7. A scene data storage device, characterized in that: Involving multiple data processing threads and multiple scenario simulation models, each of the data processing threads is associated with a message queue, and the device includes: a queue hash value calculation module, configured to calculate, upon receiving a plurality of initial scenario data output by any of the scenario simulation models, a queue hash value corresponding to each of the initial scenario data; the initial scenario data being fragmented data generated by the scenario simulation model when simulating the operation of an algorithm within the autonomous driving system under different operating scenarios; A queue cache module, configured to send each of the initial scene data to a message queue corresponding to the queue hash value; A data integration module, configured to obtain the plurality of initial scene data from the message queue through the data processing thread and perform data integration to obtain target scene data; A data writing module, configured to write the target scene data into a target database through the data processing thread; If the initial scene data has been marked with the algorithm identifier corresponding to the scene simulation model, the queue hash value calculation module is specifically configured to: When receiving a plurality of initial scene data output by any of the scene simulation models, obtaining an algorithm identifier corresponding to each initial scene data; Selecting a first target hash function from a plurality of preset hash functions according to the algorithm identifier; Calculating the queue hash value corresponding to each of the initial scene data using the first target hash function; The device also involves a message queue to be grouped and a data grouping component; if the initial scene data is not marked with the algorithm identifier corresponding to the scene simulation model, the queue hash value calculation module is specifically used to: When receiving a plurality of initial scene data output by any of the scene simulation models, sending all of the initial scene data to the message queue to be grouped; Obtaining the initial scene data from the to-be-grouped message queue by the data grouping component according to a preset first time interval; Grouping the initial scene data by the data grouping component to obtain a plurality of new initial scene data and marking corresponding algorithm identifiers; selecting, by the data grouping component, a second target hash function from a plurality of preset hash functions according to the algorithm identifier; The queue hash value corresponding to each of the new initial scene data is calculated by the data grouping component using the second target hash function.

8. An electronic device, characterized in that: It includes a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the scene data storage method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed, the scene data storage method according to any one of claims 1 to 6 is implemented.

Images