A distributed stream data processing method, device and equipment

Abstract

Embodiments of the present specification disclose a distributed stream data processing method, device and equipment. The method comprises: obtaining a first stream data set collected by a data collection device in a first time range in a first spatial partition, and a second stream data set collected by the data collection device in a second time range in a second spatial partition, each stream data in the two data sets comprising at least time attribute information describing a stream data generation time and position attribute information describing a generation position; determining, according to the time attribute information and the position attribute information, a target stream data set for the same event existing in the first stream data set and the second stream data set; performing deduplication processing on each stream data in the target stream data set; and finally outputting the deduplicated stream data to a data processing device for automatic driving decision planning. The present scheme removes repeated data that interferes with automatic driving decision planning, thereby improving the accuracy of automatic driving decision planning.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a distributed stream data processing method, device and equipment. BACKGROUND

[0002] An autonomous vehicle is a comprehensive system integrating functions such as environment perception, planning and decision-making, and multi-level auxiliary driving. It centrally uses computers, modern sensing, information fusion, communication, artificial intelligence, and automatic control technologies, and belongs to a typical high-tech complex.

[0003] Real-time operation data of an autonomous vehicle plays a crucial role in rational planning of the operation of the autonomous vehicle. During the driving of the autonomous vehicle on the road, various events may occur, such as emergency braking events, emergency stopping events, etc. For various events of the vehicle, people hope to obtain all data of the vehicle event data as soon as possible. Therefore, in order to improve the speed of obtaining vehicle event data, the vehicle road is usually spatially partitioned, and vehicle event data in different partitions is obtained. However, after spatial partitioning of the vehicle road, some vehicle events will inevitably occur in the border area between spatial partitions, i.e., a part of the vehicle event occurs in the first spatial partition and the remaining part occurs in the second spatial partition, so it is not possible to determine to which spatial partition the vehicle event belongs. In order to prevent such problems, when spatial partitioning is performed, a certain overlap area is set between any two adjacent spatial partitions, and the overlap area belongs to both the first spatial partition and the second spatial partition. Then, the vehicle event occurring in the border area between the spatial partitions belongs to both the first spatial partition and the second spatial partition. When the vehicle event data in the two spatial partitions is sent to the autonomous driving planning device, the repeated data of the same vehicle event will be processed repeatedly. The repeated vehicle event not only increases the burden of the data processing device, causing waste of data processing resources, but also interferes with the autonomous driving decision-making and planning device, causing the autonomous driving decision-making and planning device to be unable to make correct analysis and judgment for the autonomous vehicle, and thus sending incorrect instructions to the autonomous vehicle, resulting in unpredictable consequences for the autonomous vehicle.

[0004] Therefore, how to perform deduplication processing on repeated vehicle event data of the same event has become a technical problem to be solved. SUMMARY

[0005] The distributed stream data processing method, device and equipment provided by the embodiments of the present specification can perform deduplication processing on repeated vehicle event data for the same event, thereby avoiding the interference of repeated data on the automatic driving planning equipment, causing misjudgment of the analysis of the automatic driving vehicle, and improving the accuracy of the analysis result of the automatic driving vehicle by the automatic driving planning equipment.

[0006] To solve the above technical problems, the embodiments of the present specification are implemented as follows:

[0007] A distributed stream data processing method comprises,

[0008] obtaining a first stream data set collected by a data collection device in a first time range in a first spatial partition.

[0009] obtaining a second stream data set collected by the data collection device in a second time range in a second spatial partition; the first spatial partition and the second spatial partition have an overlapping area in space; the first time range and the second time range have an overlapping range in time; each stream data in the first stream data set and the second stream data set at least includes time attribute information for describing the generation time of the stream data, and position attribute information for describing the generation position of the stream data.

[0010] According to the time attribute information and the position attribute information, determining a target stream data set for the same event existing in the first stream data set and the second stream data set.

[0011] According to a preset rule, performing deduplication processing on each stream data in the target stream data set to obtain processed stream data.

[0012] outputting the processed stream data to a data processing equipment for automatic driving decision planning.

[0013] A distributed stream data processing device comprises,

[0014] The first obtaining module is configured to obtain a first stream data set collected by a data collection device in a first time range in a first spatial partition.

[0015] The second obtaining module is configured to obtain a second stream data set collected by the data collection device in a second time range in a second spatial partition; the first spatial partition and the second spatial partition have an overlapping area in space; the first time range and the second time range have an overlapping range in time; each stream data in the first stream data set and the second stream data set at least includes time attribute information for describing the generation time of the stream data, and position attribute information for describing the generation position of the stream data.

[0016] determine a target stream data set for the same event existing in the first stream data set and the second stream data set according to the time attribute information and the location attribute information.

[0017] perform deduplication processing on each stream data in the target stream data set according to a preset rule to obtain processed stream data.

[0018] output the processed stream data to a data processing device for automatic driving decision planning.

[0019] A distributed stream data processing device comprises,

[0020] at least one processor; and

[0021] a memory in communication connection with the at least one processor; wherein

[0022] the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to:

[0023] acquire a first stream data set collected by a data collection device in a first time range in a first spatial partition.

[0024] acquire a second stream data set collected by the data collection device in a second time range in a second spatial partition; the first spatial partition and the second spatial partition have an overlapping region in space; the first time range and the second time range have an overlapping range in time; each stream data in the first stream data set and the second stream data set at least comprises time attribute information for describing the generation time of the stream data and location attribute information for describing the generation location of the stream data.

[0025] determine a target stream data set for the same event existing in the first stream data set and the second stream data set according to the time attribute information and the location attribute information.

[0026] perform deduplication processing on each stream data in the target stream data set according to a preset rule to obtain processed stream data.

[0027] output the processed stream data to a data processing device for automatic driving decision planning.

[0028] At least one embodiment provided in the specification can achieve the following beneficial effects:

[0029] The vehicle event data repeatedly reported for the same event can be retained only once after the de-duplication processing, which can avoid the waste of server resources caused by a large amount of repeated data, and improve the effectiveness of server resource utilization. On the other hand, it can also avoid the interference and influence of repeated vehicle event data on the automatic driving planning device, so that the automatic driving planning device can make accurate analysis and judgment on the automatic driving vehicle by using the de-duplicated vehicle event data, and improve the accuracy of the data analysis result made by the automatic driving planning device for the automatic driving vehicle. BRIEF DESCRIPTION OF DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present specification or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments described in the present specification, and those skilled in the art can obtain other drawings according to these drawings without any creative labor.

[0031] Figure 1 A flowchart of a distributed stream data processing method provided by an embodiment of the present specification;

[0032] Figure 2 A structural diagram of a distributed stream data processing device corresponding to Figure 1 provided by an embodiment of the present specification;

[0033] Figure 3 A structural diagram of a distributed stream data processing device corresponding to Figure 1 provided by an embodiment of the present specification;

[0034] Figure 4 A road space partition diagram provided by an embodiment of the present specification. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solutions and advantages of one or more embodiments of the present specification more clear, the technical solutions of one or more embodiments of the present specification will be described clearly and completely in combination with the specific embodiments of the present specification and the corresponding drawings. Obviously, the described embodiments are only some embodiments of the present specification, not all embodiments. Based on the embodiments in the present specification, all other embodiments obtained by those skilled in the art without any creative labor are within the scope of protection of one or more embodiments of the present specification.

[0036] The technical solutions provided by the embodiments of the present application will be described in detail below in combination with the drawings.

[0037] Figure 1A flowchart of a distributed stream data processing method according to an embodiment of the present specification is shown. From a program perspective, the execution subject of the flowchart can be a program or an application client loaded on an application server. As shown in Figure 1 the flowchart can include the following steps:

[0038] Step 102: Obtain a first stream data set collected by a data collection device in a first time range in a first spatial partition.

[0039] In an embodiment of the present specification, stream data is a set of sequential, large, fast, and continuous data sequences. Generally, stream data can be regarded as a dynamic data set that grows indefinitely over time. The stream data set can be a set of vehicle event data occurring in real time, and the vehicle event can include a vehicle emergency braking event, a vehicle emergency braking event, a vehicle fault alarm event, a vehicle fuel alarm event, a vehicle overspeed event, etc.

[0040] In an embodiment of the present specification, the data collection device can be an on-board unit (OBU) or a road side unit (RSU). The on-board unit can perceive various events occurring in the vehicle during driving, and the road side unit can also perceive various events occurring in the vehicle during driving through radar or video collection devices. The on-board unit and the road side unit can be communicatively connected, and the on-board unit can send the perceived vehicle event data to the road side unit, which can send the data to an autonomous driving planning device.

[0041] In an embodiment of the present specification, a spatial partition is a division of any road in any map into multiple partitions, and each spatial partition can be a polygon in space. The time range for data collection by the data collection device can be set to any time range according to requirements, and the specific time range interval can be selected as 5 milliseconds or 10 milliseconds. The present application does not limit the time range interval.

[0042] Step 104: Obtain a second stream data set collected by the data collection device in a second time range in a second spatial partition, the first spatial partition and the second spatial partition having an overlapping region in space, the first time range and the second time range having an overlapping range in time, each stream data in the first stream data set and the second stream data set including at least time attribute information describing the generation time of the stream data and position attribute information describing the generation position of the stream data.

[0043] In an embodiment of the present specification, any two adjacent spatial partitions have an overlapping region, as shown in Figure 4As shown, the overlapping area of the first spatial partition and the second spatial partition is area 12, which belongs to both the first spatial partition and the second spatial partition. The size of each spatial partition and the size of any overlapping area can be determined according to actual requirements, and the present application does not make a specific limitation in this regard.

[0044] In the embodiments of the present application, when the data collection device collects stream data in the overlapping area of the first spatial partition and the second spatial partition, the time at which the data collection device collects stream data in the overlapping area belongs to both the first time range of the first spatial partition and the second time range of the second spatial partition. Therefore, the first time range and the second time range have an overlapping range, which is the time range in which the data collection device collects stream data in the overlapping area.

[0045] In the embodiments of the present application, any stream data can have multiple data attributes, which can include time attribute information describing the occurrence time of the stream data and position attribute information describing the occurrence position. For a specific vehicle emergency braking event, the vehicle emergency braking event data collected by the data collection device can include time attribute information and position attribute information of the vehicle emergency braking event, the time attribute information indicating the specific time at which the vehicle emergency braking event occurs, and the position attribute information indicating the specific position at which the emergency braking event occurs.

[0046] In the embodiments of the present application, the data attributes of stream data can further include type attribute and number attribute describing the stream data. Specifically, the type attribute of vehicle event data indicates the type of vehicle event, and the specific vehicle event type can include: vehicle emergency braking event, vehicle emergency braking event, vehicle fault alarm event, vehicle fuel alarm event, vehicle overspeed event, etc. The number attribute of vehicle event data can specifically include: object-ID (data uniqueness identifier) and eventID (event number), each piece of vehicle event data having a unique object-ID, and the eventID indicating the event number of a vehicle event. It should be noted that one vehicle event in one spatial partition can generate N pieces of vehicle event data, the eventID of the N pieces of vehicle event data being the same, but the object-ID of the N pieces of vehicle event data being different from each other.

[0047] Step 106: determining, according to the time attribute information and the position attribute information, a target stream data set for the same event existing in the first stream data set and the second stream data set.

[0048] In the embodiments of the present specification, since the occurrence process of any vehicle event lasts for a certain period of time, the data collected by the data collection device during the occurrence of the event will be sent to the processing device all the time. If the time interval for the processing device to process data is greater than the time interval for the data collection device to send data, there may be repeated vehicle event data for each vehicle event when the processing device processes the data in a batch. If the time interval for the processing device to process data is less than the time interval for the data collection device to send data, if a vehicle event occurs in an overlapping area, the same event will also be sent to the processing device in two spatial partitions, and there may also be repeated vehicle event data for each vehicle event when the processing device processes the data in a batch. For all vehicle event data to be processed in the batch, vehicle event data with the same or similar time attribute information and the same or similar location attribute information is determined as the target stream data set of the same event.

[0049] Step 108: According to the preset rule, each stream data in the target stream data set is processed to obtain processed stream data.

[0050] In the embodiments of the present specification, the preset rule can be to select target stream data from the target stream data set according to the receiving time information of each stream data; the preset rule can also be to select target stream data from the target stream data set according to the time attribute information of the generation time of each stream data; and the preset rule can also be to select target stream data from the target stream data set according to the location attribute information of the generation location of each stream data, and the selected target stream data is taken as the processed stream data.

[0051] Step 110: Output the processed stream data to a data processing device for automatic driving decision planning.

[0052] In the embodiments of the present specification, the data processing device for automatic driving decision planning refers to a data processing device for decision planning of the driving of an automatic driving vehicle. The vehicle event data after the deduplication processing is sent to the data processing device for automatic driving decision planning, so that the data processing device makes correct analysis and judgment on the vehicle according to the received vehicle event data, and then sends correct driving instructions to the automatic driving vehicle according to the analysis and judgment result, so that the automatic driving vehicle can safely drive on the road.

[0053] Figure 1The method described above deduplicates repeatedly reported vehicle event data for the same event, retaining only the data from one reported event. This avoids wasting server resources by processing large amounts of duplicate data, improving server resource utilization. Furthermore, it prevents duplicate vehicle event data from interfering with and impacting autonomous driving planning equipment, allowing the equipment to make accurate analyses and judgments about autonomous vehicles and improving the accuracy of its data analysis results.

[0054] based on Figure 1 In addition to the method described in the embodiments of this specification, some specific implementation schemes of the method are also provided, which will be described below.

[0055] Since time attribute information and location attribute information can be used to determine the target stream data set for the same event, in order to improve the accuracy of determining the target stream data set for the same event, the time attribute information includes the timestamp information of the event occurrence; the location attribute information includes the latitude and longitude information of the event occurrence.

[0056] In the embodiments of this specification, the timestamp information of the event indicates the specific time when the event occurred. The format of the timestamp information can be xx year xx month xx day xx hour xx minute xx second xx millisecond, and the time precision of the timestamp information can be set according to requirements. The latitude and longitude information of the event indicates the specific location where the event occurred. The format of the latitude and longitude information can be longitude: xx.xxxxxx, latitude: xx.xxxxxx. The location precision of the latitude and longitude information can be set according to requirements. This application does not limit the format of the timestamp information, the time precision of the timestamp information, the format of the latitude and longitude information, or the location precision of the latitude and longitude information.

[0057] Since the time attribute information includes the timestamp of the event and the location attribute information includes the latitude and longitude of the event, the target stream data set for the same event existing in the first stream data set and the second stream data set can be determined based on the timestamp and latitude / longitude of the event. Therefore, step 106: Determining the target stream data set for the same event existing in the first stream data set and the second stream data set based on the time attribute information and the location attribute information can specifically include:

[0058] Obtain first stream data and second stream data from the first stream data set and the second stream data set.

[0059] Obtain the first timestamp information and the first latitude and longitude information of the first stream data.

[0060] obtain second timestamp information and second longitude and latitude information of the second stream data.

[0061] generate a time difference value according to the first timestamp information and the second timestamp information.

[0062] generate a distance difference value according to the first longitude and latitude information and the second longitude and latitude information.

[0063] determine a target stream data set for the same event existing in the first stream data set and the second stream data set according to the time difference value and the distance difference value.

[0064] In the embodiments of the present specification, the first stream data and the second stream data are obtained from the first stream data set and the second stream data set. The first stream data and the second stream data can both belong to the first stream data set or the second stream data set, or the first stream data belongs to the first stream data set and the second stream data belongs to the second stream data set, or the first stream data belongs to the second stream data set and the second stream data belongs to the first stream data set.

[0065] In the embodiments of the present specification, the first stream data is parsed to obtain first object-ID information, first eventID information, first event type information, first timestamp information and first longitude and latitude information describing the first stream data. Meanwhile, the second stream data is parsed to obtain second object-ID information, second eventID information, second event type information, second timestamp information and second longitude and latitude information describing the second stream data.

[0066] In the embodiments of the present specification, according to a first specific time represented by the first timestamp information and a second specific time represented by the second timestamp information, a time difference value between the first specific time and the second specific time is calculated. According to a first specific position represented by the first longitude and latitude information and a second specific position represented by the second longitude and latitude information, a distance difference value between the first specific position and the second specific position is calculated.

[0067] In the embodiments of the present specification, according to the method of calculating the time difference value between the first timestamp information of the first stream data and the second timestamp information of the second stream data, and the distance difference value between the first longitude and latitude information of the first stream data and the second longitude and latitude information of the second stream data, the time difference value of the timestamp information and the position difference value of the longitude and latitude information of any two stream data in the first stream data set and the second stream data set are calculated, so as to obtain a plurality of time difference values and distance difference values. The stream data associated with the time difference value and the distance difference value meeting certain conditions is divided into the target stream data set of the same event.

[0068] In the embodiments of the present specification, the multiple stream data belonging to the same event are summarized into one set, and subsequently, the event can be processed only for the stream data of the event to avoid the interference of the stream data of other events on the processing of the event. Thus, the convenience of processing the stream data for the corresponding event can be improved.

[0069] After obtaining the multiple time difference values and distance difference values for the first stream data set and the second stream data set, whether the stream data associated with any group of time difference values and distance difference values belongs to the target stream data set of the same event can be determined according to certain rules. Based on this, the target stream data set for the same event existing in the first stream data set and the second stream data set can be determined according to the time difference values and the distance difference values, and specifically can include:

[0070] It is determined whether the time difference value is less than a first preset value and the distance difference value is less than a second preset value to obtain a determination result.

[0071] If the determination result indicates that the time difference value is less than the first preset value and the distance difference value is less than the second preset value, it is determined that the first stream data and the second stream data belong to the target stream data set of the same event.

[0072] If the determination result indicates that the time difference value is greater than or equal to the first preset value and / or the distance difference value is greater than or equal to the second preset value, it is determined that the first stream data and the second stream data do not belong to the target stream data set of the same event.

[0073] In the embodiments of the present specification, the first preset value is a time value, which can be associated with the time interval of the data acquisition device collecting data. If the data acquisition device can collect the same event in the process of collecting data twice, it indicates that the event is occurring all the time, and thus the interval time of the data acquisition device collecting data can be set as the first preset value. It should be noted that if the accuracy requirement of data processing is high, the first preset value can be set to be less than the interval time of the data acquisition device collecting data. Similarly, if the accuracy requirement of data processing is low and the speed requirement of data processing is high, the first preset value can be set to be greater than the interval time of the data acquisition device collecting data. The value of the first preset value is determined according to the actual requirement, which is not limited in the present application.

[0074] In the embodiments of the present specification, the second preset value is a distance value, which can be associated with the driving speed of the vehicle when the event occurs. If the driving speed of the vehicle is fast, the distance value between the corresponding latitude and longitude information in the adjacent two collected data of the data collection device will be large, and vice versa. Therefore, the second preset value needs to be set according to the actual scene requirements, which is not limited in the present application.

[0075] In the embodiments of the present specification, the time difference value and the position difference value of the time stamp information and the latitude and longitude information of any two stream data in the first stream data set and the second stream data set are calculated, and each time difference value is compared with the first preset value, and each distance difference value is compared with the second preset value. The specific comparison process is: if the Nth time difference value is less than the first preset value, the Nth distance difference value corresponding to the two stream data associated with the Nth time difference value is compared with the second preset value, and if the Nth distance difference value is less than the second preset value, the two stream data are determined as the repeated stream data of the same event. The reason for such comparison is that the two stream data with close time stamp information may belong to the stream data of two events occurring at different locations, and similarly, the two stream data with close latitude and longitude information may belong to the stream data of two events occurring at different times. Therefore, only when the time stamp information and the latitude and longitude information of the two stream data are close, that is, both conditions are met, the two stream data are determined as the repeated stream data of the same event. It should be noted that the above comparison process is not limited to the order of comparing the time difference value first and then comparing the distance difference value, but also can compare the distance difference value first and then compare the time difference value, which is not limited in the present application.

[0076] In the embodiments of the present specification, if the comparison result is that the Nth time difference value is greater than or equal to the first preset value, or the Nth distance difference value is greater than or equal to the second preset value, or the Nth time difference value is greater than or equal to the first preset value and the Nth distance difference value is greater than or equal to the second preset value, the associated stream data is determined as the repeated stream data not belonging to the same event.

[0077] In the embodiments of the present specification, the target stream data set belonging to the same event is determined according to the time difference value and the position difference value, so as to improve the accuracy of determining two stream data as the repeated stream data of the same event.

[0078] For the same event, all the stream data belonging to the event are all summarized into a target stream data set, in order to facilitate the subsequent analysis and processing of the event, it is necessary to select representative target stream data from the target stream data set of the event, and only analyze and process the selected target stream data, without analyzing and processing all the repeated stream data of the event. Based on this, step 108: according to the preset rule, the de-duplication processing is performed on each stream data in the target stream data set, and the processed stream data is obtained. Specifically, it can include:

[0079] Obtain the receiving time information of each target stream data in the target stream data set.

[0080] According to the time order of each receiving time information, each target stream data is sorted.

[0081] The target stream data corresponding to the earliest receiving time information or the latest receiving time information is determined as the processed stream data.

[0082] In the embodiments of the present specification, the receiving time of each stream data in the target stream data set can be queried from the data receiving device. Each stream data is arranged in a sequence according to the time order of the respective receiving time. The stream data corresponding to the earliest receiving time can be determined as the required stream data, and such processing can trace back the earliest time of the event. The stream data corresponding to the latest receiving time can be determined as the required stream data, and such processing can inform the data processing device that the event is still occurring at the time of receiving the stream data. Then, all the remaining repeated stream data in the target stream data set are deleted.

[0083] Step 108: according to the preset rule, the de-duplication processing is performed on each stream data in the target stream data set, and the processed stream data is obtained. Specifically, it can also include:

[0084] Obtain the timestamp information of each target stream data in the target stream data set.

[0085] According to the time order of each timestamp information, each target stream data is sorted.

[0086] The target stream data corresponding to the earliest timestamp information or the latest timestamp information is determined as the processed stream data.

[0087] Or,

[0088] Obtain the latitude and longitude information of each target stream data in the target stream data set.

[0089] According to the position order of each latitude and longitude information, each target stream data is sorted.

[0090] The target stream data corresponding to the longitude and latitude information of the earliest position or the longitude and latitude information of the latest position is determined as the processed stream data.

[0091] In the embodiments of the present specification, the timestamp information of each stream data is obtained according to the analysis result of each stream data in the target stream data set. Each stream data is arranged in a sequence according to the time carried by the respective timestamp information. The stream data corresponding to the earliest time in the time carried by the timestamp information can be determined as the required stream data, and thus the earliest time of the event can be obtained. The stream data corresponding to the latest time in the time carried by the timestamp information can be determined as the required stream data, and thus the data processing device can be informed that the event is still occurring at the latest time. Then, all the remaining repeated stream data in the target stream data set are deleted.

[0092] Alternatively, the longitude and latitude information of each stream data can also be obtained from the analysis result. Each stream data is arranged in a sequence according to the position relationship of the position carried by the respective longitude and latitude information in the vehicle driving direction. The earliest position and the latest position of the position carried by the longitude and latitude information are obtained according to the sequence. The earliest position represents the position farthest from the vehicle, and the latest position represents the position closest to the vehicle. The stream data corresponding to the earliest position in the position carried by the longitude and latitude information can be determined as the required stream data, and thus the initial position of the event can be obtained. The stream data corresponding to the latest position in the position carried by the longitude and latitude information can be determined as the required stream data, and thus the data processing device can be informed that the event is still occurring at the latest position. Then, all the remaining repeated stream data in the target stream data set are deleted.

[0093] Step 108: performing deduplication processing on each stream data in the target stream data set according to a preset rule to obtain processed stream data. Specifically, the method can further include:

[0094] Obtaining third timestamp information and third longitude and latitude information of a first target stream data in the target stream data set.

[0095] Obtaining fourth timestamp information and fourth longitude and latitude information of a second target stream data in the target stream data set.

[0096] Calculating the average of the third timestamp information and the fourth timestamp information to obtain fifth timestamp information.

[0097] Calculating the average of the third longitude and latitude information and the fourth longitude and latitude information to obtain fifth longitude and latitude information.

[0098] generate third target stream data containing the fifth timestamp information and the fifth longitude and latitude information.

[0099] determine the third target stream data as the processed stream data.

[0100] In the embodiments of the present specification, the timestamp information and the longitude and latitude information of each stream data are obtained according to the analysis results of each stream data in the target stream data set, the average timestamp information is calculated according to each timestamp information, and the average longitude and latitude information is calculated according to each longitude and latitude information. It should be noted that if the number of repeated stream data of the same event is odd, the average timestamp information and the average longitude and latitude information calculated are exactly the timestamp information and the longitude and latitude information of the stream data located in the middle position after arranging the repeated stream data of the same event according to the timestamp information or the longitude and latitude information. At this time, the stream data in the middle position is taken as the required stream data. If the number of repeated stream data of the same event is even, each timestamp information is not the same as the average timestamp information, and each longitude and latitude information is not the same as the average longitude and latitude information, that is, there is no stream data corresponding to the average timestamp information and the average longitude and latitude information in the existing repeated stream data of the same event. At this time, the time difference between the average timestamp information and each timestamp information can be calculated, the stream data corresponding to the timestamp information with the minimum time difference is named as stream data a, and then the timestamp information and the longitude and latitude information of the stream data a are changed to the average timestamp information and the average longitude and latitude information respectively to obtain updated stream data a. The distance difference between the average longitude and latitude information and each longitude and latitude information can also be calculated, the stream data corresponding to the longitude and latitude information with the minimum distance difference is also named as stream data a, and then the timestamp information and the longitude and latitude information of the stream data a are changed to the average timestamp information and the average longitude and latitude information respectively to obtain updated stream data a, and the updated stream data a is taken as the required stream data. In this way, the middle stream data in the event occurrence process can be sent to the data processing device, the approximate time range and distance range of each event occurrence can be obtained by the data processing device through early investigation, so that the data processing device can estimate the start time, the start position, the end time, the end position and other related information of the event occurrence according to the related information of the middle stream data. It should be noted that if there are two stream data named as stream data a, any one of the two stream data can be selected to be named as stream data a.

[0101] In the embodiments of the present specification, all the repeated stream data of the same event are removed, so that only one stream data of the event is retained for subsequent processing, thereby reducing the burden of subsequent data processing, avoiding the interference of multiple repeated data on the data analysis of the subsequent processing device, and improving the accuracy of data processing of the subsequent data processing device.

[0102] The data collection device continuously sends stream data to the data processing device, and the data processing device processes the stream data sent by the data collection device in batches. The data processing device sends the processed stream data to the data processing device for automatic driving decision planning after processing each batch of stream data. However, stream data in two processing batches sometimes exist for the same event. At this time, if the corresponding processed stream data for the event has been output in the first batch, the corresponding processed stream data for the event still needs to be output when the processed stream data is output in the second batch. Based on this, after the step 110 of outputting the processed stream data to the data processing device for automatic driving decision planning, the method can further include:

[0103] Record each stream data output for the first batch to obtain a first output stream data set.

[0104] Record each stream data output for the second batch to obtain a second output stream data set.

[0105] Filter the repeated stream data in the second output stream data set that is the same as the third stream data in the first output stream data set, and output the third stream data instead of the repeated stream data as the corresponding stream data of the second batch.

[0106] In the embodiments of the present specification, the repeated stream data of each event in the first batch is processed to obtain the processed stream data of each event. After the processed stream data of each event is output to the data processing device with automatic driving decision planning, the recorded each stream data that has been output is saved. After the repeated data of each event in the second batch is processed, the processed stream data of each event in the second batch is obtained. The processed stream data of each event in the second batch is filtered from the stream data that has been output in the first batch. The specific filtering method can refer to the above-mentioned de-duplication method, and therefore will not be repeated here. After filtering, if there is a stream data b in the stream data that has been output in the first batch and the stream data c in the stream data to be output in the second batch, the stream data b in the first batch is output instead of the stream data c in the second batch when the processed stream data of each event in the second batch is output. In this way, for the same event, if it appears in different data processing batches, the same stream data is output in each output batch, so that the data processing device for automatic driving decision planning knows the time range and position range of the continuous occurrence of the event.

[0107] Based on the same idea, the embodiments of the present specification also provide a device corresponding to the above-mentioned method. Figure 2 The embodiments of the present specification provide a device corresponding to the above-mentioned method. Figure 1A structural schematic diagram of a distributed stream data processing device. As shown in Figure 2 The device can include:

[0108] A first obtaining module 202 is configured to obtain a first stream data set collected by a data collection device in a first spatial partition within a first time range.

[0109] A second obtaining module 204 is configured to obtain a second stream data set collected by the data collection device in a second spatial partition within a second time range; the first spatial partition and the second spatial partition have an overlapping region in space; the first time range and the second time range have an overlapping range in time; each of the first stream data set and the second stream data set includes at least time attribute information for describing a generation time of stream data and location attribute information for describing a generation location of stream data.

[0110] A determining module 206 is configured to determine, according to the time attribute information and the location attribute information, a target stream data set for a same event existing in the first stream data set and the second stream data set.

[0111] A processing module 208 is configured to perform deduplication processing on each stream data in the target stream data set according to a preset rule, to obtain processed stream data.

[0112] An output module 210 is configured to output the processed stream data to a data processing device for automatic driving decision planning.

[0113] Optionally, the time attribute information includes timestamp information of the event; and the location attribute information includes latitude and longitude information of the event.

[0114] Optionally, the determining module 206 can specifically include:

[0115] A first obtaining unit is configured to obtain a first stream data and a second stream data from the first stream data set and the second stream data set.

[0116] A second obtaining unit is configured to obtain first timestamp information and first latitude and longitude information of the first stream data.

[0117] A third obtaining unit is configured to obtain second timestamp information and second latitude and longitude information of the second stream data.

[0118] A first generating unit is configured to generate a time difference value according to the first timestamp information and the second timestamp information.

[0119] The second generating unit is configured to generate a distance difference value according to the first latitude and longitude information and the second latitude and longitude information.

[0120] The first determining unit is configured to determine a target stream data set for a same event existing in the first stream data set and the second stream data set according to the time difference value and the distance difference value.

[0121] Optionally, the first determining unit specifically can include:

[0122] The first judging sub-unit is configured to judge whether the time difference value is less than a first preset value and the distance difference value is less than a second preset value to obtain a judgment result.

[0123] The first determining sub-unit is configured to determine that the first stream data and the second stream data belong to the target stream data set of the same event if the judgment result indicates that the time difference value is less than the first preset value and the distance difference value is less than the second preset value.

[0124] The second determining sub-unit is configured to determine that the first stream data and the second stream data do not belong to the target stream data set of the same event if the judgment result indicates that the time difference value is greater than or equal to the first preset value and / or the distance difference value is greater than or equal to the second preset value.

[0125] Optionally, the processing module 208 specifically can include:

[0126] The fourth acquiring unit is configured to acquire receiving time information of each target stream data in the target stream data set.

[0127] The first sorting unit is configured to sort each target stream data according to a time sequence of each receiving time information.

[0128] The second determining unit is configured to determine a target stream data corresponding to earliest receiving time information or latest receiving time information as the processed stream data.

[0129] Optionally, the processing module 208 specifically can include:

[0130] The fifth acquiring unit is configured to acquire the timestamp information of each target stream data in the target stream data set.

[0131] The second sorting unit is configured to sort each target stream data according to a time sequence of each timestamp information.

[0132] The third determining unit is configured to determine a target stream data corresponding to earliest timestamp information or latest timestamp information as the processed stream data.

[0133] Or,

[0134] A sixth obtaining unit is configured to obtain the longitude and latitude information of each target stream data in the target stream data set.

[0135] A third sorting unit is configured to sort each target stream data according to the position sequence of the longitude and latitude information.

[0136] A fourth determining unit is configured to determine the target stream data corresponding to the longitude and latitude information at the front position or the longitude and latitude information at the last position as the processed stream data.

[0137] Optionally, the processing module 208 can specifically include:

[0138] A seventh obtaining unit is configured to obtain third timestamp information and third longitude and latitude information of the first target stream data in the target stream data set.

[0139] An eighth obtaining unit is configured to obtain fourth timestamp information and fourth longitude and latitude information of the second target stream data in the target stream data set.

[0140] A first calculating unit is configured to calculate the average of the third timestamp information and the fourth timestamp information to obtain fifth timestamp information.

[0141] A second calculating unit is configured to calculate the average of the third longitude and latitude information and the fourth longitude and latitude information to obtain fifth longitude and latitude information.

[0142] A third generating unit is configured to generate third target stream data containing the fifth timestamp information and the fifth longitude and latitude information.

[0143] A fifth determining unit is configured to determine the third target stream data as the processed stream data.

[0144] Optionally, the output module 210 can further include:

[0145] A first recording unit is configured to record each stream data output in the first batch to obtain a first output stream data set.

[0146] A second recording unit is configured to record each stream data output in the second batch to obtain a second output stream data set.

[0147] A first screening unit is configured to screen repeated stream data in the second output stream data set that is the same as third stream data in the first output stream data set, replace the repeated stream data with the third stream data, and output the third stream data as the corresponding stream data in the second batch.

[0148] Based on the same idea, the embodiments of the present specification also provide a device corresponding to the above method.

[0149] Figure 3 The embodiments of the present specification provide a structure diagram of a distributed stream data processing device corresponding to Figure 1 As shown in the device 300 can include: Figure 3

[0150] at least one processor 310; and,

[0151] a memory 330 in communication connection with the at least one processor; wherein,

[0152] The memory 330 stores instructions 320 executable by the at least one processor 310, and the instructions are executed by the at least one processor 310 to enable the at least one processor 310 to:

[0153] Obtain a first stream data set collected by a data collection device in a first time range in a first spatial partition.

[0154] Obtain a second stream data set collected by the data collection device in a second time range in a second spatial partition; the first spatial partition and the second spatial partition have an overlapping region in space; the first time range and the second time range have an overlapping range in time; each of the first stream data set and the second stream data set includes at least time attribute information for describing the generation time of the stream data, and position attribute information for describing the generation position of the stream data.

[0155] According to the time attribute information and the position attribute information, determine a target stream data set for the same event existing in the first stream data set and the second stream data set.

[0156] According to a preset rule, the target stream data set is de-duplicated to obtain processed stream data.

[0157] The processed stream data is output to a data processing device for automatic driving decision planning.

[0158] Each of the embodiments in the present specification adopts a progressive manner for description, and the same and similar parts between each embodiment can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, for the device shown in Figure 3 , since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

[0159] ​The above merely provides an example of the present application, and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application should be included in the scope of claims of the present application.

Claims

1. A distributed streaming data processing method, characterized in that, include: Acquire the first stream of data collected by the data acquisition device within the first time range of the first spatial partition; Acquire the second stream data set collected by the data acquisition device within the second time range of the second spatial partition; The first spatial partition and the second spatial partition have overlapping areas in space; the first time range and the second time range have overlapping ranges in time; each stream data in the first stream data set and the second stream data set includes at least time attribute information describing the generation time of the stream data and location attribute information describing the generation location of the stream data; the time attribute information includes timestamp information of the event occurrence; the location attribute information includes latitude and longitude information of the event occurrence. Based on the time attribute information and the location attribute information, a target stream data set for the same event is determined that exists in the first stream data set and the second stream data set; According to preset rules, each stream data in the target stream data set is deduplicated to obtain the processed stream data; The processed streaming data is output to a data processing device used for autonomous driving decision-making and planning. The step of deduplicating each stream data in the target stream data set according to preset rules to obtain processed stream data specifically includes: Obtain the third timestamp information and third latitude and longitude information of the first target stream data in the target stream data set; Obtain the fourth timestamp information and the fourth latitude and longitude information of the second target stream data in the target stream data set; Calculate the average of the third timestamp information and the fourth timestamp information to obtain the fifth timestamp information; The average value of the third latitude and longitude information and the fourth latitude and longitude information is calculated to obtain the fifth latitude and longitude information; Generate third target stream data containing the fifth timestamp information and the fifth latitude and longitude information; The third target stream data is determined as the processed stream data.

2. The method as described in claim 1, characterized in that, The step of determining the target stream data set for the same event existing in the first stream data set and the second stream data set based on the time attribute information and the location attribute information specifically includes: Obtain first-stream data and second-stream data from the first-stream data set and the second-stream data set; Obtain the first timestamp information and the first latitude and longitude information of the first stream data; Obtain the second timestamp information and the second latitude and longitude information of the second stream data; A time difference is generated based on the first timestamp information and the second timestamp information; Based on the first latitude and longitude information and the second latitude and longitude information, a distance difference is generated; Based on the time difference and the distance difference, a target stream data set for the same event is determined that exists in the first stream data set and the second stream data set.

3. The method as described in claim 2, characterized in that, Based on the time difference and the distance difference, a target stream data set for the same event is determined, existing in the first stream data set and the second stream data set, specifically including: Determine whether the time difference is less than a first preset value and whether the distance difference is less than a second preset value, and obtain the determination result; If the determination result indicates that the time difference is less than the first preset value and the distance difference is less than the second preset value, then it is determined that the first stream data and the second stream data belong to the target stream data set of the same event; If the determination result indicates that the time difference is greater than or equal to the first preset value, and / or the distance difference is greater than or equal to the second preset value, then it is determined that the first stream data and the second stream data do not belong to the target stream data set of the same event.

4. The method as described in claim 1, characterized in that, The step of deduplicating each stream data in the target stream data set according to preset rules to obtain processed stream data specifically includes: Obtain the reception time information of each target stream data in the target stream data set; Sort the target stream data according to the time order of each of the received time information; The target stream data corresponding to the earliest or latest received time information is determined as the processed stream data.

5. The method as described in claim 1, characterized in that, The step of deduplicating each stream data in the target stream data set according to preset rules to obtain processed stream data specifically includes: Obtain the timestamp information of each target stream data in the target stream data set; The target stream data are sorted according to the time order of each timestamp information; The target stream data corresponding to the earliest or latest timestamp information is determined as the processed stream data; or, Obtain the latitude and longitude information of each target flow data in the target flow data set; The target stream data are sorted according to the positional order of the latitude and longitude information. The target stream data corresponding to the latitude and longitude information at the beginning or the end of the location is determined as the processed stream data.

6. The method as described in claim 1, characterized in that, After outputting the processed streaming data to the data processing device for autonomous driving decision-making and planning, the method further includes: Record the data from each stream of the first batch of outputs to obtain the first output stream data set; Record the data streams output in the second batch to obtain the second output stream data set; Filter out duplicate stream data in the second output stream data set that are identical to the third stream data in the first output stream data set, and use the third stream data to replace the duplicate stream data as the corresponding stream data for the second batch of output.

7. A distributed streaming data processing device, characterized in that, include: The first acquisition module is used to acquire the first stream data set collected by the data acquisition device within a first time range in the first spatial partition. The second acquisition module is used to acquire the second stream data set acquired by the data acquisition device within the second time range of the second spatial partition; The first spatial partition and the second spatial partition have overlapping areas in space; the first time range and the second time range have overlapping ranges in time; each stream data in the first stream data set and the second stream data set includes at least time attribute information describing the generation time of the stream data and location attribute information describing the generation location of the stream data; the time attribute information includes timestamp information of the event occurrence; the location attribute information includes latitude and longitude information of the event occurrence. The determination module is used to determine, based on the time attribute information and the location attribute information, a target stream data set for the same event that exists in the first stream data set and the second stream data set; The processing module is used to perform deduplication processing on each stream data in the target stream data set according to preset rules to obtain the processed stream data; The output module is used to output the processed streaming data to a data processing device for autonomous driving decision-making and planning. The processing module is specifically used for: Obtain the third timestamp information and third latitude and longitude information of the first target stream data in the target stream data set; Obtain the fourth timestamp information and the fourth latitude and longitude information of the second target stream data in the target stream data set; Calculate the average of the third timestamp information and the fourth timestamp information to obtain the fifth timestamp information; The average value of the third latitude and longitude information and the fourth latitude and longitude information is calculated to obtain the fifth latitude and longitude information; Generate third target stream data containing the fifth timestamp information and the fifth latitude and longitude information; The third target stream data is determined as the processed stream data.

8. A distributed streaming data processing device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to: Acquire the first stream of data collected by the data acquisition device within the first time range of the first spatial partition; The data acquisition device acquires a second set of streaming data within a second time range in a second spatial partition; the first spatial partition and the second spatial partition have overlapping areas in space; the first time range and the second time range have overlapping ranges in time; each streaming data in the first and second streaming data sets includes at least time attribute information describing the generation time of the streaming data and location attribute information describing the generation location of the streaming data; the time attribute information includes timestamp information of the event occurrence; the location attribute information includes latitude and longitude information of the event occurrence. Based on the time attribute information and the location attribute information, a target stream data set for the same event is determined that exists in the first stream data set and the second stream data set; According to preset rules, each stream data in the target stream data set is deduplicated to obtain the processed stream data; The processed streaming data is output to a data processing device used for autonomous driving decision-making and planning. Obtain the third timestamp information and third latitude and longitude information of the first target stream data in the target stream data set; Obtain the fourth timestamp information and the fourth latitude and longitude information of the second target stream data in the target stream data set; Calculate the average of the third timestamp information and the fourth timestamp information to obtain the fifth timestamp information; The average value of the third latitude and longitude information and the fourth latitude and longitude information is calculated to obtain the fifth latitude and longitude information; Generate third target stream data containing the fifth timestamp information and the fifth latitude and longitude information; The third target stream data is determined as the processed stream data.

Images