Test data annotation method and system

Abstract

The embodiments of this specification provide a test data annotation method, system, and its terminal, server, and storage medium. The method includes: binding relevant information of a road test vehicle to a terminal; when the vehicle is conducting a road test and collecting test data, entering annotation content corresponding to the test data collected in real time through the terminal to generate annotation information; uploading the annotation information; and annotating the uploaded corresponding test data according to the annotation information. Thus, by binding information between the mobile terminal and the test road vehicle and synchronizing time, while collecting test data, the test data can be actively annotated in real time through the screen or voice, and uploaded independently, enabling timely annotation and rapid extraction and processing of data, reducing workload, lowering costs, and improving efficiency.

Description

Technical Field

[0001] This specification relates to the field of test data processing, and in particular to a test data annotation method and system. Background Art

[0002] Currently, the field of autonomous driving and driver assistance requires extensive road testing, which in turn generates a large amount of test data, placing a significant workload on subsequent data cleaning and labeling. To meet these requirements, outsourcing often requires a large number of test fleets and data labelers. Therefore, how to achieve a certain level of data labeling within this massive amount of data is a common concern. Summary of the Invention

[0003] In view of the above problems, this specification is proposed to provide a system and method for test data fact annotation that solves the above problems or at least partially solves the above problems.

[0004] In a first aspect, the present specification provides a test data annotation method, comprising: binding relevant information of a road test vehicle on a terminal; when the vehicle is conducting a road test and collecting test data, entering annotation content corresponding to the test data collected in real time through the terminal to generate annotation information; uploading the annotation information; and annotating the uploaded corresponding test data according to the annotation information.

[0005] Binding the relevant information of the road test vehicle on the terminal specifically includes: the terminal is independent of the data acquisition system that collects test data; and / or the relevant information includes the vehicle code and time.

[0006] Wherein, after binding the relevant information of the road test vehicle on the terminal, the method further includes: synchronizing the time of the terminal with that of the vehicle.

[0007] Among them, when the vehicle is conducting a road test and collecting test data, the annotation content corresponding to the test data collected in real time is entered through the terminal, specifically including: when the vehicle is conducting a road test and collecting test data, finding the vehicle corresponding to the road test on the terminal according to the bound relevant information to find the corresponding real-time collected test data; entering text and / or voice information as the annotation content corresponding to the test data collected in real time through the terminal.

[0008] The generating of the annotation information specifically includes: combining the annotation content entered on the terminal with the relevant information to generate the annotation information.

[0009] The step of inputting the annotation content corresponding to the test data collected in real time through the terminal specifically includes: performing input operations by operating the screen or voice device of the terminal.

[0010] The uploading of the annotation information specifically includes: the terminal uploading the annotation information in real time; and / or storing the uploaded annotation information in a database of a server that processes the collected test data in the background.

[0011] Among them, the corresponding test data uploaded is labeled according to the labeling information, specifically including: determining the test data corresponding to the labeling information in the uploaded test data based on the relevant information combined in the labeling information; automatically completing data labeling and storing the determined corresponding test data; and / or extracting and displaying the test data that has completed data labeling.

[0012] Among them, it further includes: before the terminal is bound to the relevant information, the terminal is connected to the server for background processing of collected test data, so that the terminal independently communicates with the server to complete data transmission; and / or the terminal includes a mobile terminal.

[0013] On the second aspect, this specification provides a test data annotation system, including: a binding device, which binds relevant information of a road test vehicle on a terminal; a generating device, which enters the annotation content of the test data collected in real time through the terminal when the vehicle is conducting a road test and collecting test data, to generate annotation information; an uploading device, which uploads the annotation information; and a data annotation device, which annotates the uploaded corresponding test data according to the annotation information.

[0014] The binding device specifically includes: the terminal is independent of a data acquisition system that collects test data; and / or the relevant information includes the vehicle code and time.

[0015] The binding device further comprises: after binding the relevant information of the road test vehicle on the terminal, synchronizing the time of the terminal with the vehicle.

[0016] Among them, the generating device specifically includes: when the vehicle is conducting a road test and collecting test data, finding the corresponding vehicle conducting the road test on the terminal according to the bound relevant information to find the corresponding real-time collected test data; entering text or voice information through the terminal as the annotation content of the corresponding real-time collected test data.

[0017] The generating device specifically comprises: generating the annotation information by combining the input annotation content with the relevant information on the terminal.

[0018] The generating device specifically includes: performing an input operation by operating the screen or voice device of the terminal.

[0019] The uploading device specifically includes: the terminal uploading the annotation information in real time; and / or storing the uploaded annotation information in a database of a server that processes the collected test data in the background.

[0020] Among them, the data labeling device specifically includes: determining the test data corresponding to the labeling information in the uploaded test data based on the relevant information combined in the labeling information; automatically completing data labeling and storing the determined corresponding test data; and / or extracting and displaying the test data that has completed data labeling.

[0021] Among them, it further includes: before the terminal is bound to the relevant information, the terminal is connected to the server for background processing of collected test data, so that the terminal independently communicates with the server to complete data transmission; and / or the terminal includes a mobile terminal.

[0022] In a third aspect, this specification provides a mobile terminal, comprising: a terminal comprising: a processor and a memory; the memory is used to store programs for executing the aforementioned methods; the processor is configured to execute the programs stored in the memory.

[0023] In a fourth aspect, this specification provides a computer-readable storage medium, comprising: a computer program stored therein, characterized in that the program implements the steps of the aforementioned methods when executed by a processor.

[0024] In a fifth aspect, this specification provides a server comprising: a processor and a memory; the memory is used to store a program for executing the aforementioned method; and the processor is configured to execute the program stored in the memory.

[0025] In a sixth aspect, this specification provides a test data annotation system, comprising: a terminal, a background server for processing collected test data, and a data acquisition system; the terminal is connected to the server and is independent of the data acquisition system; the data acquisition system performs real-time collection of road test data when the vehicle undergoes road testing, and uploads it to the server; the terminal is a terminal that executes the corresponding steps of the method described in any one of claims 1 to 9, so as to enter annotation content and generate annotation information for the test data collected when the bound vehicle undergoes road testing in real time, and upload the annotation information to the server; the server is a server that executes the corresponding steps of the method described in any one of claims 1 to 9, so as to receive the uploaded annotation information and test data, and use the annotation information to complete data annotation for the corresponding test data.

[0026] The one or more technical solutions mentioned above in this specification have at least one or more of the following technical effects:

[0027] In the technical solution for implementing this specification, by binding information and synchronizing time between the mobile terminal and the test road vehicle, active data annotation can be performed while collecting test data, thereby achieving real-time annotation of test data.

[0028] Furthermore, it can work independently of the data collection system on mobile terminals such as mobile phones and tablets, and actively label the collected data to a certain extent. The data labeling is timely, easy to operate and flexible, which facilitates subsequent data processing and improves the extraction speed of certain specific data, thereby avoiding the subsequent workload of processing massive amounts of data, effectively improving data labeling and processing efficiency, reducing workload and lowering costs.

[0029] Furthermore, the annotation content can also support screen input operations (i.e. detailed annotation operations) and / or voice input operations (i.e. direct voice annotation operations when a single driver is driving). Furthermore, the human judgment information collected through the screen or voice input operations during collection can be used as annotation content. The annotation content can be sent separately to the background data server. After the data is uploaded, it will be automatically bound and annotated with the corresponding uploaded test data, which can better support the extraction of data with specific characteristics.

[0030] Specific plans include:

[0031] Solution 1, a test data annotation method, comprising: binding relevant information of a road test vehicle on a terminal; when the vehicle is conducting a road test and collecting test data, entering annotation content corresponding to the test data collected in real time through the terminal to generate annotation information; uploading the annotation information; and annotating the uploaded corresponding test data according to the annotation information.

[0032] Solution 2, according to the method described in Solution 1, binding the relevant information of the road test vehicle on the terminal specifically includes: the terminal is independent of the data acquisition system that collects the test data.

[0033] Solution 3, according to the method described in Solution 2, further includes: synchronizing the time of the terminal with that of the vehicle after binding the relevant information of the road test vehicle on the terminal.

[0034] Solution 4, according to the method described in Solution 1, the relevant information includes: the vehicle code and time.

[0035] Option 5, according to the method described in one of Options 1 to 4, when the vehicle is conducting a road test and collecting test data, the annotation content corresponding to the test data collected in real time is entered through the terminal, specifically including: when the vehicle is conducting a road test and collecting test data, finding the vehicle conducting the road test on the terminal based on the bound relevant information to find the corresponding real-time collected test data; entering text and / or voice information as the annotation content corresponding to the test data collected in real time through the terminal.

[0036] Solution 6, generating annotation information according to the method described in Solution 1, specifically includes: generating annotation information by combining the annotation content entered on the terminal with the relevant information.

[0037] Solution 7, according to the method described in Solution 1, entering the annotation content corresponding to the test data collected in real time through the terminal, specifically including: performing the entry operation by operating the screen or voice device of the terminal.

[0038] Solution 8, according to the method described in Solution 1, uploading the annotation information specifically includes: the terminal uploading the annotation information in real time.

[0039] Solution 9, according to the method described in Solution 1, uploading the annotation information further includes: storing the uploaded annotation information in a database of a server that processes the collected test data in the background.

[0040] Scheme 10, according to the method described in Scheme 1, annotates the corresponding uploaded test data according to the annotation information, specifically including: determining the test data corresponding to the annotation information in the uploaded test data based on the relevant information combined in the annotation information; automatically completing data annotation and storing the determined corresponding test data.

[0041] Solution 11, according to the method described in Solution 1, annotates the corresponding uploaded test data according to the annotation information, further including: extracting and displaying the test data that has completed data annotation.

[0042] Solution 12, according to the method described in Solution 1, further includes: before the terminal is bound to the relevant information, connecting the terminal to a server that processes the collected test data in the background, so that the terminal can independently communicate with the server to complete data transmission.

[0043] Option 13, the method according to Option 1 further includes: the terminal includes a mobile terminal.

[0044] Solution 14, a test data annotation system, comprising: a binding device, which binds relevant information of a road test vehicle on a terminal; a generating device, which, when the vehicle is conducting a road test and collecting test data, enters annotation content corresponding to the test data collected in real time through the terminal to generate annotation information; an uploading device, which uploads the annotation information; and a data annotation device, which annotates the uploaded corresponding test data according to the annotation information.

[0045] Solution 15, according to the system described in Solution 14, the binding device specifically includes: the terminal is independent of the data acquisition system for collecting test data.

[0046] Solution 16. According to the system described in Solution 15, the binding device further includes: after binding the relevant information of the road test vehicle on the terminal, synchronizing the time of the terminal with the vehicle.

[0047] Solution 17, according to the system of Solution 14, the relevant information includes: the code and time of the vehicle.

[0048] Option 18, according to the system described in one of Options 14 to 17, the generating device specifically includes: when the vehicle is conducting a road test and collecting test data, finding the corresponding vehicle conducting the road test on the terminal based on the bound relevant information to find the corresponding real-time collected test data; entering text and / or voice information through the terminal as annotation content of the corresponding real-time collected test data.

[0049] Solution 19, according to the system described in Solution 14, the generating device specifically includes: generating annotation information on the terminal by combining the entered annotation content with the relevant information.

[0050] Solution 20, according to the system described in Solution 14, the generating device specifically includes: performing an input operation by operating the screen or voice device of the terminal.

[0051] Solution 21, according to the system described in Solution 14, the uploading device specifically includes: the terminal uploading the annotation information in real time.

[0052] Solution 22, according to the system of Solution 14, the uploading device also includes: the uploaded annotation information is stored in a database of a server that processes the collected test data in the background.

[0053] Solution 23. According to the system described in Solution 14, the data labeling device specifically includes: determining the test data corresponding to the labeling information in the uploaded test data based on the relevant information combined in the labeling information; automatically completing data labeling and storing the determined corresponding test data.

[0054] Solution 24, according to the system described in Solution 14, the data labeling device specifically includes: extracting and displaying the test data for which data labeling has been completed.

[0055] Solution 25, according to the system described in Solution 14, further includes: before the terminal is bound to the relevant information, the terminal is connected to the server that processes the collected test data in the background, so that the terminal can independently communicate with the server to complete data transmission.

[0056] Option 26, according to the system of Option 14, further includes: the terminal includes a mobile terminal.

[0057] Solution 27, a terminal, comprising: a processor and a memory; the memory is used to store a program for executing the method described in any one of Solutions 1-13; the processor is configured to execute the program stored in the memory.

[0058] Option 28, a computer-readable storage medium storing a computer program, characterized in that when the program is executed by a processor, the steps of the method described in any one of Options 1 to 13 are implemented.

[0059] Solution 29, a server device, comprising: a processor and a memory; the memory is used to store a program for executing the method described in any one of Solutions 1 to 13; the processor is configured to execute the program stored in the memory.

[0060] Scheme 30, a test data annotation system, including: a terminal, a background server for processing collected test data, and a data acquisition system; the terminal is connected to the server and is independent of the data acquisition system; the data acquisition system performs real-time collection of road test data when the vehicle undergoes road testing and uploads it to the server; the terminal is a terminal that executes the corresponding steps of the method described in any one of Schemes 1 to 13, so as to enter annotation content and generate annotation information for the test data collected when the bound vehicle undergoes road testing in real time, and upload the annotation information to the server; the server is a server that executes the corresponding steps of the method described in any one of Schemes 1 to 13, so as to receive the uploaded annotation information and test data, and use the annotation information to complete data annotation for the corresponding test data. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present description. The same reference symbols are used throughout the drawings to represent the same components. In the drawings:

[0062] Figure 1 A schematic diagram of an example application scenario of the solution of this specification;

[0063] Figure 2 A schematic diagram of a system flow chart for implementing the solution of this specification;

[0064] Figure 3 This is a flowchart of a test data annotation method in one embodiment of the solution of this specification;

[0065] Figure 4 This is a structural block diagram of a test data annotation system in one embodiment of the solution of this specification;

[0066] Figure 5 This is an example diagram showing data annotation that has been completed in one embodiment of the solution of this specification. DETAILED DESCRIPTION

[0067] The technical solution of this specification is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of this specification and the specific features in the embodiments are detailed descriptions of the technical solution of this specification, rather than limitations on the technical solution of this specification. Unless there is a conflict, the embodiments of this specification and the technical features in the embodiments can be combined with each other.

[0068] The term "and / or" in this document simply describes a relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. Additionally, the character " / " in this document generally indicates that the related objects are in an "or" relationship.

[0069] Please refer to Figure 1 , a schematic diagram of an application scenario of an embodiment involved in the technical solution of this specification. In the case of automatic driving / driving assistance, road tests are carried out by vehicles to collect road test data. Among them, the driving subject of automatic driving is the vehicle without human intervention, and the entire driving process is completed automatically by the vehicle; the driving subject of driving assistance is human, but the vehicle controls the accelerator, brakes and steering wheel instead of the human when the auxiliary function is turned on, such as adaptive cruise control ACC. In any case, the human must be prepared to take over the vehicle. The mobile terminal is bound and synchronized with the vehicle for collecting test data, and the relevant information of the corresponding vehicle is bound. These relevant information include but are not limited to: coding, time, etc. For example, the vehicle identification code VIN information of the corresponding vehicle is entered or confirmed in the mobile terminal to achieve unique binding with the vehicle, and the time of the mobile terminal is synchronized with the time of the vehicle.

[0070] During a vehicle road test, the vehicle automatically collects test data from the road being tested through its own equipment / systems, such as a data acquisition system (including but not limited to sensors, cameras, computer data file systems, etc.). During the acquisition process, an operator can also perform some input operations (such as screen input and / or voice input) through a mobile terminal associated with the vehicle at the time when annotation is required. For example, when vehicle A is traveling in a certain area at a certain time T1 and a pedestrian needs to be avoided, the operator needs to mark this situation. The operator can add real-time judgment information (e.g., "failed to automatically avoid pedestrian") to the test data D collected by vehicle A at this time T based on the VIN information on the mobile terminal. The judgment information entered in this operation, such as text or voice, is used as annotation content (this allows the annotated data to be directly extracted and analyzed in subsequent analysis). Furthermore, the annotation information can be formed by combining the VIN information and the time point of test data collection. Of course, the annotation content includes but is not limited to the operator's human judgment information, and the generated annotation information can further include relevant information about the corresponding vehicle.

[0071] The test data can be automatically uploaded by the vehicle's data acquisition system when connected to the server that processes and collects the test data in the background. It can also be manually read when not connected and uploaded when the server can be connected. The test data can be uploaded in real time or non-real time. Specifically, the collected test data is uploaded to the database of the background server when the network is connected; and the collected test data can include vehicle information and time information, etc. Similarly, the annotation information (i.e., including the annotation content) can be independently transmitted to the database of the background server in real time or non-real time through the mobile terminal. In a specific example, the mobile terminal can establish a connection with the background server (such as: installing a corresponding APP on the mobile terminal or accessing the website of the corresponding server, etc.). Once the real-time annotation of a certain test data at a certain time point is completed in real time and the annotation information is generated, the annotation information can be uploaded to the server when the network is connected (and then stored in the database). The annotation information can be uploaded in real time or non-real time to ensure that the annotation content of the corresponding test data is the content entered in real time.

[0072] Furthermore, after the collected test data enters the database, the collected test data is automatically annotated based on the bound relevant information and the annotation information in the database. For example, the annotation content in the annotation information is attached to the corresponding test data, and the annotated test data can then be extracted and displayed for easy subsequent processing. For example, the data annotation method on the server side can be achieved by mapping the annotation content in the stored annotation information to the corresponding position on the timeline of the collected test data via a timestamp. Furthermore, the data after the server completes the data annotation of the test data using the annotation information can include, for example: vehicle information, timestamp, regularized label content, manual notes (optional), etc. Thus, on the one hand, the mobile terminal is bound to the information of the test road vehicle and synchronized with the time, and the active data annotation is performed while the vehicle's data acquisition system is collecting test data, enabling real-time annotation of the corresponding test data. On the other hand, the mobile terminal is independent of the data acquisition system and independently transmits the annotation information to the database, and the background server automatically annotates the uploaded collected test data. The data annotation is timely, convenient, and flexible, facilitating subsequent data processing, improving the extraction speed of certain specific data, avoiding the subsequent workload of processing massive amounts of data, and effectively improving data annotation and processing efficiency, reducing workload, and lowering costs. Furthermore, on mobile terminals, the annotation content can also support screen input operations (i.e. detailed annotation operations) and / or voice input operations (i.e. direct voice annotation operations when a single driver is driving). These human judgment information (as the content of the annotation of the test data or the annotation content) is included in the annotation information and sent separately to the background data server. After the corresponding test data is uploaded, it is automatically bound and annotated, which can better support the extraction of data with specific characteristics.

[0073] like Figure 2A flowchart of a field system implementation is shown below. The mobile terminal is a tablet computer, bound to the corresponding vehicle, such as vehicle VIN ABC123. The vehicle collects data, obtaining test data (such as speed and lane markings). Simultaneously, the operator can use the tablet computer to enter real-time judgment information and other information as needed. For example, for test data D1 collected at 1:01:23 PM on November 13, 2019 (vehicle speed 30 km / h), the operator enters "Pedestrian Detection Failed" (annotation). This annotation is generated and uploaded to the backend server database when connected to the network. When the test data file for ABC123 is also uploaded to the backend server database, the server automatically searches for the test data for vehicle ABC123 at 1:01:23 PM on November 13, 2019, maps the annotation containing the timestamp "Pedestrian Detection Failed" to the data, and inserts the annotation "Pedestrian Detection Failed." This allows the bound relevant information to automatically annotate the test data corresponding to the annotation information in real time. The mobile terminal is linked to the vehicle and synchronized with the time. It is independent of the data acquisition system and can independently upload annotation information to the server database. When the vehicle collects test data, it can perform specific annotations using the tablet. Once the data acquisition system uploads the test data to the server database, it can automatically complete the data annotation of the test data and the corresponding annotation information. The tablet can enter the test data to be annotated in a variety of ways, such as text and voice.

[0074] Please refer to Figure 3 According to an embodiment of the present invention, a test data annotation method is provided. The method comprises at least the following steps:

[0075] S310: Binding relevant information of the road test vehicle on the terminal.

[0076] In one embodiment, a vehicle conducting road testing may be equipped with a data acquisition system, which may include but is not limited to a camera (graphics, images, etc.), a sensor (displacement, speed, etc.), a data storage system, and / or a data processing system, etc. Alternatively, an existing data acquisition system may be used.

[0077] In one embodiment, the terminal can be independent of the data acquisition system, for example, no data or signal connection is required. The terminal can preferably be a mobile terminal, such as a tablet computer, a mobile phone, etc.

[0078] Furthermore, relevant information about the vehicle undergoing the road test is bound to the mobile terminal, including but not limited to the vehicle code and time. For example, the operator enters or confirms the VIN information of the corresponding vehicle on the mobile terminal M, so that the annotation content and annotation information can be bound to the test data collected during the test of the vehicle with the corresponding VIN. In addition, the mobile terminal time is synchronized with the vehicle time so that the time point when the test data is collected during the vehicle road test is consistent with the time point when the operation information (e.g., the content annotated for the test data) is entered through the mobile terminal.

[0079] Furthermore, a mobile terminal can be bound to more than one vehicle, and a vehicle can be bound to more than one mobile terminal. However, when collecting test data, the mobile terminal used by the operator only enters annotation content for the current vehicle for which the test data is being collected. For example, mobile terminal M is bound to vehicles A, B, and C, but vehicle C is actually collecting test data. The operator following vehicle C uses his mobile terminal M to determine that it is vehicle C and annotates the test data for vehicle C in real time. For another example, mobile terminals M1, M2, and M3 are all bound to vehicles A, B, and C, but vehicle C is collecting test data. The operator following vehicle C uses the three mobile terminals M1, M2, and M3 to determine that they correspond to vehicle C, and the real-time annotations made by the three mobile terminals also correspond to the test data of vehicle C.

[0080] In one embodiment, when binding, the mobile terminal can be associated with a server related to the backend storage and processing of collected road test data, such as downloading a corresponding data annotation APP to connect to and access the server, or accessing it through a website URL, etc. The server can store the road test data uploaded by the data acquisition system through a database. Furthermore, the vehicle code can be entered in the mobile terminal, and there are various ways to enter it: manual entry, automatic scanning of barcodes / QR codes, etc., or accessing the backend server to find the corresponding vehicle code, etc. Similarly, the time when the vehicle was road tested can also be entered to determine time consistency and achieve time synchronization.

[0081] For example: The driver / operator opens the data annotation app and binds the terminal to the vehicle with VIN ABC123 undergoing road testing.

[0082] Thus, the mobile terminal is bound to the corresponding vehicle and time synchronization is completed so that the input operation of the mobile terminal can maintain temporal correspondence with the test data collected during the road test. Specifically, it can be determined that the input operation of the mobile terminal at a certain time point corresponds to the test data collected by the vehicle to which it is bound at the same time point during the road test.

[0083] S320: When the vehicle is conducting a road test and collecting test data, annotation content corresponding to the test data collected in real time is input through the terminal to generate annotation information.

[0084] In one embodiment, a vehicle travels along a road to be tested, and a data acquisition system installed on the vehicle can collect real-time test data of the road. The test data includes, but is not limited to, road information (e.g., traffic lights, number of lanes, speed limit, etc.), lane line information (e.g., color, virtual or real), surrounding vehicle information (e.g., distance, speed, direction, etc.), non-vehicle information (e.g., obstacles, pedestrians, non-motor vehicles, etc.), and vehicle data (e.g., location information, perception information, control information, etc.). Furthermore, the collected test data can be generated into a data file, uploaded to a backend server that performs data storage and processing, and further stored in a database on the server.

[0085] In one embodiment, when a vehicle is conducting a road test and collecting data, the terminal can find the vehicle that is conducting the road test based on the bound relevant information, thereby finding the corresponding real-time collected test data. In this way, the operator can use the terminal bound to the vehicle, such as the screen or voice device (such as a microphone) of a mobile terminal to operate, and use text or voice to enter some human judgment information (such as: text or voice entry of the left lane of the road is missing) corresponding to the test data collected in real time at a certain point in time as the annotation content of the corresponding real-time collected test data, and combine the annotation with various parameter information such as the bound vehicle number, time, weather, and cycle to generate annotation information. For example, the mobile terminal opens a data annotation APP or website address such as the aforementioned one that can access the backend server on the screen, completes the entry of the annotation content of the corresponding test data by operating the text entry area displayed on the screen, or completes the entry of the annotation content of the corresponding test data by using the microphone voice entry area displayed on the screen, and then combines the annotation content with the bound relevant information and time to generate annotation information. The mobile terminal independent of the data acquisition system can mark the corresponding test data in a timely manner, thereby improving the timeliness and accuracy of the annotation.

[0086] Continuing with the example of binding the ABC123 vehicle, during testing, if pedestrian detection failed at 13:01:23 on November 13, 2019, the terminal is used to record the event and select the regularized label for pedestrian detection failure. This generates and uploads a labeled data entry similar to (ABC123 2019-11-13 13:01:23 Pedestrian Detection Failed). The labeled information will include "ABC123 2019-11-13 13:01:23 Pedestrian Detection Failed."

[0087] S330: The terminal uploads the annotation information.

[0088] In one embodiment, the aforementioned annotation information that has been generated can be uploaded to a server that stores or processes test data in the background when connected to the server (connected to the Internet) through a terminal (preferably a mobile terminal) that is independent of the data acquisition system (such as the above-mentioned data acquisition system provided in the vehicle), and further stored in the database of the server. For example, you can open the aforementioned data annotation APP or website URL in the mobile terminal and choose to upload the annotation information, or automatically upload the annotation information after the input operation is completed and the annotation information is generated. The server receives the uploaded data and stores it. In this way, the mobile terminal is independent of the data acquisition system and can upload in real time or non-real time without waiting for the data acquisition system to upload data. And the terminal uploads the annotation information in real time, which helps to improve the working efficiency of the system.

[0089] S340: Annotate the uploaded corresponding test data according to the annotation information.

[0090] In one embodiment, when a vehicle conducts a road test, it can collect and upload test data in real time, or it can complete all test data collection and then upload it, or it can preset a period to periodically upload the collected test data to the backend server. The test data is transmitted to the server, which receives the uploaded test data and automatically completes the data annotation and storage by matching it with the annotation information in the database through the annotation information and parameters such as the vehicle code and time in the test data. For example, the annotation content in the annotation information is added to the corresponding test data. In this way, the real-time annotation of the corresponding test data is effectively completed, with high efficiency, low cost, and reduced workload. Continuing with the aforementioned example of binding vehicle ABC123, the server-side data processing system automatically scans all annotation information. When it processes "ABC123 2019-11-13 13:01:23 pedestrian detection failed", it will search for the test data of the corresponding vehicle ABC23 at 2019-11-13 13:01:23 and insert the annotation content "Pedestrian detection failed" at the data point where the annotation can be inserted.

[0091] Furthermore, when the backend server processes and displays data, it can extract the data that has been automatically annotated for display (see Figure 5 The example of test data that has been labeled is shown), which may include metadata (including various parameters of the test data and / or the vehicle during the road test, as well as the labeled content. The metadata may include, for example: information about the lane line of the vehicle, the position, speed, and direction of the vehicle in front of the vehicle, the speed and direction of the vehicle, etc. for subsequent analysis and use. Since the data labeling has been completed in a timely manner, it will be more efficient to extract test data with specific labeled features.

[0092] Please refer to Figure 4 , according to a test data annotation system in one embodiment of the solution of this specification. The system corresponds to the aforementioned method. The system at least includes:

[0093] Binding device 410: Binds the relevant information of the road test vehicle on the terminal. The specific processing of binding device 410 is as described in step S310.

[0094] In one embodiment, a vehicle conducting road testing may be equipped with a data acquisition system, which may include but is not limited to a camera (graphics, images, etc.), a sensor (displacement, speed, etc.), a data storage system, and / or a data processing system, etc. Alternatively, an existing data acquisition system may be used.

[0095] In one embodiment, the terminal can be independent of the data acquisition system, for example, no data or signal connection is required. The terminal can preferably be a mobile terminal, such as a tablet computer, a mobile phone, etc.

[0096] Furthermore, relevant information about the vehicle undergoing the road test is bound to the mobile terminal, including but not limited to the vehicle code and time. For example, the operator enters or confirms the VIN information of the corresponding vehicle on the mobile terminal M, so that the annotation content and annotation information can be bound to the test data collected during the test of the vehicle with the corresponding VIN. In addition, the mobile terminal time is synchronized with the vehicle time so that the time point when the test data is collected during the vehicle road test is consistent with the time point when the operation information (e.g., the content annotated for the test data) is entered through the mobile terminal.

[0097] Furthermore, a mobile terminal can be bound to more than one vehicle, and a vehicle can be bound to more than one mobile terminal. However, when collecting test data, the mobile terminal used by the operator only enters annotation content for the current vehicle for which the test data is being collected. For example, mobile terminal M is bound to vehicles A, B, and C, but vehicle C is actually collecting test data. The operator following vehicle C uses his mobile terminal M to determine that it is vehicle C and annotates the test data for vehicle C in real time. For another example, mobile terminals M1, M2, and M3 are all bound to vehicles A, B, and C, but vehicle C is collecting test data. The operator following vehicle C uses the three mobile terminals M1, M2, and M3 to determine that they correspond to vehicle C, and the real-time annotations made by the three mobile terminals also correspond to the test data of vehicle C.

[0098] In one embodiment, when binding, the mobile terminal can be associated with a server related to the backend storage and processing of collected road test data, such as downloading a corresponding data annotation APP to connect to and access the server, or accessing it through a website URL, etc. The server can store the road test data uploaded by the data acquisition system through a database. Furthermore, the vehicle code can be entered in the mobile terminal, and there are various ways to enter it: manual entry, automatic scanning of barcodes / QR codes, etc., or accessing the backend server to find the corresponding vehicle code, etc. Similarly, the time when the vehicle was road tested can also be entered to determine time consistency and achieve time synchronization.

[0099] For example: The driver / operator opens the data annotation app and binds the terminal to the vehicle with VIN ABC123 undergoing road testing.

[0100] Thus, the mobile terminal is bound to the corresponding vehicle and time synchronization is completed so that the input operation of the mobile terminal can maintain temporal correspondence with the test data collected during the road test. Specifically, it can be determined that the input operation of the mobile terminal at a certain time point corresponds to the test data collected by the vehicle to which it is bound at the same time point during the road test.

[0101] Generating device 420: When the vehicle is conducting a road test and collecting test data, the terminal inputs the annotation content corresponding to the test data collected in real time to generate annotation information. The specific processing of generating device 420 is as described in step S320.

[0102] In one embodiment, a vehicle travels along a road to be tested, and a data acquisition system installed on the vehicle can collect real-time test data of the road. The test data includes, but is not limited to, road information (e.g., traffic lights, number of lanes, speed limit, etc.), lane line information (e.g., color, virtual or real), surrounding vehicle information (e.g., distance, speed, direction, etc.), non-vehicle information (e.g., obstacles, pedestrians, non-motor vehicles, etc.), and vehicle data (e.g., location information, perception information, control information, etc.). Furthermore, the collected test data can be generated into a data file, uploaded to a backend server that performs data storage and processing, and further stored in a database on the server.

[0103] In one embodiment, when a vehicle is conducting a road test and collecting data, the terminal can find the vehicle that is conducting the road test based on the bound relevant information, thereby finding the corresponding real-time collected test data. In this way, the operator can use the terminal bound to the vehicle, such as the screen or voice device (such as a microphone) of a mobile terminal to operate, and use text or voice to enter some human judgment information (such as: text or voice entry of the left lane of the road is missing) corresponding to the test data collected in real time at a certain point in time as the annotation content of the corresponding real-time collected test data, and combine the annotation with various parameter information such as the bound vehicle number, time, weather, and cycle to generate annotation information. For example, the mobile terminal opens a data annotation APP or website address such as the aforementioned one that can access the backend server on the screen, completes the entry of the annotation content of the corresponding test data by operating the text entry area displayed on the screen, or completes the entry of the annotation content of the corresponding test data by using the microphone voice entry area displayed on the screen, and then combines the annotation content with the bound relevant information and time to generate annotation information. The mobile terminal independent of the data acquisition system can mark the corresponding test data in a timely manner, thereby improving the timeliness and accuracy of the annotation.

[0104] Continuing with the example of binding the ABC123 vehicle, during testing, if pedestrian detection failed at 13:01:23 on November 13, 2019, the terminal is used to record the event and select the regularized label for pedestrian detection failure. This generates and uploads a labeled data entry similar to (ABC123 2019-11-13 13:01:23 Pedestrian Detection Failed). The labeled information will include "ABC123 2019-11-13 13:01:23 Pedestrian Detection Failed."

[0105] Uploading device 430: The terminal uploads the annotation information. The specific processing of the uploading device 430 is as described in step S330.

[0106] In one embodiment, the aforementioned annotation information that has been generated can be uploaded to a server that stores or processes test data in the background when connected to the server (connected to the Internet) through a terminal (preferably a mobile terminal) that is independent of the data acquisition system (such as the above-mentioned data acquisition system provided in the vehicle), and further stored in the database of the server. For example, you can open the aforementioned data annotation APP or website URL in the mobile terminal and choose to upload the annotation information, or automatically upload the annotation information after the input operation is completed and the annotation information is generated. The server receives the uploaded data and stores it. In this way, the mobile terminal is independent of the data acquisition system and can upload in real time or non-real time without waiting for the data acquisition system to upload data. And the terminal uploads the annotation information in real time, which helps to improve the working efficiency of the system.

[0107] Data labeling device 440: performs data labeling on the uploaded corresponding test data according to the labeling information. The specific processing of data labeling device 440 is as described in step S340.

[0108] In one embodiment, when a vehicle conducts a road test, it can collect and upload test data in real time, or it can complete all test data collection and then upload it, or it can preset a period to periodically upload the collected test data to the backend server. The test data is transmitted to the server, which receives the uploaded test data and automatically completes the data annotation and storage by matching it with the annotation information in the database through the annotation information and parameters such as the vehicle code and time in the test data. For example, the annotation content in the annotation information is added to the corresponding test data. In this way, the real-time annotation of the corresponding test data is effectively completed, with high efficiency, low cost, and reduced workload. Continuing with the aforementioned example of binding vehicle ABC123, the server-side data processing system automatically scans all annotation information. When it processes "ABC123 2019-11-13 13:01:23 pedestrian detection failed", it will search for the test data of the corresponding vehicle ABC23 at 2019-11-13 13:01:23 and insert the annotation content "Pedestrian detection failed" at the data point where the annotation can be inserted.

[0109] Furthermore, when the backend server processes and displays data, it can extract the data that has been automatically annotated for display (see Figure 5 The example of test data that has been labeled is shown), which may include metadata (including various parameters of the test data and / or the vehicle during the road test, as well as the labeled content. The metadata may include, for example: information about the lane line of the vehicle, the position, speed, and direction of the vehicle in front of the vehicle, the speed and direction of the vehicle, etc. for subsequent analysis and use. Since the data labeling has been completed in a timely manner, it will be more efficient to extract test data with specific labeled features.

[0110] In another embodiment of the solution of this specification, a terminal is further provided, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the method steps described above when executing the program. Furthermore, the terminal includes a communicator to implement wireless or wired communication with the server to complete the access to the server and data transmission involved in the method steps described above. Furthermore, the terminal may be a mobile terminal. For ease of explanation, only the parts related to the embodiments of this specification are shown. For specific technical details not disclosed, please refer to the method section of the embodiments of this specification. The terminal may be any electronic device that can communicate, transmit, store, interact and process data, such as a mobile phone, a tablet computer, a portable computer, a PDA (Personal Digital Assistant), an on-board computer, etc., including various electronic devices.

[0111] In another embodiment of the solution of this specification, a server device is also provided, including a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the method described above when executing the program. Furthermore, the server includes a communicator to implement wireless / wired communication with a terminal or a data acquisition system for vehicle road testing to complete the steps of the method described above involving receiving data uploaded from the terminal and / or data acquisition system. Further, the memory of the server includes a database for storing data uploaded from the terminal and / or data acquisition system. For ease of explanation, only the parts related to the embodiments of this specification are shown. For specific technical details not disclosed, please refer to the method section of the embodiments of this specification. The server can be a server device formed by various electronic devices, such as a PC, a network cloud server, or even a server function set on any electronic device such as a mobile phone, a tablet computer, a PDA (Personal Digital Assistant), a POS (Point of Sales), a car computer, a desktop computer, etc.

[0112] In another embodiment of the solution of the present specification, a test data annotation system is further provided, including: a terminal, a background server for processing collected test data, and a data acquisition system; the terminal is connected to the server and is independent of the data acquisition system; the data acquisition system performs real-time acquisition of road test data when the vehicle undergoes road testing and uploads it to the server; the terminal is a terminal that executes the corresponding steps of the aforementioned method, so as to input annotation content of the test data collected when the bound vehicle undergoes road testing in real time, generate annotation information, and upload the annotation information to the server; the server is a terminal that executes the corresponding steps of the aforementioned method, so as to receive the uploaded annotation information and test data, and use the annotation information to complete data annotation for the corresponding test data.

[0113] Based on this understanding, all or part of the processes in the method of the above embodiment implemented in this specification can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, it can implement the steps of each of the above method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can include: any entity or device, medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium that can carry the computer program code. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0114] The technical solution of this specification can be used for real-time data annotation during vehicle data collection during road testing, and to perform a certain degree of data annotation on the collected test data, thereby improving the extraction speed of certain specific data. At the same time, the terminal or mobile terminal is only bound to and synchronized with the vehicle-related information. It is independent of the original data collection system and works independently. The operator operates in the subsequent collection process and enters information such as text or voice at the time when annotation is required to form marking information. Its information entry operation supports screen entry operation or voice entry operation, which supports both detailed marking operations and direct voice marking when a single driver is driving. The marking information will be submitted to the database in real time, and the collected data will be automatically annotated after entering the database. Furthermore, for the data that has been automatically annotated, the background system will extract and display the data, such as displaying the specific content that has been annotated including metadata, for subsequent analysis and use.

[0115] So far, the technical solutions of this specification have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy to understand that the scope of protection of this specification is obviously not limited to these specific embodiments. Without departing from the principles described in this specification, those skilled in the art can make equivalent changes or replacements to the relevant technical features, and the technical solutions after these changes or replacements will fall within the scope of this specification.

Claims

1. A test data annotation method, characterized in that: Applied to a terminal, the method includes: Binding relevant information of a road test vehicle to a terminal and synchronizing the time of the terminal with the time of the vehicle, wherein the relevant information includes a code of the vehicle; When the vehicle is conducting a road test and collecting test data, annotation content corresponding to the test data collected in real time is input through the terminal to generate annotation information, the annotation information including the annotation content, the vehicle code, and the time when the annotation content was input, and the test data including the vehicle code and the time when the test data was collected; Uploading the annotation information to the server in real time; in, The server is configured to: annotate the test data uploaded to the server and corresponding to the annotation information according to the annotation information; The vehicle is provided with a data acquisition system, the data acquisition system is used to collect the test data, and the terminal is independent of the data acquisition system; The marking of the test data uploaded to the server and corresponding to the marking information according to the marking information includes: Obtain annotation information having the same vehicle code and the same time as the test data, and add annotation content in the annotation information to the test data, where the same time is the same as the time when the test data is collected and the time when the annotation content is entered.

2. The method according to claim 1, wherein When the vehicle is conducting a road test and collecting test data, the terminal enters the annotation content corresponding to the test data collected in real time, specifically including: When the vehicle is conducting a road test and collecting test data, the terminal finds the vehicle that is conducting the road test according to the bound relevant information to find the corresponding real-time collected test data; Text and / or voice information is input through the terminal as annotation content corresponding to the test data collected in real time.

3. The method according to claim 1, wherein Entering annotation content corresponding to the test data collected in real time through the terminal specifically includes: The input operation is performed by operating the screen or voice device of the terminal.

4. The method according to claim 1, wherein The real-time uploading of the annotation information to the server includes: The uploaded annotation information is stored in the database of the server, which is located in the background and is used to process the collected test data.

5. The method according to claim 1, wherein The server is also used to: Extract and display the test data for which data annotation has been completed.

6. The method according to claim 1, wherein Further including: Before the terminal is bound to the relevant information, the terminal is connected to the server so that the terminal independently communicates with the server to complete data transmission. The server is located in the background and is used to process the collected test data.

7. The method according to claim 1, wherein Further including: The terminal includes a mobile terminal.

8. A test data annotation method, characterized in that: Applied to a server, the method includes: Receiving test data uploaded by a test vehicle, wherein the test data is test data collected by the vehicle during a road test, and the test data includes a code of the vehicle and a time when the test data was collected; Receive annotation information uploaded by the terminal in real time; Annotating the test data uploaded to the server and corresponding to the annotation information according to the annotation information; in, The terminal is bound to relevant information of the vehicle, the relevant information including the vehicle code, and the time of the terminal is synchronized with the time of the vehicle; The annotation information includes annotation content, the vehicle code, and the time when the annotation content was entered. The annotation information is generated by entering the annotation content corresponding to the test data collected in real time through the terminal when the vehicle is undergoing road testing and collecting test data. The vehicle is provided with a data acquisition system, the data acquisition system is used to collect the test data, and the terminal is independent of the data acquisition system; The marking of the test data uploaded to the server and corresponding to the marking information according to the marking information includes: Obtain annotation information having the same vehicle code and the same time as the test data, and add annotation content in the annotation information to the test data, where the same time is the same as the time when the test data is collected and the time when the annotation content is entered.

9. The method according to claim 8, wherein The method further comprises: Extract and display the test data for which data annotation has been completed.

10. A test data annotation system, characterized in that: The system comprises: The binding device is used to: bind relevant information of the road test vehicle to the terminal and synchronize the time of the terminal with the time of the vehicle, wherein the relevant information includes the code of the vehicle; The generating device is configured to: when the vehicle is conducting a road test and collecting test data, input annotation content corresponding to the test data collected in real time through the terminal to generate annotation information, wherein the annotation information includes the annotation content, the vehicle code, and the time when the annotation content was entered; and the test data includes the vehicle code and the time when the test data was collected; The uploading device is used to: upload the annotation information to the data annotation device in real time; The data labeling device is used to: label the test data uploaded to the data labeling device and corresponding to the labeling information according to the labeling information; in, The vehicle is provided with a data acquisition system, the data acquisition system is used to collect the test data, and the terminal is independent of the data acquisition system; The step of labeling the test data uploaded to the data labeling device and corresponding to the labeling information according to the labeling information includes: Obtain annotation information having the same vehicle code and the same time as the test data, and add annotation content in the annotation information to the test data, where the same time is the same as the time when the test data is collected and the time when the annotation content is entered.

11. The system according to claim 10, wherein: The generator is also used to: When the vehicle is conducting a road test and collecting test data, the vehicle conducting the road test is found on the terminal based on the bound relevant information to find the corresponding real-time collected test data; text and / or voice information is entered through the terminal as annotation content of the corresponding real-time collected test data.

12. The system according to claim 10, wherein: The generating device is further used to perform an input operation by operating the screen or voice device of the terminal.

13. The system according to claim 10, wherein: The uploading device is also used to store the uploaded annotation information in the database of the data annotation device. The data annotation device is a server located in the background, and the server is used to process the collected test data.

14. The system according to claim 10, wherein: The data labeling device is also used to extract and display the test data for which data labeling has been completed.

15. The system according to claim 10, wherein: Further including: Before the terminal is bound to the relevant information, the terminal is connected to the data labeling device so that the terminal can independently communicate with the data labeling device to complete data transmission. The data labeling device is a server located in the background, and the server is used to process the collected test data.

16. The system according to claim 10, wherein: Further including: The terminal includes a mobile terminal.

17. A terminal, characterized in that: include: processor and memory; The memory is used to store a program for executing the method according to any one of claims 1 to 7; The processor is configured to execute the program stored in the memory.

18. A computer-readable storage medium storing a computer program, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

19. A server, characterized in that: include: processor and memory; The memory is used to store a program for executing the method according to any one of claims 8 to 9; The processor is configured to execute the program stored in the memory.

20. A test data annotation system, characterized in that: include: Terminal, backend server for processing collected test data, data acquisition system; The terminal is connected to the backend server, and the terminal is independent of the data acquisition system; The data acquisition system collects real-time road test data when the vehicle is undergoing road testing and uploads the data to the backend server; The terminal is a terminal for executing the corresponding steps of the method according to any one of claims 1 to 7, for real-time inputting annotation content of test data collected during road testing of the bound vehicle, generating annotation information, and uploading the annotation information to the backend server; The background server is a server that executes the corresponding steps of the method according to any one of claims 8 to 9, receives uploaded annotation information and test data, and uses the annotation information to complete data annotation for the corresponding test data.

Images