Method, device, electronic device and storage medium for testing event detection algorithm

Abstract

The present disclosure provides a method, device, electronic device and storage medium for testing an event detection algorithm, which relates to the field of artificial intelligence technology, especially to the field of unmanned driving and intelligent transportation. The specific implementation scheme is: determining target test data according to road information and vehicle information; processing the target test data using a target event detection algorithm to obtain an output result for the target test data; and determining the test result of the target event detection algorithm according to the output result and a reference value.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to the fields of unmanned driving and intelligent transportation. More specifically, the present disclosure provides a method, device, electronic device, storage medium, and computer program product for testing an event detection algorithm. Background Art

[0002] The AI ​​(artificial intelligence) engine can integrate multiple event detection algorithms, each of which can be used to detect whether a target event has occurred in a vehicle, such as whether the vehicle is driving in the wrong direction or crossing a solid line. The staff needs to test each event detection algorithm and put it into use after the event detection algorithm passes the test, thereby realizing the digitalization and intelligence of transportation. Summary of the invention

[0003] The present disclosure provides a method, an apparatus, an electronic device, a storage medium, and a computer program product for testing an event detection algorithm.

[0004] According to one aspect of the present disclosure, a method for testing an event detection algorithm is provided, comprising: determining target test data based on road information and vehicle information; processing the target test data using a target event detection algorithm to obtain an output result for the target test data; and determining a test result of the target event detection algorithm based on the output result and a benchmark value.

[0005] According to another aspect of the present disclosure, a device for testing an event detection algorithm is provided, comprising a data determination module, an output result determination module and a test result determination module. The data determination module is used to determine target test data according to road information and vehicle information; the output result determination module is used to process the target test data using the target event detection algorithm to obtain an output result for the target test data; and the test result determination module is used to determine the test result of the target event detection algorithm according to the output result and a reference value.

[0006] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method provided by the present disclosure.

[0007] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the method provided by the present disclosure.

[0008] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the method provided by the present disclosure is implemented.

[0009] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0011] Figure 1 is a schematic diagram of an application scenario of a method and device for testing an event detection algorithm according to an embodiment of the present disclosure;

[0012] Figure 2 is a schematic flow chart of a method for testing an event detection algorithm according to an embodiment of the present disclosure;

[0013] Figure 3 is a schematic flow chart of a method for determining an evaluation index according to an embodiment of the present disclosure;

[0014] Figure 4 is a schematic flow chart of a method for determining target test data according to an embodiment of the present disclosure;

[0015] Figure 5 is a schematic diagram of a method for testing an event detection algorithm according to an embodiment of the present disclosure;

[0016] Figure 6 is a schematic structural block diagram of an apparatus for testing an event detection algorithm according to an embodiment of the present disclosure; and

[0017] Figure 7 It is a structural block diagram of an electronic device used to implement the method of the test event detection algorithm of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0018] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0019] Figure 1 It is a schematic diagram of an application scenario of a method and device for a test event detection algorithm according to an embodiment of the present disclosure.

[0020] It should be noted that Figure 1 What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0021] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0022] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptops, desktop computers, etc.

[0023] The server 105 may be a server that provides various services, such as a background management server (only an example) that provides support for websites browsed by users using the terminal devices 101, 102, and 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as the test results of the target event detection algorithm determined based on the road information and vehicle information, etc.) to the terminal device.

[0024] It should be noted that the method for testing the event detection algorithm provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the device for testing the event detection algorithm provided in the embodiment of the present disclosure can generally be set in the server 105. The method for testing the event detection algorithm provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the device for testing the event detection algorithm provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0025] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0026] Figure 2is a schematic flowchart of a method for testing an event detection algorithm according to an embodiment of the present disclosure.

[0027] like Figure 2 As shown, the method 200 for testing an event detection algorithm may include operations S210 to S230.

[0028] In operation S210, target test data is determined based on road information and vehicle information.

[0029] The event detection algorithm can be used to detect whether the vehicle has committed any traffic violation. For example, the event detection algorithm can be used to detect whether the vehicle has committed any traffic violation such as crossing a solid line, changing lanes on a solid line, changing lanes continuously, driving in the wrong direction, a motor vehicle entering a non-motor vehicle lane, or abnormal parking.

[0030] Testing an event detection algorithm may mean testing the event detection algorithm to be detected. After the test, test results such as the recall rate, accuracy rate, and precision rate of the event detection algorithm may be obtained, thereby determining whether the event detection algorithm can be put into use online.

[0031] The road information may include at least one of a lane line, a lane driving direction, and a lane no-parking area.

[0032] For example, road information can be pulled from the video annotation platform through the interface of the video annotation platform. The video annotation platform can process the video data and annotate the road information contained in the video.

[0033] The vehicle information may include dynamic information, such as at least one of the vehicle's driving track, the vehicle's driving direction, and the vehicle's parking location. The vehicle information may also include static information, such as at least one of the vehicle license plate number, the vehicle brand, and the vehicle body color.

[0034] For example, vehicle information can be pulled from a video parsing service. The video parsing service is used to parse video data to obtain vehicle information. The data packet containing the vehicle information can be downloaded from the server where the video parsing service is located through an interface to achieve cross-network transmission of the vehicle information.

[0035] In the process of pulling vehicle information, in some embodiments, the full amount of vehicle information can be pulled. In other embodiments, incremental vehicle information can also be pulled by timestamp, for example, the displacement of the data pulled last time can be recorded by timestamp, and the server where the video analysis service is located can be scanned regularly, and incremental vehicle information can be found during the scanning process, so as to pull the incremental vehicle information.

[0036] In addition, road information and vehicle information can be stored separately according to dimensions such as video identification and time, making it easier to trace the road information and vehicle information.

[0037] The target test data may represent data that needs to be processed by the target event detection algorithm. The target test data may be stored, for example, in a YAML (a data serialization format that can be recognized by computing) format.

[0038] In some examples, when the formats of the road information and the vehicle information meet the input format requirements of the target event detection algorithm, the road information and the vehicle information can be directly input into the target event detection algorithm as target test data.

[0039] In other examples, when the format of the road information and vehicle information does not meet the input format requirements of the target event detection algorithm, the road information and vehicle information can be modified into parameters that meet the input format of the target event detection algorithm, and then the parameters are determined as target test data.

[0040] In operation S220, the target test data is processed using a target event detection algorithm to obtain an output result for the target test data.

[0041] The output result may indicate whether the vehicle has triggered the target event to be detected by the target event detection algorithm. It should be understood that the output result is determined by the target event detection algorithm, and when the accuracy of the target event detection algorithm is low, the output result may be inaccurate.

[0042] In some embodiments, the output result of the target event detection algorithm may include a confidence level. When the confidence level is greater than or equal to a confidence level threshold, it indicates that the target event has occurred in the vehicle, such as a violation of the vehicle's compaction line. When the confidence level is less than the confidence level threshold, it indicates that the target event has not occurred in the vehicle. The confidence level may be 0.8.

[0043] In other embodiments, the output result of the target event detection algorithm may include a reference value. For example, when the confidence level is greater than or equal to a confidence threshold, a prompt message of "a target event has occurred in the vehicle" is output; when the confidence level is less than the confidence threshold, a prompt message of "no target event has occurred in the vehicle" is output.

[0044] In operation S230, a test result of the target event detection algorithm is determined according to the output result and the reference value.

[0045] The reference value may indicate whether the vehicle corresponding to the target test data triggers the correct result of the target event.

[0046] In some embodiments, image data can be pulled from a video analysis service, and the image data can include at least one of subject image data and background image data, and the subject can include motor vehicles, non-motor vehicles, pedestrians, etc. The background image data can reflect information such as the position of the subject on the road and the driving trajectory. For example, the image data can be viewed by a staff member, and a benchmark value can be determined based on the image data. For example, a model can also be pre-trained, and then the model can be used to annotate the benchmark value. In addition, the image data can also be stored, for example, using HDFS (Hadoop Distributed File System) to store the image data.

[0047] The test result may include at least one of a recall rate, an accuracy rate, a precision rate, a number of correct output results, and a number of incorrect output results.

[0048] In some embodiments, the above operations S210 to S230 may be performed once at a predetermined interval, and the predetermined interval may be 1 day. The test results obtained each time may also be stored, and a version number may be configured for each test result during the storage process, so that each test result may be stored in isolation according to the version number, so that the stored data may be traced, screened, and reused. In addition, the test results may be cleaned up regularly, for example, only the test data within the past 30 days may be retained, thereby reducing the amount of data storage.

[0049] In some technical solutions, the testing process of the event detection algorithm is implemented by manual operation of the staff, such as manually screening videos, cropping videos, combining target test data, manually comparing benchmark values ​​and outputting results, etc. It can be seen that the above technical solutions require the staff to spend a lot of time.

[0050] Compared with the above-mentioned technical solution of manual operation by staff, the technical solution provided by the embodiment of the present disclosure uses the target event detection algorithm to process road information and vehicle information to obtain output results, and also determines the test results of the target event detection algorithm based on the output results and the benchmark value. Therefore, the target event detection algorithm can be automatically tested, thereby reducing the workload of staff, reducing test time, and improving test efficiency.

[0051] According to another embodiment of the present disclosure, the operation of determining target test data based on road information and vehicle information may include the following operations: combining the road information, vehicle information and the predetermined parameter template into target test data based on a first predetermined mapping relationship between fields of the road information and fields of the predetermined parameter template, and a second predetermined mapping relationship between fields of the vehicle information and fields of the predetermined parameter template.

[0052] For example, the predetermined parameter template can be customized according to actual needs, and the road information can be traversed and pulled according to the first predetermined mapping relationship in a multi-threaded manner, and then the field of the road information is filled into the corresponding field of the predetermined parameter template. The vehicle information can be traversed and pulled according to the second predetermined mapping relationship, and then the field of the vehicle information is filled into the corresponding field of the predetermined parameter template, so as to obtain the target test data.

[0053] It should be noted that different target event detection algorithms to be tested may require different input formats of target test data. The disclosed embodiment predetermines a predetermined parameter template corresponding to the target event detection algorithm, and then uses the predetermined parameter template to combine road information and vehicle information into target test data, thereby realizing the configuration of target test data. When the input format of the target test data changes, the target test data can be quickly determined by adjusting the predetermined parameter template, thereby facilitating data maintenance.

[0054] According to another embodiment of the present disclosure, the operation of processing target test data using a target event detection algorithm to obtain an output result for the target test data may include the following operations: generating a test case according to the target test data, and then calling an interface of the target event detection algorithm through the test case to obtain an output result.

[0055] For example, a code generator can be developed based on a data-driven approach (a test method that reads test data from a data file and drives the test process), and then the target test data can be combined with the code generated by the code generator to obtain a test case based on pytest (a unit test framework for python). In addition, the code can include an interface request method, which is used to call the interface of the target event detection algorithm to be tested, so that after running the test case, the target event detection algorithm will process the target test data to obtain the output result.

[0056] The technical solution provided by the embodiment of the present disclosure utilizes test cases to determine output results, which can improve test efficiency and reduce the workload required for testing.

[0057] According to another embodiment of the present disclosure, determining the test result of the target event detection algorithm based on the output result and the benchmark value may include the following operations: comparing the output result with the benchmark value to obtain an evaluation index, and then determining the test result based on the evaluation index.

[0058] For example, the evaluation index may include at least one of a first index, a second index, a third index, and a fourth index, wherein the first index indicates that the output result is a positive example and the output result is correct, the second index indicates that the output result is a positive example and the output result is wrong, the third index indicates that the output result is a negative example and the output result is correct, and the fourth index indicates that the output result is a negative example and the output result is wrong.

[0059] For example, the precision can be calculated by formula 1, and the recall can be calculated by formula 2.

[0060]

[0061]

[0062] Among them, acc represents accuracy, rec represents recall, TP represents the first indicator, FP represents the second indicator, TN represents the third indicator, and FN represents the fourth indicator.

[0063] The embodiment of the present disclosure first determines the evaluation index, and then can use different formulas to calculate a variety of test results, thereby comprehensively testing the target event detection algorithm.

[0064] Figure 3 is a schematic flowchart of a method for determining evaluation indicators according to an embodiment of the present disclosure.

[0065] like Figure 3 As shown, in this embodiment, the operation of comparing the output result with the reference value to obtain the evaluation index may include operations S301 to S308.

[0066] In operation S301, the values ​​of the evaluation indicators are initialized. For example, the initial values ​​of the first indicator, the second indicator, the third indicator, and the fourth indicator are all set to 0.

[0067] In operation S302, determine whether the confidence of the output result is greater than the confidence threshold. For example, the output result after the target event detection algorithm processes the target test data includes the confidence, and compare the size relationship between the confidence and the confidence threshold. If the confidence is greater than the confidence threshold, proceed to operation S303; if the confidence is less than or equal to the confidence threshold, proceed to operation S306.

[0068] In operation S303, it is determined whether the reference value is 1. A reference value of 1 may indicate that a target event occurs to the vehicle, such as a vehicle crossing a solid line. If yes, the process proceeds to operation S304. If no, the process proceeds to operation S305.

[0069] In operation S304, the value of the first index is increased by 1.

[0070] In operation S305 , the value of the second index is increased by 1.

[0071] In operation S306, it is determined whether the reference value is 1. If not, the process proceeds to operation S307. If yes, the process proceeds to operation S308.

[0072] In operation S307 , the value of the third index is increased by 1.

[0073] In operation S308, the value of the fourth index is increased by 1.

[0074] According to another embodiment of the present disclosure, the method of the above-mentioned test event detection algorithm may further include the following operations: searching for image data according to the storage path of the image data corresponding to the target test data, and then tiling the image data.

[0075] In some technical solutions, when pulling image data from a video analysis service, the same vehicle driving trajectory corresponds to multiple types of image data, such as main image data and background image data, and different types of image data or multiple image data of the same type are stored in different directories. Staff need to manually open multiple directories to find the required image data.

[0076] Compared with the above technical solutions, the embodiments of the present disclosure can display the image data corresponding to the target test data in a tiled manner, such as in an Excel file, without the need for staff to manually open multiple directories to search for images, thereby improving the efficiency of staff in determining benchmark values.

[0077] In addition, in some embodiments, the relevant data of each test case can be displayed in a flat format, and the relevant information may include the use case title, use case description, the identification of the test data corresponding to each test case, the target event detection algorithm identification, the output results, the benchmark value, etc.

[0078] According to another embodiment of the present disclosure, a test report in a predetermined format may be generated, and the test report may be sent to the staff via email, etc. The content of the test report may include information such as the number of test cases and the test results of the target event detection algorithm.

[0079] According to another embodiment of the present disclosure, the test results can be monitored, for example, the fluctuation of the test results can be detected and a fluctuation curve can be drawn. In addition, a fluctuation threshold can be set, and when the difference between the maximum and minimum values ​​of the test results within a predetermined period is greater than or equal to the fluctuation threshold, an alarm prompt is issued through communication tools such as email.

[0080] It should be noted that in some embodiments, the collected vehicle information and road information involve multiple areas, and some scenarios only need to determine the test results of specific areas. For example, for a scenario a, among multiple parallel lanes, one part is a forward driving lane and the other part is a reverse driving lane. It is necessary to detect whether the vehicle in the forward driving lane has pressed the solid line, and it is temporarily unnecessary to detect whether the vehicle in the reverse driving lane has pressed the solid line. It can be seen that if the test result is determined based on the vehicle information corresponding to the vehicle in the reverse driving lane, the accuracy of the test result will be reduced.

[0081] Figure 4 is a schematic flowchart of a method for determining target test data according to an embodiment of the present disclosure.

[0082] like Figure 4 As shown, in this embodiment, the road information includes actual road information and pseudo road information. Accordingly, the operation of determining the target test data according to the road information and the vehicle information may include the following operations S411 to S412.

[0083] In operation S411, the pseudo road information and the vehicle information are processed using a target event detection algorithm to obtain valid vehicle information in the vehicle information.

[0084] The actual road information may represent real road information, such as real lane lines, real driving directions, real no-parking areas, etc.

[0085] Pseudo road information can represent non-real road information. For example, when the target event to be detected by the target event detection algorithm is the vehicle compaction line, the center line of the road can be used as pseudo road information (hereinafter referred to as pseudo lane line information). For example, when the target event to be detected by the target event detection algorithm is the vehicle reversing, the direction opposite to the real driving direction of the road can be used as pseudo road information (hereinafter referred to as pseudo driving direction information).

[0086] For example, in the above scenario a, the center line of the forward driving lane can be used as pseudo lane line information.

[0087] Then, according to the first predetermined mapping relationship and the second predetermined mapping relationship, the pseudo lane line information, the vehicle information and the predetermined parameter template can be combined into pseudo test data, and then a pseudo test case is generated according to the pseudo test data, and then the pseudo test case is run. The pseudo test case will call the interface of the target event detection algorithm to obtain the output result for the pseudo test data.

[0088] Since the position indicated by the pseudo lane line information at this time is the center line of the forward driving lane, all vehicles in the forward driving lane will press the center line, thereby triggering the target event detected by the target event detection algorithm. Therefore, the target event detection algorithm can filter out the vehicle information corresponding to the vehicle in the forward driving lane from the full amount of vehicle information (including the vehicle information corresponding to the vehicle in the forward driving lane and the vehicle information corresponding to the vehicle in the reverse driving lane), and use the filtered vehicle information as the valid vehicle information.

[0089] For example, in another scenario b, among multiple parallel lanes, some are forward driving lanes and the other are reverse driving lanes. It is necessary to detect whether the vehicle in the forward driving lane is reverse driving, but it is not necessary to detect whether the vehicle in the reverse driving lane is reverse driving. At this time, the reverse direction of the driving direction of the forward driving lane can be used as pseudo driving direction information.

[0090] Then, the pseudo driving direction information, vehicle information and the predetermined parameter template can be combined into pseudo test data, and then the pseudo test data can be processed by the target event detection algorithm to obtain the output result.

[0091] Since the direction indicated by the pseudo lane line information is opposite to the real direction at this time, the vehicles traveling in the forward driving lane will be judged as going in the wrong direction by the target event detection algorithm. Therefore, the target event detection algorithm can filter out the vehicle information corresponding to the vehicles in the forward driving lane from the full amount of vehicle information, and use the filtered vehicle information as the valid vehicle information.

[0092] In operation S412, target test data is determined according to the actual road information and the valid vehicle information.

[0093] For example, actual road information, vehicle information, and a predetermined parameter template may be combined into target test data according to a first predetermined mapping relationship and a second predetermined mapping relationship.

[0094] The technical solution provided by the embodiment of the present disclosure adopts a mock scenario construction method to filter out valid vehicle information from the full amount of vehicle information, thereby avoiding the problem of low accuracy of test results.

[0095] Figure 5 is a schematic diagram of a method for testing an event detection algorithm according to an embodiment of the present disclosure.

[0096] like Figure 5As shown, this embodiment may involve a video analysis service 501, a video annotation platform 502, a test data synthesizer 503, a code generator 504, an interface request implementation layer 505, a valid vehicle information collector 506, a benchmark value generator 507, a test report generation module 508, a monitoring alarm module 509 and a storage device 510.

[0097] The video analysis service 501 can process the video data to obtain vehicle information and image data. The vehicle information and image data can be pulled from the video analysis service 501.

[0098] The video annotation platform 502 can process the video data to obtain road information for the road. The road information can be pulled from the video annotation platform 502, and the road information can include actual road information and pseudo road information.

[0099] The test data synthesizer 503 may combine the pseudo road information, the vehicle information and the predetermined parameter template into pseudo test data according to the first predetermined mapping relationship and the second predetermined mapping relationship.

[0100] The effective vehicle information collector 506 can filter out effective vehicle information from the full amount of vehicle information, for example, filter out the vehicle information corresponding to the vehicle located in the forward driving lane as mentioned above.

[0101] Valid vehicle information can be screened in the following manner: pseudo test data can be combined with the code generated by the code generator 504 to obtain a pseudo test case. The interface to be called can be encapsulated in the interface request implementation layer 505, and the interface to be called can include the interface of the target event detection algorithm to be tested. The test case can call the interface of the target event detection algorithm in the interface request implementation layer 505, so that the target event detection algorithm processes the pseudo test data and obtains the output result for the pseudo test data. The valid vehicle information collector 506 can screen out valid vehicle information from the full amount of vehicle information based on the output result for the pseudo test data.

[0102] Then, the test data synthesizer 503 and the code generator 504 can be used again to determine the target test data through the actual road information and the effective vehicle information. For example, the test data synthesizer 503 combines the actual road information, the effective vehicle information and the predetermined parameter template into the target test data according to the first predetermined mapping relationship and the second predetermined mapping relationship. Then, the code generator 504 combines the target test data with the code generated by the code generator 504 to obtain the target test case. Then, the target test case calls the interface request implementation layer 505, so that the target event detection algorithm processes the target test data and obtains the output result for the target test data.

[0103] The image data from the video analysis service 501 and the relevant data of each test case can be displayed in a tiled manner so that the staff can mark the benchmark values.

[0104] The reference value generator 507 can bind the reference value, the use case identifier, and the output result to obtain the corresponding relationship between the reference value, the use case identifier, and the output result. Then the reference value generator 507 can call the interface of the interface request implementation layer 505, and the interface request implementation layer 505 compares the reference value and the output result for the target test data to obtain the test result.

[0105] The test report generating module 508 may generate a test report in a predetermined format according to the test results.

[0106] The monitoring alarm module 509 can draw a fluctuation curve according to the fluctuation of the test results, and can also issue an alarm prompt through communication tools such as email when the difference between the maximum and minimum values ​​of the test results within a predetermined period is greater than or equal to the fluctuation threshold.

[0107] The storage device 510 can store data generated in the process of determining the test results, for example, at least one of image data, vehicle information, road information, pseudo test data, target test data, valid vehicle information, and test results. When storing, image data can be stored through HDFS, and non-image data can be stored through YAML.

[0108] Figure 6 It is a schematic structural block diagram of an apparatus for testing an event detection algorithm according to an embodiment of the present disclosure.

[0109] like Figure 6 As shown, the device 600 for testing an event detection algorithm may include a data determination module 610 , an output result determination module 620 and a test result determination module 630 .

[0110] The data determination module 610 is used to determine target test data according to the road information and the vehicle information.

[0111] The output result determination module 620 is used to process the target test data using the target event detection algorithm to obtain the output result for the target test data.

[0112] The test result determination module 630 is used to determine the test result of the target event detection algorithm according to the output result and the reference value.

[0113] According to another embodiment of the present disclosure, the data determination module includes a combination submodule, which is used to combine road information, vehicle information and a predetermined parameter template into target test data based on a first predetermined mapping relationship between fields of road information and fields of a predetermined parameter template, and a second predetermined mapping relationship between fields of vehicle information and fields of a predetermined parameter template.

[0114] According to another embodiment of the present disclosure, the output result determination module includes a generation submodule and a call submodule. The generation submodule is used to generate a test case according to the target test data; the call submodule is used to call the interface of the target event detection algorithm through the test case to obtain the output result.

[0115] According to another embodiment of the present disclosure, the test result determination module includes a comparison submodule and a first determination submodule. The comparison submodule is used to compare the output result with the reference value to obtain an evaluation index; the first determination submodule is used to determine the test result according to the evaluation index; wherein the evaluation index includes at least one of the first index, the second index, the third index, and the fourth index, the first index indicates that the output result is a positive example and the output result is correct, the second index indicates that the output result is a positive example and the output result is wrong, the third index indicates that the output result is a negative example and the output result is correct, and the fourth index indicates that the output result is a negative example and the output result is wrong.

[0116] According to another embodiment of the present disclosure, the device for testing event detection algorithm further includes a search module and a display module. The search module is used to search for image data according to the storage path of image data corresponding to target test data; the display module is used to tile and display the image data.

[0117] According to another embodiment of the present disclosure, the road information includes actual road information and pseudo road information; the data determination module includes a screening submodule and a second determination submodule. The screening submodule is used to process the pseudo road information and vehicle information using a target event detection algorithm to obtain valid vehicle information in the vehicle information; the second determination submodule is used to determine the target test data according to the actual road information and the valid vehicle information.

[0118] According to another embodiment of the present disclosure, when the target event to be detected by the target event detection algorithm is a vehicle compaction line, the pseudo road information is the center line of the road; and when the target event to be detected by the target event detection algorithm is a vehicle reversing, the pseudo road information is pseudo driving direction information that is opposite to the actual driving direction of the road.

[0119] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0120] In the technical solution of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0121] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, comprising at least one processor; and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned method of the test event detection algorithm.

[0122] According to an embodiment of the present disclosure, the present disclosure further provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the method of the above-mentioned test event detection algorithm.

[0123] According to an embodiment of the present disclosure, the present disclosure further provides a computer program product, including a computer program, and the computer program implements the method of the above-mentioned test event detection algorithm when being executed by a processor.

[0124] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0125] like Figure 7 As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0126] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0127] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as the method of testing the event detection algorithm. For example, in some embodiments, the method of testing the event detection algorithm may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the method of testing the event detection algorithm described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the method of testing the event detection algorithm by any other appropriate means (e.g., by means of firmware).

[0128] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0129] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0130] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0131] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0132] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0133] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0134] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0135] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for testing an event detection algorithm, comprising: Determine target test data according to road information and vehicle information; wherein the road information includes actual road information and pseudo road information; Processing the target test data using a target event detection algorithm to obtain an output result for the target test data; and Determining a test result of the target event detection algorithm according to the output result and the benchmark value; Among them, determining the target test data based on the road information and the vehicle information includes: using the target event detection algorithm to process the pseudo road information and the vehicle information to obtain valid vehicle information in the vehicle information; determining the target test data based on the actual road information and the valid vehicle information.

2. The method according to claim 1, wherein: Determining target test data according to the road information and the vehicle information includes: According to a first predetermined mapping relationship between the field of the road information and the field of the predetermined parameter template, and a second predetermined mapping relationship between the field of the vehicle information and the field of the predetermined parameter template, the road information, the vehicle information and the predetermined parameter template are combined into the target test data.

3. The method according to claim 1, wherein: The processing of the target test data by using the target event detection algorithm to obtain an output result for the target test data includes: Generate test cases according to the target test data; and The output result is obtained by calling the interface of the target event detection algorithm through the test case.

4. The method according to claim 1, wherein: Determining the test result of the target event detection algorithm according to the output result and the reference value includes: Comparing the output result with the reference value to obtain an evaluation index; and Determining the test result according to the evaluation index; Among them, the evaluation index includes at least one of a first index, a second index, a third index, and a fourth index, the first index indicates that the output result is a positive example and the output result is correct, the second index indicates that the output result is a positive example and the output result is wrong, the third index indicates that the output result is a negative example and the output result is correct, and the fourth index indicates that the output result is a negative example and the output result is wrong.

5. The method according to claim 1, further comprising: searching the image data according to a storage path of the image data corresponding to the target test data; as well as The image data is displayed in a tiled manner.

6. The method according to any one of claims 1 to 5, wherein: In the case where the target event to be detected by the target event detection algorithm is a vehicle compaction line, the pseudo road information is the center line of the road; as well as In the case that the target event to be detected by the target event detection algorithm is a vehicle traveling in the wrong direction, the pseudo road information is pseudo driving direction information opposite to the actual driving direction of the road.

7. A device for testing an event detection algorithm, comprising: A data determination module, used to determine target test data according to road information and vehicle information; wherein the road information includes actual road information and pseudo road information; an output result determination module, configured to process the target test data using a target event detection algorithm to obtain an output result for the target test data; and A test result determination module, used to determine the test result of the target event detection algorithm according to the output result and the reference value; Wherein, the data determination module includes: a screening submodule, configured to process the pseudo road information and the vehicle information using the target event detection algorithm to obtain valid vehicle information in the vehicle information; and The second determination submodule is used to determine the target test data according to the actual road information and the valid vehicle information.

8. The device according to claim 7, wherein: The data determination module comprises: A combination submodule is used to combine the road information, the vehicle information and the predetermined parameter template into the target test data according to a first predetermined mapping relationship between the field of the road information and the field of the predetermined parameter template, and a second predetermined mapping relationship between the field of the vehicle information and the field of the predetermined parameter template.

9. The device according to claim 7, wherein: The output result determination module comprises: A generating submodule, used to generate a test case according to the target test data; and The calling submodule is used to call the interface of the target event detection algorithm through the test case to obtain the output result.

10. The device according to claim 7, wherein: The test result determination module comprises: A comparison submodule, used to compare the output result with the reference value to obtain an evaluation index; and A first determination submodule, used to determine the test result according to the evaluation index; Among them, the evaluation index includes at least one of a first index, a second index, a third index, and a fourth index, the first index indicates that the output result is a positive example and the output result is correct, the second index indicates that the output result is a positive example and the output result is wrong, the third index indicates that the output result is a negative example and the output result is correct, and the fourth index indicates that the output result is a negative example and the output result is wrong.

11. The apparatus according to claim 7, further comprising: A search module, used for searching the image data according to a storage path of the image data corresponding to the target test data; as well as A display module is used to display the image data in a tiled manner.

12. The device according to any one of claims 7 to 11, wherein: In the case where the target event to be detected by the target event detection algorithm is a vehicle compaction line, the pseudo road information is the center line of the road; as well as In the case that the target event to be detected by the target event detection algorithm is a vehicle traveling in the wrong direction, the pseudo road information is pseudo driving direction information opposite to the actual driving direction of the road.

13. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 6.

15. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images