Vehicle perception system evaluation method, vehicle control method and storage medium

Abstract

The embodiment of the invention provides a vehicle sensing system evaluation method, a vehicle control method and a storage medium, and relates to the technical field of automatic driving. The vehicle sensing system evaluation method comprises the following steps: acquiring a sensing result sequence generated by a vehicle sensing system based on a sensor data sequence; wherein the sensor data sequence comprises multiple frames of sensor data detected by a vehicle sensor, different frames of sensor data correspond to different timestamps, the sensing result sequence comprises multiple frames of sensing results corresponding to different timestamps, and the sensing results comprise state features of multiple sensing objects; generating a perception evaluation index for the vehicle perception system according to the perception result sequence; wherein the perception evaluation index comprises a first stability index used for representing the time sequence stability of the relative state characteristics of different perception objects at adjacent timestamps.

Description

Technical Field

[0001] The present disclosure relates to the field of autonomous driving technology, and more particularly, to a method for evaluating a vehicle perception system, a vehicle control method, and a storage medium. Background Art

[0002] With the development of autonomous driving technology, the perception ability of the perception system has become an important factor affecting the performance of autonomous driving. The perception system can use various modal data such as image data generated by cameras and point cloud data generated by lidar to perform feature processing using machine learning and deep learning algorithms to obtain the perception results of the vehicle on the environment, such as the perception results of various vehicles, pedestrians, cones, lanes, traffic lights, etc. in the actual spatial environment. The autonomous driving vehicle can perform planning, decision-making, and control based on the perception results of the perception system to control the vehicle to drive normally. Therefore, the perception results of the perception system are crucial for the control of autonomous driving vehicles, and it is necessary to evaluate the perception results of the vehicle perception system to determine the performance of the vehicle perception system. However, in the related art, it is necessary to manually judge the accuracy of the perception results, or evaluate the perception results based on a manually set test set and corresponding ground truth data, resulting in low efficiency of the perception system evaluation. Summary of the Invention

[0003] In view of this, an embodiment of the present disclosure proposes a new technical solution for evaluating a vehicle perception system.

[0004] According to a first aspect of an embodiment of the present disclosure, there is provided a method for evaluating a vehicle perception system, the method including:

[0005] Obtaining a perception result sequence generated by a vehicle perception system based on a sensor data sequence; wherein, the sensor data sequence includes multiple frames of sensor data detected by vehicle sensors, different frames of sensor data correspond to different timestamps, the perception result sequence includes multiple frames of perception results corresponding to different timestamps respectively, and the perception results include state features of multiple perception objects;

[0006] Generating a perception evaluation index for the vehicle perception system according to the perception result sequence; wherein, the perception evaluation index includes a first stability index for characterizing the temporal stability of the relative state features between different perception objects at adjacent timestamps.

[0007] Optionally, the generating a perception evaluation index for the vehicle perception system according to the perception result sequence includes:

[0008] Fusing the state features of different perception objects according to the perception result sequence to obtain a first state feature sequence; wherein, the first state feature sequence includes multiple frames of relative state features corresponding to different timestamps;

[0009] Performing stability detection on the first state feature sequence based on a preset first stability detection algorithm to obtain the first stability index.

[0010] Optionally, the fusing the state features of different perception objects according to the perception result sequence to obtain a first state feature sequence includes:

[0011] Determining a second state feature sequence corresponding to each perception object according to the perception result sequence; wherein the second state feature sequence includes second state features of the perception object corresponding to different timestamps;

[0012] Determining a group of perception objects to be fused according to the object type of the perception object; wherein the object type includes a dynamic type, a static discrete type, and a static continuous type;

[0013] Fusing the second state features of different perception objects in the group of perception objects to be fused at the same timestamp to obtain the first state feature sequence.

[0014] Optionally, the first stability detection algorithm includes a Markov chain algorithm and / or a Markov random field algorithm; the performing stability detection on the first state feature sequence based on a preset first stability detection algorithm to obtain the first stability index includes:

[0015] Performing stability detection on the first state feature sequence based on the first stability detection algorithm to determine an abnormal state timestamp from the first state feature sequence; wherein the abnormal state timestamp is a timestamp at which the relative state feature jumps;

[0016] Determining the first stability index according to the proportion of the abnormal state timestamp in the first state feature sequence.

[0017] Optionally, the method further includes:

[0018] Generating multimedia data based on the perception result sequence corresponding to the abnormal state timestamp;

[0019] Displaying the first stability index and the multimedia data to the user.

[0020] Optionally, the relative state feature includes one or more of the following:

[0021] A relative position feature for characterizing the relative position relationship of different perception objects;

[0022] A coexistence state feature for characterizing whether different perception objects exist in the same frame of perception result.

[0023] Optionally, the perception evaluation index further includes a second stability index for characterizing the temporal stability of the state characteristics of the same perception object at adjacent timestamps; generating the perception evaluation index for the vehicle perception system according to the perception result sequence includes:

[0024] According to the perception result sequence, determine the second state feature sequence corresponding to each perception object; wherein, the second state feature sequence includes the second state features of the perception object corresponding to different timestamps;

[0025] Based on a preset second stability detection algorithm, perform stability detection on the second state feature sequence of each perception object respectively to obtain the second stability index.

[0026] Optionally,

[0027] The second stability detection algorithm includes one or more of an outlier detection algorithm, a differential step point detection algorithm, an ADF stability detection algorithm, and an ACF stability detection algorithm.

[0028] Optionally, the vehicle perception system is an offline perception system; obtaining the perception result sequence generated by the vehicle perception system based on the sensor data sequence includes:

[0029] Input the sensor data sequence into the offline perception system to obtain the perception result sequence output by the offline perception system; wherein, the computer program running the offline perception system is the same as the online perception system of the vehicle.

[0030] According to a second aspect of the embodiments of the present disclosure, a vehicle control method is provided, and the method includes:

[0031] Obtain the target perception result generated by the target perception system based on the sensor data;

[0032] Control the vehicle to travel according to the target perception result;

[0033] Wherein, the target perception system is a vehicle perception system whose perception evaluation index meets a preset target, the perception evaluation index includes a first stability index for characterizing the relative state temporal stability between different perception objects, the perception evaluation index is generated according to the perception result sequence generated by the target perception system based on the sensor data sequence, the sensor data sequence includes multiple frames of sensor data detected by vehicle sensors, different frames of sensor data correspond to different timestamps, the perception result sequence includes multiple frames of perception results corresponding to different timestamps respectively, and the perception results include the state characteristics of multiple perception objects.

[0034] According to a third aspect of the embodiments of the present disclosure, there is provided a computer-readable storage medium having a computer program stored thereon, and the computer program, when executed by a processor, implements the method according to any one of the first aspect and / or the second aspect.

[0035] Based on the vehicle perception system evaluation method provided by the embodiments of the present disclosure, a perception evaluation index for the perception system can be generated according to the perception result sequence generated by the vehicle perception system based on the sensor data sequence, and the perception evaluation index includes the temporal stability of the relative state characteristics between different perception objects. There is no need for manual verification or construction of a test data set, which improves the efficiency and reliability of the perception system evaluation, and more effective feedback can be obtained based on less data input.

[0036] Other features and advantages of the present disclosure will become clear through the following detailed description of the exemplary embodiments of the present disclosure with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0038] Figure 1 is a schematic diagram of an intelligent networked system to which the method provided by the embodiments of the present disclosure can be applied.

[0039] Figure 2 is according to Figure 1 a schematic diagram of a vehicle provided according to the illustrated embodiment.

[0040] Figure 3 is a schematic flowchart of a vehicle perception system evaluation method provided by the embodiments of the present disclosure.

[0041] Figure 4 is a schematic flowchart of a vehicle perception system evaluation method provided by the embodiments of the present disclosure.

[0042] Figure 5 is a schematic flowchart of a vehicle control method provided by the embodiments of the present disclosure.

[0043] Figure 6 is a schematic structural diagram of an electronic device provided by the embodiments of the present disclosure. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0044] Now, various exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. It should be noted that: Unless otherwise specifically stated, the relative arrangements of components and steps, numerical expressions, and numerical values set forth in these embodiments do not limit the scope of the present disclosure.

[0045] The following description of at least one exemplary embodiment is merely illustrative and is in no way a limitation on the present disclosure, its application, or use.

[0046] Technologies, methods, and devices known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, the above technologies, methods, and devices should be considered as part of the specification.

[0047] In all the examples shown and discussed herein, any specific values should be construed as merely exemplary and not as a limitation. Thus, other examples of the exemplary embodiments may have different values.

[0048] It should be noted that like reference numerals and letters denote like items in the following figures, and thus, once an item is defined in one figure, further discussion thereof is not required in subsequent figures.

[0049] It should be noted that the actions of collecting, storing, using, processing, transmitting, providing, disclosing, deleting, etc. of data in the present disclosure are all carried out on the premise of complying with the relevant regulations and policies on data protection in the country or region where it is located and with the full authorization of the corresponding data owners.

[0050] First, the application scenario of the embodiments of the present disclosure will be described.

[0051] Figure 1 It is a schematic diagram of an intelligent connected system 100 to which the method provided by the embodiments of the present disclosure can be applied. As Figure 1 shown, the intelligent connected system 100 may include: a vehicle 101, a server 102, and a user terminal 103.

[0052] In some examples, the vehicle 101 may be a vehicle with an autonomous driving function. Among them, autonomous driving is also known as driverless or intelligent driving. A vehicle with an autonomous driving function can perform driving tasks such as environmental perception, decision-making and planning, and control execution. The levels of autonomous driving can refer to the automotive intelligence grading standard formulated by the Society of Automotive Engineers (SAE). For example, the L0 level is manual driving, L1 is assisted driving, L2 is partial autonomous driving, L3 is conditional autonomous driving, L4 is highly autonomous driving, and L5 is fully autonomous driving. The above classification method for the levels of autonomous driving is only for example, and the embodiments of the present disclosure do not limit the classification criteria and levels of autonomous driving.

[0053] In some examples, the server 102 can be a single server or a distributed server cluster composed of multiple servers, and its deployment method can include a local server or a cloud server. The server 102 can communicate with the vehicle 101 and / or the user terminal 103 based on a communication network, and provide various services for the vehicle 101 and / or the user terminal 103. For example, the server can receive the perception data sent by the vehicle and provide services such as high-precision maps, data analysis, and decision-making planning for the vehicle. Also, for example, the server can receive query instructions or control instructions sent by the user terminal and provide corresponding services for the user.

[0054] In some examples, the user terminal 103 can be any form of electronic device that provides services for users, such as a personal computer, a laptop, a smart tablet, a smart phone, a smart wearable device, etc. The user can interact with the vehicle or the server through the human-machine interaction terminal configured on the vehicle 101, or can also interact with the vehicle or the server through the user terminal 103. For example, query the status and / or parameters of the vehicle through the user terminal, or control the vehicle to perform set tasks and / or modify configuration parameters, etc.; among them, the user terminal runs an application program based on the intelligent networked system to realize the interaction with the vehicle or the server. The application program can be a local application, a web application or a small program, etc., which is not limited here.

[0055] In some examples, the above application program running on the user terminal can provide authentication or authorization services for users. Users who have successfully authenticated and been granted corresponding permissions can query and / or control the vehicle within the granted permissions.

[0056] The vehicle 101, the server 102, and the user terminal 103 can communicate through the communication link provided by the communication network 104. The communication network 104 can include one or more networks of any type. For example, the communication network 104 can include the Internet, a local area network (LAN), a wide area network (WAN), a virtual private network (VPN), a public switched telephone network (PSTN), a satellite communication network, Wi-Fi, 2G, 3G, 4G, 5G, 6G, NB-IoT, eMTC, infrared, Bluetooth, NFC, etc. networks that provide communication, or a combination of the above multiple networks. The communication networks between the vehicle 101 and the server 102, between the user terminal 103 and the server 102, and between the user terminal 103 and the vehicle 101 can be the same or different.

[0057] It should be noted that Figure 1The structure of the intelligent networked system 100 shown is only schematic. The intelligent networked system in the embodiments of the present disclosure is not limited to the above structure and may include more or fewer devices as needed, or the devices may be combined or split. For example, the intelligent networked system may also not include a user terminal and / or a server; for another example, the user terminal and the server may be combined and deployed.

[0058] Figure 2 is provided according to Figure 1 the schematic diagram of a vehicle 101 shown in the embodiment. As Figure 2 shown, the vehicle 101 may include a sensing component 1011, a computing platform 1012, an execution component 1013, etc. Among them, the sensing component 1011, the computing platform 1012, and the execution component 1013 may be connected by a bus or other means.

[0059] In some examples, the sensing component 1011 may be used to collect information about the vehicle itself or the outside. The sensing component 1011 may include at least one of a vision sensing unit, a radar, a positioning and navigation unit, an inertial measurement unit (IMU), or other sensing units. Among them, the vision sensor unit may include one or more cameras, the radar may include at least one of a lidar, a millimeter-wave radar, an ultrasonic radar, or other radars, and the positioning and navigation unit may include at least one of a GPS system, a Beidou system, or other global positioning systems.

[0060] In some examples, the computing platform 1012 may include a device with computing capabilities for processing the sensed information collected by the sensing component 1011 to obtain control information and sending corresponding control instructions to the execution component 1013, so that the execution component 1013 performs corresponding actions, thereby realizing the control of the vehicle 101. Exemplarily, the computing platform 1012 may perform actions such as simultaneous localization and mapping (SLAM), path planning, and behavior decision-making on the vehicle, thereby realizing the autonomous control of the vehicle. The computing platform 1012 may include at least one processor and at least one memory. Each processor may execute the instructions stored in the memory alone or jointly to implement the method provided by the embodiments of the present disclosure. The processors in the embodiments of the present disclosure may include at least one of a central processing unit (CPU), a graphic process unit (GPU), a neural-network processing unit (NPU), a tensor processing unit (TPU), a data processing unit (DPU), a digital signal processor (DSP), a field programmable gate array (FPGA), a system on chip (SOC), an application specific integrated circuit (ASIC), a microcontroller unit (MCU), or other processors. The memory may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, a magnetic disk, or an optical disk. In addition to storing instructions, the memory may also store data, such as high-precision maps, path information, the position, direction, speed, etc. of the vehicle. The data stored in the memory may be acquired and used by the processor.

[0061] In some examples, the computing platform of the vehicle may independently execute computing tasks or communicate with the server to complete computing tasks. For example, the computing platform of the vehicle may cooperate with the server to complete corresponding computing tasks.

[0062] The computing platform 1012 can be set in the vehicle 101, and part or all of the computing platform 1012 can also be set in the server corresponding to the vehicle. For example, functions with higher real-time requirements in the computing platform 1012 are set in the vehicle, and another part of functions with lower real-time requirements are set in the server corresponding to the vehicle.

[0063] In some examples, the execution component 1013 is used to perform corresponding actions based on the control of the computing platform 1012, so that the vehicle 101 completes a moving task. The execution component 1013 can include, for example, a power component, a braking component, a transmission component, a steering component, etc.

[0064] It should be noted that Figure 2 The structure of the vehicle 101 shown in is only schematic. The vehicle in the embodiments of the present disclosure is not limited to the above structure, and may also include more or fewer components as needed, or the devices may be combined or split. For example, the vehicle may not include the above computing platform. For another example, the vehicle may further include a communication component, an interface component, a multimedia component, an input component, an output component, etc.

[0065] The embodiments of the present disclosure can be applied to an autonomous driving scenario, especially a scenario for evaluating the perception system of an autonomous vehicle. The perception system in the embodiments of the present disclosure may include Figure 2 the perception components shown in, or may also include Figure 2 the perception components shown in and part or all of the computing platform. The perception result of the perception system is crucial for the control of the autonomous vehicle, and it is necessary to evaluate the perception result of the vehicle perception system to determine the performance of the vehicle perception system. However, in the related art, it is necessary to manually judge the accuracy of the perception result, or evaluate the perception result based on a manually set test set and corresponding ground truth data, resulting in low efficiency in evaluating the perception system.

[0066] Figure 3 is a schematic flowchart of a method for evaluating a vehicle perception system provided by an embodiment of the present disclosure. The method for evaluating the vehicle perception system can be executed by Figure 1 the vehicle and / or server shown in, or can also be executed by other electronic devices. As Figure 3 shown, the method for evaluating the vehicle perception system in this embodiment may include the following steps S310 to step S320.

[0067] Step S310, obtaining a sequence of perception results generated by the vehicle perception system based on a sequence of sensor data.

[0068] Among them, the sensor data sequence may include multiple frames of sensor data detected by vehicle sensors, and different frames of sensor data correspond to different timestamps. Exemplarily, the sensor data sequence may be multiple frames of sensor data collected by vehicle sensors at a set acquisition frequency within a continuous time period. For example, vehicle sensors continuously collect data at an acquisition frequency of 100HZ during driving, so as to obtain one frame of sensor data every 10 milliseconds. The timestamp corresponding to the sensor data may be the moment when the sensor data is collected. The timestamp may be an absolute time, such as the absolute time of 9:51:23.300 seconds on February 11, 2025; the timestamp may also be a relative time, such as the relative time starting from the moment when the vehicle starts; the timestamp may also be a time serial number, such as a serial number starting from 0 and incrementing. In this way, the relative time can be obtained based on the product of the serial number and the acquisition period. Further, the vehicle sensors may include one or more types of sensors such as vision sensors, radar sensors, and positioning sensors. For example, the sensor data may include image data collected based on vision sensors and / or point cloud data collected based on radar sensors, and may also include data collected by other sensors.

[0069] The perception result sequence may include multiple frames of perception results corresponding to different timestamps respectively. For example, each frame of sensor data corresponds to one frame of perception result. The perception result sequence may include state features of multiple perception objects. The perception objects may include one or more of objects such as vehicles, pedestrians, cones, lane lines, traffic lights, etc. Different perception objects can be distinguished by object identifiers, and the object identifier can also be called a tracking identifier (Tracking ID), which is generated by the vehicle perception system. The state features of the perception object may include one or more of features such as type, position, size, speed, acceleration, color, etc.

[0070] In some examples, the perception objects in the perception result sequence may have different object types, and the object types may include dynamic types, static discrete types, and static continuous types. Among them, the perception objects of the dynamic type may be moving objects, such as obstacles that can move autonomously like pedestrians and vehicles; the perception objects of the static discrete type may be objects in a stationary state and with an area smaller than a preset threshold, such as traffic lights, zebra crossings, stop lines, parking spaces, etc.; the perception objects of the static continuous type may be objects in a stationary state and with an area larger than a preset threshold, such as road edges, lane lines, guiding lines, etc.

[0071] In some examples, the vehicle perception system may be an online perception system of the vehicle. For example, the state features of multiple perception objects output by the vehicle online perception system can be directly obtained, so as to obtain the perception result sequence. In this way, the performance of the vehicle online perception system can be evaluated.

[0072] In some other examples, the vehicle perception system can be an offline perception system. For example, it can be an offline perception system used to simulate the actual vehicle's online perception system, and the computer program running on this offline perception system is the same as that of the vehicle's online perception system. Sensor data sequences can be input into the offline perception system in chronological order. The offline perception system parses and identifies the sensor data to obtain the state characteristics of multiple perceived objects output by the offline perception system, thereby obtaining a perception result sequence.

[0073] Exemplarily, the sensor data sequence and the perception result sequence can be implemented in the form of a matrix. For example, read, load, and activate the sensor data required for an autonomous vehicle with an offline disk drop. Load the multiple sensor data of the autonomous driving into the system memory, such as wide-angle cameras, fisheye cameras, positioning sensors, ultrasonic radar sensors, etc. Align the time between each sensor and form a time-series matrix data as the sensor data sequence, and input the formed sensor data sequence into the offline perception system. The offline perception system can simulate an actual vehicle in a laboratory environment and then output the results of the vehicle-end perception module. Perform operations on the sensor data sequence using the perception algorithm, align and preprocess the operation results in terms of time to form a perception data matrix, and use this perception data matrix as the perception result sequence.

[0074] In this way, based on the offline perception system, the performance and potential problems of the perception system installed on the actual vehicle can be predicted in advance in the laboratory environment, which can effectively advance the performance inspection of the perception system to the laboratory stage and reduce the risk of actual vehicle testing to a certain extent.

[0075] Step S320: Generate a perception evaluation index for the vehicle perception system according to the perception result sequence.

[0076] In some examples, the perception evaluation index can include a first stability index used to characterize the temporal stability of the relative state characteristics between different perceived objects at adjacent timestamps. The first stability index can be obtained based on the change of the relative state characteristics between different perceived objects at adjacent timestamps. For example, the number of jumps of the relative state characteristics can be determined according to the change of the relative state characteristics at adjacent timestamps in the perception result sequence, and the first stability index can be obtained based on the number of jumps, duration, or time ratio of the relative state characteristics. Among them, the jump of the relative state characteristics can be that the change value of the relative state characteristics at adjacent timestamps is greater than a preset threshold, or the fluctuation within multiple consecutive timestamps is greater than a preset threshold.

[0077] In some examples, the relative state feature includes one or more of the following: a relative position feature for characterizing the relative position relationship of different perception objects; a coexistence state feature for characterizing whether different perception objects exist in the same frame of perception result.

[0078] In one implementation, the relative state feature may include a relative position feature for characterizing the relative position relationship of different perception objects. The relative position feature may include a relative distance (such as a relative lateral distance and / or a relative longitudinal distance). If the change amount of the relative distance between two perception objects at adjacent two timestamps is greater than a preset distance change threshold, it may be determined that the relative state feature has a jump.

[0079] Taking two different perception objects as a vehicle and a lane line respectively, the preset distance change threshold is set to 5 meters. If within two adjacent timestamps, the relative distance between the vehicle and the lane line changes from 1 meter to 7 meters, that is, the change value is 6 meters, which is greater than the threshold of 5 meters, it can be determined that the relative position relationship between the vehicle and the lane line has a jump, that is, the jump count is incremented by one.

[0080] It should be noted that different types of perception objects may correspond to different preset distance change thresholds. For example, the preset distance change threshold between two static type perception objects < the preset distance change threshold between a dynamic type perception object and a static continuous type perception object < the preset distance change threshold between a dynamic type perception object and a static discrete type perception object. If both perception objects are static objects, the preset distance change threshold can be set to 3 meters. If the two perception objects are a dynamic type perception object and a static continuous type perception object respectively, the preset distance change threshold can be set to 5 meters. If the two perception objects are a dynamic type perception object and a static discrete type perception object respectively, the preset distance change threshold can be set to 10 meters.

[0081] In another implementation, the relative state feature may further include a coexistence state feature for characterizing whether different perception objects exist in the same frame of perception result. Exemplarily, the coexistence state feature may be represented by a sequence composed of 0 and 1. For example, if different perception objects exist in the same frame of perception result, the coexistence state feature of this frame of perception result is 1; if different perception objects do not exist simultaneously in the same frame of perception result, the coexistence state feature of this frame of perception result is 0. If the coexistence state feature of two perception objects changes within adjacent timestamps, such as the number of times of changing from 0 to 1 or from 1 to 0, it can be determined that the relative state feature has a jump, that is, the jump count is incremented by one.

[0082] By adopting the above method, based on the data mining and analysis method, the perception evaluation index for the perception system can be generated according to the perception result sequence generated by the vehicle perception system based on the sensor data sequence, and the perception evaluation index includes the temporal stability of the relative state characteristics between different perception objects. There is no need for manual verification or construction of a test data set, which improves the efficiency and reliability of the perception system evaluation, and more effective feedback can be obtained based on less data input.

[0083] In some embodiments, when the above perception evaluation index includes the first stability index, the above step S320 may include the following steps S321 to step S322:

[0084] Step S321: Integrate the state characteristics of different perception objects according to the perception result sequence to obtain the first state characteristic sequence.

[0085] Wherein, the first state characteristic sequence may include multiple frames of relative state characteristics corresponding to different timestamps.

[0086] In some examples, the step S321 may include the following steps S3211 to step S3213:

[0087] Step S3211: Determine the second state characteristic sequence corresponding to each perception object according to the perception result sequence.

[0088] Wherein, the second state characteristic sequence may include the second state characteristics of the perception object corresponding to different timestamps.

[0089] Exemplarily, different perception objects can be distinguished according to the object identifier. For each perception object, according to the perception results of the perception object corresponding to each timestamp, the second state characteristic corresponding to the timestamp is determined, so as to obtain the second state characteristic sequence of the perception object.

[0090] The second state characteristic may include all state characteristics in the perception result, or may include some state characteristics in the perception result. Exemplarily, the perception results obtained based on the vehicle perception system include state characteristics such as the type, position, size, speed, acceleration, color, etc. of the perception object. The second state characteristic may include all or part of the above state characteristics. For example, it may only include the type and position, or may include the type, position, and speed, etc.

[0091] In some examples, both the perception result sequence and the second state feature sequence can be implemented in the form of a matrix. For example, the data with features in the perception result sequence can be parsed, such as obstacles that can move autonomously like pedestrians and vehicles; independent (discrete) and stationary signs like traffic lights, zebra crossings, stop lines, and parking spaces; continuous and prohibited (stationary) guiding lines like curbs, lane lines, and guiding lines. And data cleaning and preprocessing are performed on the feature data matrix in time series. According to the object type of the perception object, three matrices with time series features are formed as the second state feature sequence.

[0092] Step S3212: Determine the perception object group to be fused according to the object type of the perception object.

[0093] Among them, the object type can include a dynamic type, a static discrete type, and a static continuous type. The perception object group to be fused can include at least two perception objects.

[0094] In one implementation, perception objects of different object types can be used as the perception object group to be fused. For example, perception objects of the dynamic type and the static discrete type can be combined to obtain the perception object group to be fused. For example, a vehicle and a traffic light; or, perception objects of the dynamic type and the static continuous type can be combined to obtain the perception object group to be fused. For example, a vehicle and a lane line; or, perception objects of the static discrete type and the static continuous type can be combined to obtain the perception object group to be fused. For example, a traffic light and a lane line.

[0095] In another implementation, perception objects of the same object type can be used as the perception object group to be fused. For example, two dynamic objects, such as two vehicles, can be used as the perception object group to be fused.

[0096] In another implementation, regardless of the object type of the perception object, multiple perception objects can be combined in pairs to obtain the perception object group to be fused.

[0097] Step S3213: Fuse the second state features of different perception objects in the perception object group to be fused at the same timestamp to obtain the first state feature sequence.

[0098] Among them, both the above-mentioned first state feature sequence and the second state feature sequence can be implemented in the form of a matrix. It can include using the matrix obtained by feature fusion of the second state feature sequences of different perception objects as the first state feature sequence. The feature fusion method can include feature splicing, feature addition, feature multiplication, or weighted summation of the second state features at the same timestamp to obtain the first state feature corresponding to this timestamp, and obtaining the first state feature sequence based on the first state features of multiple timestamps.

[0099] Step S322: Perform stability detection on the first state feature sequence based on a preset first stability detection algorithm to obtain a first stability index.

[0100] In some examples, the first stability detection algorithm may include a Markov chain algorithm and / or a Markov random field algorithm.

[0101] The above-mentioned Markov Chain (MC) is a mathematical model for modeling random processes, and its core feature is memorylessness, that is, the current state only depends on the previous state. Markov chains can be used to model the state transitions of sensing objects, such as the movement trajectories of sensing objects. By calculating the state transition probability, the stability of the state feature sequence can be evaluated. In the Markov chain algorithm of this embodiment, the state space of the Markov chain can be defined according to the values of the relative state features of different sensing objects. For example, the state space can be all possible values of the relative state features. Taking the relative state feature as the relative distance as an example, it can be divided into states such as close distance, medium distance, and long distance. A transition probability matrix of the Markov chain can be constructed based on the first state feature sequence, and this transition probability matrix is used to describe the probability of transitioning from one state to another. By analyzing the transition probability matrix, the stability of the relative state can be evaluated. The first stability index can be determined based on the transition probability matrix. If the transition probabilities are concentrated on the diagonal (i.e., the probability of the state remaining unchanged is relatively high), the first stability index is relatively high; conversely, if the transition probability distribution is relatively uniform, the first stability index is relatively low.

[0102] A Markov Random Field (MRF) is an undirected graph model suitable for modeling complex data distributions with spatial or structural dependencies. MRF can be used to model the spatial relationships between sensing objects. For example, by defining potential functions, the interactions between different sensing objects can be quantified, and the stability of the state feature sequence can be evaluated through maximum a posteriori (MAP) inference. In this embodiment, the nodes of the undirected graph in the Markov random field can represent the relative states of different sensing objects, and the edges of the undirected graph represent the dependencies between the nodes. For example, node A and node B respectively represent the relative positions of two sensing objects, and the edge (A, B) represents the association between these two relative positions. The potential function of the Markov random field can be used to quantify the joint probability of the node state and the combination of adjacent node states. For example, a potential function is defined to represent the similarity or difference in the relative positions of two sensing objects. Through the potential function and the undirected graph structure, the joint probability distribution of the entire system can be calculated. The first stability index can be evaluated by analyzing the change in the joint probability distribution. If the change in the joint probability distribution is small, the first stability index is relatively high.

[0103] In this step S322, the stability of the first state feature sequence can be detected based on the first stability detection algorithm, and the abnormal state timestamps can be determined from the first state feature sequence; the first stability index can be determined according to the proportion of the abnormal state timestamps in the first state feature sequence. Among them, the abnormal state timestamp can be the timestamp when the first state feature jumps.

[0104] Exemplarily, based on the Markov chain algorithm and / or the Markov random field algorithm, the timestamp when the first state feature jumps or becomes unstable in the sequence can be determined, so as to obtain the abnormal state timestamp. The ratio of the number of abnormal state timestamps to the total number of timestamps in the first state feature sequence can be used as the first stability index, or the number of abnormal state timestamps in the first state feature sequence can be used as the first stability index.

[0105] In this way, without manual analysis and annotation, by performing data mining and analysis on the first state feature sequence based on the Markov chain algorithm and / or the Markov random field algorithm, the first stability index of the vehicle perception system can be obtained, improving the efficiency of evaluating the vehicle perception system.

[0106] In some examples, the perception result sequence corresponding to the abnormal state timestamp can be normalized, the dimension can be unified, and the weighted sum can be performed to give a comprehensive score for the perception system, realizing the evaluation of the perception system.

[0107] In some examples, based on the perception result sequence corresponding to the abnormal state timestamp, multimedia data can be generated and the first stability index and the multimedia data can be presented to the user.

[0108] Exemplarily, the data before and after the perception result sequence corresponding to the abnormal state timestamp can be played back to generate multimedia data including information such as images, videos, voices, texts, etc. The first stability index can be presented in the form of a chart, or the multimedia data can be presented in the form of a video.

[0109] In this way, it is convenient for the user to view the corresponding index and abnormal data, locate and analyze the problems, so as to facilitate the user to improve the perception algorithm of the perception system targeted.

[0110] In some embodiments of the present disclosure, the above-mentioned perception evaluation index may include one or more types of indexes. For example, the perception evaluation index may include a first stability index for characterizing the temporal stability of the relative state characteristics between different perception objects at adjacent timestamps; for another example, the perception evaluation index may include a second stability index for characterizing the temporal stability of the state characteristics of the same perception object at adjacent timestamps; for still another example, the perception evaluation index may include the above-mentioned first stability index and the second stability index.

[0111] In some embodiments, when the above-mentioned perception evaluation index includes the second stability index, the above-mentioned step S320 may include the following steps S325 to step S326:

[0112] Step S325: Determine the second state feature sequence corresponding to each perception object according to the perception result sequence.

[0113] Wherein, the second state feature sequence may include the second state features of the perception object corresponding to different timestamps. The specific implementation manner of this step S325 may refer to the description of step S3211 in the foregoing embodiments of the present disclosure, and will not be elaborated herein.

[0114] Step S326: Perform stability detection on the second state feature sequence of each perception object respectively based on a preset second stability detection algorithm to obtain the second stability index.

[0115] Exemplarily, the second stability detection algorithm includes one or more of an outlier detection algorithm, a differential step point detection algorithm, an ADF stability detection algorithm, and an ACF stability detection algorithm.

[0116] In some examples, an outlier detection algorithm may be performed on the second state feature sequence according to the mean and standard deviation of the second state feature sequence to obtain the abnormal state timestamps, and the second stability index may be determined based on the number and / or proportion of the abnormal state timestamps.

[0117] For example, the above outlier detection algorithm may include methods such as 3sigma and Z-score. Based on outlier detection methods such as 3sigma and Z-score, data that is extremely large or extremely small in the time series feature matrix is detected. Among them: The 3sigma method is an outlier detection method based on the normal distribution. Its core idea is that in the normal distribution, the probability that a data point falls within the range of the mean (μ) plus or minus 3 times the standard deviation (σ) is approximately 99.7%. Therefore, data points outside this range are considered outliers, and the outlier status timestamps are obtained. The Z-score method is a standardization method for measuring the distance between a data point and the mean. It can judge outliers by calculating the number of standard deviations between the data point and the mean. For example, an absolute value of Z-score greater than 3 can be used as the threshold for outliers.

[0118] In some examples, based on difference step point detection algorithms such as first-order difference and second-order difference, outlier step points in the second state feature sequence can be detected.

[0119] Among them, the first-order difference can capture the change trend of the second state feature sequence by calculating the difference value of the second state feature between adjacent timestamps. In a stable situation, the difference value should be relatively stable, while an outlier will cause a significant change in the difference value. The second-order difference can perform a difference calculation on the basis of the first-order difference result, and can further analyze the acceleration change of the data, and can capture step points more sensitively.

[0120] Exemplarily, if the difference value of the second state feature between adjacent timestamps exceeds the set difference threshold range, then this timestamp can be used as an outlier status timestamp, and the second stability index can be determined based on the number and / or proportion of outlier status timestamps. The difference threshold range can be set according to the mean and standard deviation of all difference values. For example, it can be within the range of the mean ± 3 times the standard deviation.

[0121] In this way, through the difference step point detection algorithm for time series detection of the state features of the sensing object, state mutations can be effectively identified, such as abnormal events such as jumps in position, sudden acceleration or deceleration, and the corresponding outlier status timestamps.

[0122] In some examples, based on algorithms such as ADF (Augmented Dickey-Fuller Test) and ACF (Autocorrelation Function), it can be detected whether the data in the second state feature sequence is stable, and the outlier state features and outlier status timestamps among them can be determined.

[0123] The ADF stability detection algorithm can determine stationarity by checking for the presence of a unit root in the autoregressive model of a time series. A unit root refers to the case where the autoregressive coefficient is 1. If a unit root exists, the sequence is a non-stationary sequence. The ADF test introduces lag terms based on the classical Dickey-Fuller test to eliminate the autocorrelation of the sequence, making it more applicable to actual data and the detection results more accurate. The specific method of the ADF stability detection algorithm can include: estimating the model parameters by the least squares method, calculating the ADF statistic of the second state feature sequence, comparing the ADF statistic with the critical value. If the ADF statistic is less than the critical value, the sequence is considered stationary; if the ADF statistic is greater than or equal to the critical value, the sequence is considered non-stationary. Among them, the critical value can be obtained by simulation calculation and is determined according to the sample size and significance level (such as 1%, 5%, 10%). For example, it can be obtained through a large number of Monte Carlo simulations.

[0124] The ACF stability detection algorithm can analyze the correlation between data at each time point in a time series. The autocorrelation function of a stationary sequence usually decays rapidly as the lag order increases, while the autocorrelation function of a non-stationary sequence may show slow decay or periodic fluctuations. Exemplarily, by analyzing the ACF graph of the second state feature sequence, the stationarity of the second state feature sequence can be intuitively judged: if the ACF graph decays rapidly to near zero at a small lag order, the second state feature sequence is stationary; if the ACF graph shows obvious periodicity or slow decay, the second state feature sequence is non-stationary.

[0125] In this way, based on the preset second stability detection algorithm, the stability of the second state feature sequence of each sensing object is detected separately to obtain the second stability index of each sensing object.

[0126] In some embodiments, sensing objects of different object types may have different second stability indices.

[0127] Exemplarily, for a dynamic type of sensing object, its second stability index may include at least one of the following: the number / ratio of subtype jumps, the number / ratio of longitudinal position jumps greater than 5 meters, the number / ratio of lateral position jumps greater than 2 meters, the number / ratio of longitudinal speed jumps greater than 10 meters per second, the number / ratio of lateral speed jumps of 10 meters per second, the number / ratio of angle jumps greater than 90 degrees.

[0128] For a static discrete type of sensing object, its second stability index may include at least one of the following: the number / ratio of subtype jumps, the number / ratio of longitudinal position jumps greater than 10 meters, the number / ratio of lateral position jumps greater than 5 meters, the number / ratio of height jumps greater than 2 meters.

[0129] For a static continuous type of perceived object, taking a lane line as an example, its second stability index may include: the number / ratio of times that the lane line curvature radius is less than a preset threshold.

[0130] In this way, different metrics can be used for stability evaluation of perceived objects of different object types, so as to more precisely evaluate the accuracy of the perception results of the vehicle perception system.

[0131] Figure 4 It is a schematic flowchart of a method for evaluating a vehicle perception system provided by an embodiment of the present disclosure. This method for evaluating a vehicle perception system can be Figure 1 executed by the vehicle and / or server shown, or can be executed by other electronic devices. As Figure 4 shown, the method for evaluating a vehicle perception system in this embodiment may include the following steps S410 to step S450.

[0132] Step S410, obtain a sensor data sequence of the vehicle and input it into the vehicle perception system.

[0133] In some examples, this step S410 can be implemented by a data loading module. Exemplarily, the data loading module can establish communication with the vehicle offline disk data, and is used to read, load, and activate the sensor data required for an offline disk automatic driving vehicle. This module reads and loads various sensor data of the automatic driving into the system memory, such as a wide-angle camera, a fish-eye camera, a positioning sensor, an ultrasonic radar sensor, etc. This module performs time alignment between various sensors and forms a time-series matrix data to obtain a sensor data sequence of the vehicle, and is responsible for passing the sensor data sequence to the downstream offline perception system.

[0134] Step S420, obtain a perception result sequence generated by the vehicle perception system based on the sensor data sequence.

[0135] In some examples, the vehicle perception system can be an offline perception system, and this offline perception system can simulate an actual vehicle in a laboratory environment and then output the results of the vehicle-end perception module. Perform arithmetic operations on the perception algorithm for the data passed by the upstream data loading module, perform time alignment and preprocessing on the arithmetic results to form a perception data matrix, obtain a perception result sequence, and pass the perception result sequence to the downstream data analysis and data mining module.

[0136] In some examples, the specific implementation manner of this step S420 can refer to the description of Figure 3 step S310 shown, and details are not described here again.

[0137] Step S430, determine a second state feature sequence corresponding to each perceived object according to the perception result sequence.

[0138] In some examples, step S430 can be executed by a data parsing and processing module, which can receive the sequence of perception results output by the upstream offline perception system module and perform real-time processing on the perception results in the sequence of perception results. Exemplarily, data with features in the sequence of perception results can be parsed, such as obstacles that can move autonomously, such as pedestrians and vehicles; independent (discrete) and stationary signs such as traffic lights, zebra crossings, stop lines, and parking spaces; continuous and prohibited (stationary) guiding lines such as curbs, lane lines, and guiding lines. And data cleaning and preprocessing can be performed on the feature data in terms of time series to form multiple second state feature sequences with time series features and pass them to the downstream data analysis and mining module.

[0139] Step S440, obtaining abnormal feature data according to the second state feature sequences corresponding to multiple perception objects.

[0140] In some examples, step S430 can be executed by a data analysis and data mining module, which is used to process the second state feature sequences passed in by the upstream data parsing and processing module and perform analysis and mining on the feature matrix data. Data analysis is performed using methods such as outlier detection methods, time series feature matrix data stability detection methods, consistency detection methods between time series feature matrices, rationality analysis methods between time series feature matrices, and other analysis methods for the requirements of time series feature matrix data of other autonomous driving perception modules. The results obtained from data analysis are subjected to data mining methods such as clustering and classification to obtain abnormal feature data that does not meet the expectations of the autonomous driving perception module. These abnormal feature data are attached with the state features in the data parsing and processing module, and the abnormal feature data is passed to the downstream perception system evaluation and problem output module. The abnormal feature data can include abnormal state features and abnormal state timestamps.

[0141] In some examples, outlier detection methods such as 3sigma and Z-score can be used to detect data that is abnormally large or small in the second state feature sequence. Step point detection algorithms such as first-order difference and second-order difference are used to detect abnormal step points in the second state feature sequence. ADF, ACF, etc. are used to detect whether the data in the second state feature sequence is stable. The second state feature sequences are combined in pairs, and the reliability analysis of the data state is performed using methods such as Markov chain and Markov random field. Clustering methods such as DBSCAN (Density-Based Spatial Clustering of Applications with Noise) and KNN (K-Nearest Neighbors) can be used to cluster the data analysis results, and the clustering results are abnormal feature data.

[0142] Step S450: Determine the perception evaluation index according to the abnormal feature data.

[0143] In some embodiments, this step S450 can be executed by the perception system evaluation and problem output module. The perception system evaluation and problem output module can process the abnormal feature data output by the upstream, and after normalizing, unifying the dimensions, and weighted summing these abnormal feature data, obtain the comprehensive score of the vehicle perception system to realize the evaluation of the vehicle perception system.

[0144] In the following example, the Pythagorean fuzzy set method is used to weight the abnormal data matrix, and the TOPSIS method is used as the scoring calculation method for comprehensive evaluation analysis. Among them, the TOPSIS method (Technique for Order Preference by Similarity to Ideal Solution) is a multi-attribute decision analysis method that can be applied to comprehensive evaluation and ranking problems. It can evaluate the relative advantages and disadvantages of each object by calculating the distances between each evaluation object and the ideal solution (optimal solution) and the negative ideal solution (worst solution).

[0145] Furthermore, the above abnormal feature data can be classified to obtain the perception algorithm corresponding to each abnormal feature data, and the corresponding graphs, tables, and videos can be output, which can be used as abnormal data examples. In this way, a large number of valuable cases can be output for the autonomous driving system to achieve rapid iterative updates, improve the ability of the autonomous driving perception system to handle actual problems, and accelerate the evolution of the autonomous driving perception system's capabilities.

[0146] By adopting the above method, it is possible to use a mathematical model to replace manual rules, use the principle that the data output by the autonomous driving perception system should conform to the laws of the physical world, combine the characteristics of the data itself as supervision, and use data analysis and data mining methods to realize the performance evaluation of the autonomous driving perception system and potential problem analysis. From the perspectives of the stability, smoothness, consistency, rationality, self-consistency of the data output by the perception system and the downstream data requirements, etc., analyze and give the abnormal data output of the perception subsystem. It can be not limited by the construction of the data set and the accumulation of problem data, and can find problems such as instability, non-conformity with expectations, non-conformity with the rules of the physical world, and inconsistency based on the data output by the perception system itself, so as to achieve less data input and more effective feedback. And by using a mathematical model to replace the rules obtained by manual inspection and sorting, the situation of infinite increase in inspection rules with the increase of problems is effectively converged.

[0147] Figure 5 It is a schematic flowchart of a vehicle control method provided by an embodiment of the present disclosure. This vehicle control method can be executed by Figure 1 the vehicle and / or server shown. AsFigure 5 As shown in the figure, the vehicle control method of this embodiment may include:

[0148] Step S510: Obtain the target perception result generated by the target perception system based on the sensor data.

[0149] Step S520: Control the vehicle to travel according to the target perception result.

[0150] Among them, the target perception system is a vehicle perception system whose perception evaluation index meets the preset target. The perception evaluation index includes a first stability index used to characterize the temporal stability of the relative state between different perception objects. The perception evaluation index is generated based on the perception result sequence generated by the target perception system based on the sensor data sequence. The sensor data sequence includes multiple frames of sensor data detected by the vehicle sensors, and different frames of sensor data correspond to different timestamps. The perception result sequence includes multiple frames of perception results corresponding to different timestamps respectively, and the state characteristics of multiple perception objects are included in the perception results.

[0151] In some examples, the target perception system may be a vehicle perception system after an offline perception system that has been tested in the laboratory is deployed to the vehicle side. In this way, the stability and safety of the perception system can be evaluated before the perception system is actually installed on the vehicle, providing a reliable basis for the iterative update of the autonomous driving perception system.

[0152] It should be noted that the acquisition method of the perception evaluation index in this embodiment can refer to the description in the foregoing embodiments of the present disclosure, and will not be elaborated here.

[0153] Figure 6 It is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure. As Figure 6 shown, the electronic device 1000 may include a memory 1010 and a processor 1020. The memory 1010 may be used to store computer instructions, and the processor 1020 may be used to call the computer instructions from the memory 1010 to execute all or part of the steps of any method in the foregoing embodiments of the present disclosure. Among them, the processor may be one or more, and the one or more processors may execute instructions alone or jointly. The memory may also be one or more, and the one or more memories may store the above computer instructions alone or jointly. Optionally, the electronic device may be Figure 1 a server and / or a vehicle in

[0154] Embodiments of the present disclosure also provide a vehicle, which may include a memory and a processor. The memory may be used to store computer instructions, and the processor may be used to call the computer instructions from the memory to execute all or part of the steps of any of the methods in the foregoing embodiments of the present disclosure. Among them, the processor may be one or more, and the one or more processors may execute instructions alone or jointly. The memory may also be one or more, and the one or more memories may store the above computer instructions alone or jointly.

[0155] The vehicle in the foregoing embodiments of the present disclosure may be an electric vehicle, a hybrid vehicle, a fuel cell vehicle, or other types of vehicles. The vehicle may be an autonomous vehicle or a non-autonomous vehicle. Exemplarily, the vehicle provided in this embodiment may be Figure 1 or Figure 2 the vehicle shown.

[0156] Embodiments of the present disclosure also provide a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements any of the methods in the foregoing embodiments of the present disclosure. Optionally, the computer-readable storage medium may be a non-transitory storage medium, but is not limited thereto, and it may also be a transitory storage medium.

[0157] Embodiments of the present disclosure also provide a chip, which may include a processing unit, and the processing unit may be used to execute all or part of the steps of any of the methods in the foregoing embodiments of the present disclosure. The chip may be in the form of an application-specific integrated circuit (ASIC), a system-on-chip (SOC), a field-programmable gate array (FPGA), etc., and this embodiment does not limit this. Optionally, the chip may further include a storage unit, and the storage unit may be used to store computer instructions. The processing unit may be used to call the computer instructions from the storage unit to execute all or part of the steps of any of the methods in the foregoing embodiments of the present disclosure.

[0158] Embodiments of the present disclosure also provide a computer program product, which may include a computer program. When the computer program is executed by a processor, it may implement any of the methods in the foregoing embodiments of the present disclosure.

[0159] The present disclosure may be a system, a method, and / or a computer program product. The computer program product may include a computer-readable storage medium, on which computer-readable program instructions for causing a processor to implement any of the methods in the foregoing embodiments of the present disclosure are uploaded.

[0160] A computer-readable storage medium can be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium may be, for example—but not limited to—an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanically encoded device such as a punched card or raised structures in grooves having instructions stored thereon, and any suitable combination of the foregoing. The computer-readable storage medium as used herein is not construed as being a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., an optical pulse through an optical fiber cable), or an electrical signal transmitted through a wire.

[0161] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to respective computing / processing devices, or can be downloaded to an external computer or an external storage device through a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include a copper transmission cable, an optical fiber transmission, a wireless transmission, a router, a firewall, a switch, a gateway computer, and / or an edge server. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing / processing device.

[0162] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine - related instructions, microcode, firmware instructions, state - setting data, or source code or object code written in any combination of one or more programming languages, which may include object - oriented programming languages - such as Smalltalk, C++, etc., and conventional procedural programming languages - such as the "C" language or similar programming languages. The computer - readable program instructions may be executed entirely on the user's computer, partially on the user's computer, executed as a stand - alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network - including a local area network (LAN) or a wide area network (WAN) - or, alternatively, may be connected to an external computer (e.g., through the Internet using an Internet service provider). In some embodiments, by utilizing the state characteristics of the computer - readable program instructions to customize an electronic circuit, such as a programmable logic circuit, a field - programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit can execute the computer - readable program instructions to implement various aspects of the present disclosure.

[0163] Aspects of the present disclosure are described herein with reference to the flowchart and / or block diagram of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowchart and / or block diagram, and the combinations of blocks in the flowchart and / or block diagram, can be implemented by computer - readable program instructions.

[0164] These computer - readable program instructions can be provided to a processor of a general - purpose computer, a special - purpose computer, or other programmable data - processing apparatus to produce a machine such that the instructions, when executed by the processor of the computer or other programmable data - processing apparatus, create a means for implementing the functions / acts specified in one or more blocks of the flowchart and / or block diagram. These computer - readable program instructions can also be stored in a computer - readable storage medium, which causes a computer, a programmable data - processing apparatus, and / or other devices to operate in a particular manner, so that the computer - readable medium storing the instructions includes a manufacture that includes instructions for implementing various aspects of the functions / acts specified in one or more blocks of the flowchart and / or block diagram.

[0165] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices, causing a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other devices to produce a computer-implemented process, such that the instructions executed on the computer, other programmable data processing apparatus, or other devices implement the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0166] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two consecutive blocks may in fact be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block of the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified functions or actions, or by a combination of dedicated hardware and computer instructions. It should be noted that implementation by hardware, implementation by software, and implementation by a combination of software and hardware are all equivalent.

[0167] The embodiments of the present disclosure have been described above. The above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art in the field without departing from the scope and spirit of the described embodiments. The selection of the terms used herein is intended to best explain the principles of the embodiments, the practical application, or the technical improvement of the technology in the market, or to enable other ordinary skill in the art in the field to understand the embodiments disclosed herein. The scope of the present disclosure is defined by the appended claims.

Claims

1. A method for evaluating a vehicle perception system, characterized in that The method includes: Obtaining a sequence of perception results generated by a vehicle perception system based on a sequence of sensor data; wherein, the sequence of sensor data includes multiple frames of sensor data detected by vehicle sensors, different frames of sensor data correspond to different timestamps, the sequence of perception results includes multiple frames of perception results corresponding to different timestamps respectively, and the perception results include state features of multiple perception objects; Generating a perception evaluation index for the vehicle perception system according to the sequence of perception results; wherein, the perception evaluation index includes a first stability index for characterizing the temporal stability of the relative state features between different perception objects at adjacent timestamps.

2. The method according to claim 1, wherein The generating a perception evaluation index for the vehicle perception system according to the sequence of perception results includes: Fusing the state features of different perception objects according to the sequence of perception results to obtain a first sequence of state features; wherein, the first sequence of state features includes multiple frames of relative state features corresponding to different timestamps; Performing stability detection on the first sequence of state features based on a preset first stability detection algorithm to obtain the first stability index.

3. The method according to claim 2, wherein The fusing the state features of different perception objects according to the sequence of perception results to obtain a first sequence of state features includes: Determining a second sequence of state features corresponding to each perception object according to the sequence of perception results; wherein, the second sequence of state features includes second state features of the perception object corresponding to different timestamps; Determining a group of perception objects to be fused according to the object type of the perception object; wherein, the object type includes a dynamic type, a static discrete type, and a static continuous type; Fusing the second state features of different perception objects in the group of perception objects to be fused at the same timestamp to obtain the first sequence of state features.

4. The method according to claim 2, characterized in that, The first stability detection algorithm includes a Markov chain algorithm and / or a Markov random field algorithm; the performing stability detection on the first sequence of state features based on a preset first stability detection algorithm to obtain the first stability index includes: Performing stability detection on the first sequence of state features based on the first stability detection algorithm, and determining an abnormal state timestamp from the first sequence of state features; wherein, the abnormal state timestamp is a timestamp at which the relative state feature undergoes a jump; Determining the first stability index according to the proportion of the abnormal state timestamp in the first sequence of state features.

5. The method according to claim 4, wherein The method further includes: Generating multimedia data based on the sequence of perception results corresponding to the abnormal state timestamp; Displaying the first stability index and the multimedia data to the user.

6. The method according to claim 2, wherein The relative state feature includes one or more of the following: A relative position feature for characterizing the relative position relationship between different perception objects; A coexistence state feature for characterizing whether different perception objects exist in the same frame of perception result.

7. The method according to claim 1, wherein The perception evaluation index further includes a second stability index for characterizing the temporal stability of the state features of the same perception object at adjacent timestamps; the generating a perception evaluation index for the vehicle perception system according to the sequence of perception results includes: Determine a second state feature sequence corresponding to each sensed object according to the sequence of sensed results; wherein, the second state feature sequence includes second state features of the sensed object corresponding to different timestamps. Based on a preset second stability detection algorithm, perform stability detection on the second state feature sequence of each sensed object respectively to obtain the second stability index; wherein, the second stability detection algorithm includes one or more of an outlier detection algorithm, a differential step point detection algorithm, an ADF stability detection algorithm, and an ACF stability detection algorithm.

8. The method according to any one of claims 1 to 7, characterized in that, The vehicle sensing system is an offline sensing system; the obtaining of the sequence of sensed results generated by the vehicle sensing system based on the sensor data sequence includes: Input the sensor data sequence into the offline sensing system to obtain the sequence of sensed results output by the offline sensing system; wherein, the computer program running on the offline sensing system is the same as the online sensing system of the vehicle.

9. A vehicle control method, characterized in that, The method includes: Obtain a target sensed result generated by a target sensing system based on sensor data. Control the vehicle to travel according to the target sensed result. Wherein, the target sensing system is a vehicle sensing system whose sensing evaluation index meets a preset target, the sensing evaluation index includes a first stability index for characterizing the relative state time series stability between different sensed objects, the sensing evaluation index is generated according to the sequence of sensed results generated by the target sensing system based on the sensor data sequence, the sensor data sequence includes multiple frames of sensor data detected by vehicle sensors, different frames of sensor data correspond to different timestamps, the sequence of sensed results includes multiple frames of sensed results corresponding to different timestamps respectively, and the sensed results include state features of multiple sensed objects.

10. A computer-readable storage medium, characterized in that, A computer program is stored thereon, and when the computer program is executed by a processor, it implements: the method according to any one of claims 1 to 9.

Images