Monitoring method and device of intelligent driving system, medium and equipment
By monitoring the current environmental information and preconfigured data of the intelligent driving system in real time, detecting its sensor data and processing results, the problem of abnormalities in the intelligent driving system affecting driving safety is solved, and effective monitoring and safety improvement of system-level defects is achieved.
Patent Information
- Application Number
- CN202411315914.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-19
- Publication Date
- 2024-11-15
AI Technical Summary
When an abnormality occurs in the fields of autonomous driving and assisted driving, it is easy to affect the driving safety of movable devices, and it is difficult for the prior art to effectively monitor and discover system-level defects.
By determining the current environment information of the intelligent driving system, obtaining sensor data and result label data based on the preconfigured test data set, processing and comparison, and real-time monitoring of the system whether there are defects in the system, realizing system-level defect detection of the intelligent driving system.
Effectively monitor the intelligent driving system, promptly detect system-level defects, improve the driving safety of mobile devices, and ensure the functional safety of the intelligent driving system.
Smart Images

Figure CN118953392A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to intelligent driving technology, and in particular to a monitoring method, device, medium and equipment for an intelligent driving system. Background Art
[0002] In the field of intelligent driving such as autonomous driving and assisted driving, intelligent driving systems usually include functions such as data collection, data processing, data feature extraction and perception, trajectory prediction, planning and decision-making, etc., to achieve intelligent control of movable devices (such as vehicles, robots, drones, etc.). If an abnormality occurs in the intelligent driving system, it is easy to affect the driving safety of movable devices. Summary of the invention
[0003] The embodiments of the present disclosure provide a monitoring method, device, medium and equipment for an intelligent driving system, which can realize overall monitoring of the intelligent driving system, timely discover defects of the intelligent driving system, and improve the driving safety of movable equipment.
[0004] A first aspect of an embodiment of the present disclosure provides a monitoring method for an intelligent driving system, including: determining current environmental information corresponding to the intelligent driving system; determining, based on a preconfigured test data set, first test sensor data corresponding to the current environmental information and first result label data corresponding to the first test sensor data; the test data set includes test sensor data under different environments and corresponding result label data; based on the intelligent driving system, processing the first test sensor data to obtain a first processing result corresponding to the first test sensor data; and determining a monitoring result of the intelligent driving system based on the first result label data and the first processing result.
[0005] According to a second aspect of an embodiment of the present disclosure, a monitoring device for an intelligent driving system is provided, comprising: a first processing module for determining current environmental information corresponding to the intelligent driving system; a second processing module for determining, based on a preconfigured test data set, first test sensor data corresponding to the current environmental information and first result label data corresponding to the first test sensor data; a third processing module for processing the first test sensor data based on the intelligent driving system to obtain a first processing result corresponding to the first test sensor data; and a fourth processing module for determining a monitoring result of the intelligent driving system based on the first result label data and the first processing result.
[0006] According to a third aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to execute the monitoring method of the intelligent driving system described in any of the above embodiments of the present disclosure.
[0007] According to a fourth aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing executable instructions of the processor; the processor is configured to read the executable instructions from the memory and execute the instructions to implement the monitoring method of the intelligent driving system described in any of the above embodiments of the present disclosure; or, the electronic device comprises the monitoring device of the intelligent driving system described in any of the above embodiments of the present disclosure.
[0008] A fifth aspect of the embodiments of the present disclosure provides a computer program product. When the instructions in the computer program product are executed by a processor, the monitoring method of the intelligent driving system provided by any of the above embodiments of the present disclosure is executed.
[0009] Based on the monitoring method, device, medium and equipment of the intelligent driving system provided by the above-mentioned embodiments of the present disclosure, when the intelligent driving system is working, the current environmental information corresponding to the intelligent driving system can be determined, and based on the preconfigured test data set, the first test sensor data corresponding to the current environmental information and the first result label data corresponding to the first test sensor data can be determined; based on the intelligent driving system, the first test sensor data is processed to obtain the first processing result corresponding to the first test sensor data; and then the monitoring result of the intelligent driving system is determined based on the first processing result and the first result label data. Since the first test sensor data that conforms to the environment corresponding to the current environmental information is determined based on the current environmental information corresponding to the intelligent driving system in real time, the first test sensor data is used to test the intelligent driving system, the first processing result is equivalent to the test output result of the intelligent driving system, and the first result label data is the expected output result corresponding to the first test sensor data. By comparing the first processing result and the first result label data, it can be effectively determined whether the intelligent driving system has defects, and the effective monitoring of the system-level defects of the intelligent driving system is realized. The overall system-level defects of the intelligent driving system can be discovered in time, and the functional safety of the intelligent driving system can be ensured, thereby helping to improve the driving safety of movable equipment. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] Figure 1 This is an exemplary application scenario of the monitoring method of the intelligent driving system provided by the present disclosure;
[0011] Figure 2 is a flow chart of a monitoring method of an intelligent driving system provided by an exemplary embodiment of the present disclosure;
[0012] Figure 3 is a flow chart of a monitoring method of an intelligent driving system provided by another exemplary embodiment of the present disclosure;
[0013] Figure 4is a flowchart of a monitoring method of an intelligent driving system provided by yet another exemplary embodiment of the present disclosure;
[0014] Figure 5 is a flowchart of a monitoring method of an intelligent driving system provided by another exemplary embodiment of the present disclosure;
[0015] Figure 6 is a flowchart of a monitoring method of an intelligent driving system provided by yet another exemplary embodiment of the present disclosure;
[0016] Figure 7 is a flowchart of a monitoring method of an intelligent driving system provided by another exemplary embodiment of the present disclosure;
[0017] Figure 8 is a flowchart of a monitoring method of an intelligent driving system provided by yet another exemplary embodiment of the present disclosure;
[0018] Fig. 9 is a flowchart of a monitoring method of an intelligent driving system provided by another exemplary embodiment of the present disclosure;
[0019] Fig.10 is a schematic diagram of the system architecture of an intelligent driving system provided by an exemplary embodiment of the present disclosure;
[0020] Fig.11 is a structural schematic diagram of a monitoring device for an intelligent driving system provided by an exemplary embodiment of the present disclosure;
[0021] Fig.12 is a structural schematic diagram of a monitoring device for an intelligent driving system provided by another exemplary embodiment of the present disclosure;
[0022] Fig.13 is a structural schematic diagram of a monitoring device for an intelligent driving system provided by yet another exemplary embodiment of the present disclosure;
[0023] Fig.14 is a structural schematic diagram of a monitoring device for an intelligent driving system provided by another exemplary embodiment of the present disclosure;
[0024] Fig.15 is a structural schematic diagram of a monitoring device for an intelligent driving system provided by yet another exemplary embodiment of the present disclosure;
[0025] Fig.16 It is a structural diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION
[0026] To explain the present disclosure, example embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. It should be understood that the present disclosure is not limited to the example embodiments.
[0027] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.
[0028] SUMMARY OF THE DISCLOSURE
[0029] In the process of realizing the present disclosure, the inventors found that in the field of intelligent driving such as autonomous driving and assisted driving, the intelligent driving system usually includes functions such as data collection, data processing, data feature extraction and perception, trajectory prediction, planning and decision-making, etc., to realize intelligent control of movable devices (such as vehicles, robots, drones, etc.). If the intelligent driving system is abnormal, it is easy to affect the driving safety of the movable device.
[0030] Exemplary Overview
[0031] Figure 1 This is an exemplary application scenario of the monitoring method of the intelligent driving system provided by the present disclosure. Figure 1As shown, the mobile device includes an intelligent driving system 11, and the intelligent driving system 11 is connected to a sensor group 12, and the sensor group 12 may include at least one type of sensor. The number of sensors of each type may be one or more. For example, the sensor group 12 may include sensors 121, 122, ..., and 12n, where n is a positive integer. The types of sensors may include, for example, cameras (Camera, or cameras), radars (Radar), laser radars (Lidar), millimeter wave radars, etc. That is, sensors 12i (i = 1, 2, ..., n) may be any type of sensor. The intelligent driving system 11 is also coupled to a monitoring device 13 (which may be referred to as monitoring device 13) of the intelligent driving system disclosed in the present invention, and the monitoring device 13 of the intelligent driving system is used to execute the monitoring method of the intelligent driving system disclosed in the present invention to realize the monitoring of the intelligent driving system 11. The intelligent driving system 11 may include one or more functions such as data acquisition, data processing, data feature extraction and perception, trajectory prediction, and planning and decision-making. Any function may be implemented by corresponding functional software (including at least one of algorithms, functional models, etc. for implementing the corresponding functions). On the mobile device side, the functional software corresponding to any function can call the hardware in the intelligent driving system that can support the calculation of the corresponding function, and realize the function through the collaboration of software and hardware. For example, the data feature extraction and perception function is realized by the functional software such as the algorithm and model of the feature extraction and perception function. In order to improve the execution efficiency of the functional software, the neural network processor is called to realize the model reasoning of the data feature extraction and perception function, so as to realize the data feature extraction and perception function in combination with software and hardware. The monitoring device 13 can be any processor, controller, chip and other devices on the mobile device that can execute the monitoring method of the intelligent driving system disclosed in the present invention. In practical applications, the monitoring device 13 can be realized by any processor, controller, chip and other devices already in the intelligent driving system 11, and the monitoring function of the present invention is realized by configuring the application program of the method of the embodiment of the present invention into the corresponding device. Alternatively, the monitoring device 13 of the intelligent driving system can be any processor, controller, chip and other devices independent of the intelligent driving system 11, as shown in the figure, and is not limited to the setting method in the figure. The mobile device with the intelligent driving system 11 can be a vehicle, a robot, a drone and other equipment.During the driving process of the mobile device or during the operation of the intelligent driving system 11, the monitoring method of the intelligent driving system disclosed in the present invention can be used to determine the current environmental information corresponding to the intelligent driving system 11, and based on the preconfigured test data set 14, determine the first test sensor data corresponding to the current environmental information and the first result label data corresponding to the first test sensor data; based on the intelligent driving system 11, process the first test sensor data to obtain the first processing result corresponding to the first test sensor data; and then determine the monitoring result of the intelligent driving system 11 based on the first processing result and the first result label data. Since the first test sensor data that conforms to the environment corresponding to the current environmental information is determined based on the current environmental information corresponding to the intelligent driving system 11 in real time, the first test sensor data is used to test the intelligent driving system 11, the first processing result is equivalent to the test output result of the intelligent driving system 11, and the first result label data is the expected output result corresponding to the first test sensor data. By comparing the first processing result and the first result label data, it can be effectively determined whether the intelligent driving system 11 has system-level defects, and the effective monitoring of the system-level defects of the intelligent driving system 11 is achieved. The overall system-level defects of the intelligent driving system 11 can be discovered in time, which helps to improve the driving safety of the mobile device.
[0032] In actual applications, the intelligent driving system 11 may correspond to an intelligent driving chip on a mobile device, which may specifically include one or more chips. For example, functions such as data collection, data processing, data feature extraction and perception, trajectory prediction, planning and decision-making may be implemented by different chips respectively, or all or part of the above functions may be integrated in the same chip for implementation, without specific limitation.
[0033] Exemplary Methods
[0034] Figure 2 is a flow chart of a monitoring method of an intelligent driving system provided by an exemplary embodiment of the present disclosure. This embodiment can be applied to electronic devices, such as a computing platform or chip on a mobile device, such as Figure 2 As shown, the method of the embodiment of the present disclosure may include the following steps:
[0035] Step 210: Determine current environment information corresponding to the intelligent driving system.
[0036] Among them, the current environment information is the environment information of the scene where the mobile device where the intelligent driving system is located is currently located. The current environment information may include current driving environment condition information, current road scene information, etc. The current driving environment condition information may include information such as brightness, color temperature, noise, and fog index. Alternatively, the current driving environment condition information may include weather (such as rainy days, foggy days), time period (night or daytime), ambient lighting, and other information. The current road scene information may include information such as meeting scenes, ramps merging into the main road, lane change scenes, road element information, and traffic participants. Meeting scenes may include, for example, lane keeping status, merging status, overtaking status, and the like. Traffic participants may include, for example, pedestrians, vehicles, cyclists, animals, and the like. Road element information may include, for example, traffic lights, traffic signs, lane lines, crosswalks, and other information.
[0037] In some optional embodiments, if the current moment (or current frame) is not the start moment after the intelligent driving system is started, the current environment information can be determined based on the historical environment information sensed by the intelligent driving system at the historical moment. If the current moment is the start moment after the intelligent driving system is started, since the intelligent driving system has not yet started the environment perception function, there is no historical environment information, and the preset environment information can be determined as the current environment information.
[0038] Step 220 : determining first test sensor data corresponding to the current environment information and first result label data corresponding to the first test sensor data based on the preconfigured test data set.
[0039] The test data set includes test sensor data in different environments (which may be referred to as test environments) and corresponding result label data.
[0040] In some optional embodiments, the test data set may be sensor data collected in advance for various different environments as test sensor data, and the expected output results of the test sensor data for each environment are annotated according to the specific functions of the intelligent driving system to obtain result label data corresponding to the test sensor data. That is, the result label data corresponding to the test sensor data is the expected result data of the test sensor data after being processed by the intelligent driving system. For example, the function of the intelligent driving system is to perceive based on the sensor data collected in real time, plan and make decisions based on the perception results, and obtain planning and decision results. Then the result label data includes planning and decision result label data corresponding to the test sensor data.
[0041] In some optional embodiments, the result label data corresponding to the test sensor data may also include result label data corresponding to the intermediate result of the intelligent driving system. For example, the result label data corresponding to the test sensor data may also include expected perception result label data.
[0042] In some optional embodiments, the first test sensor data corresponding to the current environment information may be target test sensor data determined from a preconfigured test data set and matching the current environment information. Accordingly, the result label data corresponding to the target test sensor data is used as the first result label data.
[0043] In some optional embodiments, the first test sensor data corresponding to the current environmental information may be test sensor data after certain adjustments are made to the target test sensor data in the test data set that matches the current environmental information. The adjustment may be an adjustment to the information about the driving environmental conditions in the target test sensor data. For example, brightness, color temperature, noise, fog index, etc. Because the test data set has limited coverage of environmental types, the target test sensor data that matches the current environmental information determined from the test data set may be close to the current environmental information, but there is still a certain gap. The gap is reduced by adjustment, so that the adjusted test sensor data is more in line with the current driving environmental conditions.
[0044] In some optional embodiments, the first test sensor data may include data collected by at least one sensor such as a camera (i.e., a radar), a laser radar (Lidar), a millimeter-wave radar, etc. The specific sensor type can be determined according to the type of sensor on which the intelligent driving system relies. For example, if the data based on which the intelligent driving system is actually working is an image collected by a camera, that is, the intelligent driving system processes the image to obtain processing results such as perception results, planning and decision results, etc., then the first test sensor data is test image data. If the data based on which the intelligent driving system is working is Radar data collected by Radar, the first test sensor data is test Radar data. If the data based on which the intelligent driving system is working includes sensor data (such as images and Radar data) collected by multiple sensors respectively, the first test sensor data includes test sensor data of multiple sensors.
[0045] Step 230: Based on the intelligent driving system, the first test sensor data is processed to obtain a first processing result corresponding to the first test sensor data.
[0046] The content of the first processing result is determined by the function of the intelligent driving system. For example, if the result of the intelligent driving system processing the sensor data is a planning decision result, then the first processing result is the test planning decision result corresponding to the first test sensor data. The first test sensor data can be transmitted to the intelligent driving system so that the intelligent driving system performs image signal processing, feature extraction and perception, planning and decision-making on the first test sensor data to obtain the first processing result.
[0047] In some optional embodiments, the first test sensor data can be processed during the idle time between two adjacent frames of normal operation of the intelligent driving system to obtain a first processing result. The sensor type and data size of the first test sensor data can be consistent with the sensor data input by the normal operation of the intelligent driving system to meet the input data conditions of the intelligent driving system, ensuring that the intelligent driving system can process the first test sensor data according to the processing flow of the sensor data input by normal operation. Alternatively, in the case where the input of the intelligent driving system can support data of different sizes, the data size of the first test sensor data can be smaller than the size of the real-time sensor data when the intelligent driving system is working normally. For example, the sensor data is an image, and the image size of the first test sensor data can be smaller than the size of the image captured by the preset camera on the movable device to reduce the amount of calculation during the test process.
[0048] In some optional embodiments, the first test sensor data can be transmitted to the intelligent driving system together with the sensor data collected in real time by the sensor on the movable device, so that the intelligent driving system can simultaneously process the first test sensor data and the sensor data collected in real time to obtain a first processing result.
[0049] Step 240: Determine a monitoring result of the intelligent driving system based on the first result label data and the first processing result.
[0050] Among them, the first result label data is the expected processing result corresponding to the first test sensor data, and the first processing result is the test processing result corresponding to the first test sensor data. Since the first processing result is the processing result of the first test sensor data by the intelligent driving system, whether the first processing result is normal represents whether the intelligent driving system as a whole has defects. Therefore, the monitoring result of the intelligent driving system can be determined based on the comparison result between the first result label data and the first processing result. The monitoring result may include the result of whether the intelligent driving system is normal, or the monitoring result may include the result of whether the intelligent driving system is abnormal, or the monitoring result may include the consistency between the first processing result and the first result label data, or the result of whether the first processing result is consistent with the first result label data.
[0051] In some optional embodiments, the intelligent driving system can be monitored in real time or regularly according to the above steps 210 to 240. That is, the intelligent driving system can be monitored at every moment (or every frame) of the processing of the intelligent driving system. The intelligent driving system can also be monitored according to a preset cycle. For example, the monitoring process of the above steps 210 to 240 is triggered once every m (m is a positive number) seconds. The specific monitoring frequency is not limited.
[0052] The monitoring method of the intelligent driving system provided in the present embodiment is based on the first test sensor data that is determined to conform to the environment corresponding to the current environmental information in real time corresponding to the intelligent driving system. The first test sensor data is used to test the intelligent driving system, and the first processing result is equivalent to the test output result of the intelligent driving system. The first result label data is the expected output result corresponding to the first test sensor data. By comparing the first processing result with the first result label data, it is possible to effectively determine whether the intelligent driving system has defects, thereby achieving effective monitoring of system-level defects of the intelligent driving system, and timely discovering the overall system-level defects of the intelligent driving system, thereby helping to improve the driving safety of movable equipment.
[0053] Figure 3 It is a flowchart of a monitoring method of an intelligent driving system provided by another exemplary embodiment of the present disclosure.
[0054] In some optional embodiments, in the above Figure 2 Based on the embodiment shown, Figure 3 As shown, step 210 of determining the current environment information corresponding to the intelligent driving system may include the following steps:
[0055] Step 2110, determining the historical environment information at the previous moment as the current environment information.
[0056] Among them, historical environmental information is environmental information obtained by sensing and processing sensor data collected by preset sensors at the previous moment. The previous moment is any moment before the current moment. For example, if the current moment is represented as moment t, the previous moment can be moment t-1 or moment t-2. Optionally, the previous moment can be the moment closest to the current moment among the historical moments with historical environmental information to ensure the timeliness of the current environmental information. For example, if there is historical environmental information sensed at moment t-1, the previous moment is moment t-1, if there is no historical environmental information at moment t-1, and there is historical environmental information at moment t-2, then the previous moment is moment t-2, and so on. Since the time interval between the previous moment and the current moment is usually very small, the change in environmental information can be ignored, so the historical environmental information at the previous moment can be used as the current environmental information.
[0057] In some optional embodiments, if the current moment is the starting moment after the intelligent driving system is started, there is often no historical environmental information at the previous moment. In this case, the preset environmental information can be determined as the current environmental information. Or the current environmental information can be determined based on other information related to the current environment that can be obtained in real time. For example, time information can be obtained from a mobile device, and whether it is daytime or nighttime can be determined based on the time information, and the current environmental information can be determined based on whether it is daytime or nighttime.
[0058] In the embodiment of the present disclosure, by determining the historical environmental information at the previous moment as the current environmental information, the real-time and accuracy of the current environmental information can be ensured because the environmental changes are small in a short period of time.
[0059] Figure 4 It is a flowchart of a monitoring method of an intelligent driving system provided by yet another exemplary embodiment of the present disclosure.
[0060] In some optional embodiments, based on any of the above embodiments, the current environment information includes current driving environment condition information and current road scene information.
[0061] Among them, the current driving environment condition information and the current road scene information can refer to the above content.
[0062] In some optional embodiments, based on any of the above embodiments, Figure 4 As shown, step 220 of determining the first test sensor data corresponding to the current environment information and the first result label data corresponding to the first test sensor data based on the preconfigured test data set may include:
[0063] Step 2210, based on the current driving environment condition information and the current road scene information, determine the target test sensor data that matches the current environment information and the result label data corresponding to the target test sensor data from the test data set.
[0064] Among them, an environmental matching rule can be set, and based on the environmental matching rule, the current driving environment condition information and the current road scene information are matched with the test data set, and the target environment information closest to the current driving environment condition information and the current road scene information is determined from the test data set, and the test sensor data corresponding to the target environment information is used as the target test sensor data matching the current environment information. The environmental matching rule may include a driving environment condition matching rule and a road scene matching rule. The driving environment condition matching rule is used to match the current driving environment condition information with the test driving environment conditions of each test environment in the test data set. The road scene matching rule is used to match the current road scene information with the test road scene of each test environment in the test data set. Combined with the matching results of the current driving environment condition information and the current road scene information, the target test sensor data and the corresponding result label data that match the current environment information are determined. The driving environment condition matching rule can be set according to the condition type included in the driving environment condition. The condition type includes, for example, brightness, color temperature, dense fog index, noise, etc. The road scene matching rules can be set according to the scene content included in the road scene (such as the scene of meeting a car, the scene of the ramp merging into the main road, the scene of changing lanes, road elements, traffic participants, etc.). For example, the current driving environment condition information includes brightness value, color temperature value, dense fog index value, etc., and a group of test data with high similarity (such as greater than the similarity threshold) with the brightness value, color temperature value, and dense fog index value of the current driving environment condition information is selected from the test data set, and then the test sensor data with the highest similarity to the current road scene information and the corresponding result label data are determined from the group of test data according to the current road scene information. Alternatively, a group of test data with high similarity is first determined from the test data set according to the current road scene information, and then the test sensor data with the highest similarity and the corresponding result label data are determined from the group of test data according to the current driving environment condition information. Alternatively, the similarity between the current driving environment information and the current road scene information and each test environment in the test data set can be comprehensively determined, and the test sensor data and the corresponding result label data corresponding to the test environment with the highest similarity are used as the target test sensor data matching the current environment information and the result label data corresponding to the target test sensor data. The similarity between each test environment and the current environment information can be determined by calculating the Euclidean distance, cosine similarity, etc. between the test environment and the current environment information.
[0065] Step 2220: Based on the current driving environment condition information, adjust the target test sensor data to obtain first test sensor data.
[0066] Among them, the current driving environment condition information includes brightness, color temperature, noise, fog index, etc. Adjusting the target test sensor data based on the current driving environment condition information refers to adjusting the driving environment conditions of the target test sensor data to be the same or similar to the current driving environment condition information, so that the adjusted test sensor data is more consistent with the current driving environment condition information. For example, if the target test sensor data is an image, then the image brightness, color temperature, noise, and fog index of the target test sensor data are adjusted based on the current driving environment condition information, so that the adjusted image is more consistent with the image collected under the current driving environment conditions. However, it does not affect the content related to road scene information (such as meeting scenes, lane lines, traffic participants, etc.) in the target test sensor data.
[0067] Step 2230: Use the result label data corresponding to the target test sensor data as the first result label data.
[0068] Among them, since the adjustment of the target test sensor data does not affect the road scene related content, it only adjusts the driving environment conditions involved in the target test sensor data to be consistent or similar to the current driving environment conditions information. For example, the target test sensor data is an image, and one or more conditions of the image brightness, color temperature, noise, fog index, etc. are adjusted, and the road scene information such as lane lines and traffic participants included in the image is not adjusted. The result data is the processing result of the intelligent driving system for the road scene information, such as the planning and decision results based on the lane lines and traffic participants on the road. Therefore, the expected result data will not change. For example, there is a pedestrian at position P in the image, and there is still a pedestrian at position P in the image after the driving environment conditions are adjusted. Therefore, the result label data corresponding to the target test sensor data can be used as the first result label data corresponding to the first test sensor data.
[0069] In the disclosed embodiment, after determining the target test sensor data and the corresponding result label data matching the current environmental information from the test data set, the target test sensor data is adjusted based on the current driving environmental condition information, so that the adjusted first test sensor data is more in line with the current driving environmental condition, thereby realizing the mutable global built-in self-test (MG-bist) of the intelligent driving system, that is, the global built-in self-test that changes according to the real-time environmental information, ensuring the effective monitoring of the intelligent driving system, timely discovering the defects of the intelligent driving system, and further improving the safety of the intelligent driving system.
[0070] Figure 5 It is a flowchart of a monitoring method of an intelligent driving system provided by another exemplary embodiment of the present disclosure.
[0071] In some optional embodiments, based on any of the above embodiments, Figure 5 As shown, the method of the embodiment of the present disclosure may also include:
[0072] Step 310: Acquire current frame sensor data collected by a preset sensor.
[0073] The preset sensor is a sensor on the mobile device connected to the intelligent driving system. The preset sensor may include at least one of a camera, a radar, a laser radar, a millimeter wave radar, and the like. The current frame sensor data is the data collected by the preset sensor at the current moment. For example, if the preset sensor is a camera, the current frame sensor data is the image collected at the current moment.
[0074] Step 230 of processing the first test sensor data based on the intelligent driving system to obtain a first processing result corresponding to the first test sensor data includes:
[0075] Step 2310: Based on the intelligent driving system, the current frame sensor data and the first test sensor data are processed to obtain a first processing result corresponding to the first test sensor data and a second processing result corresponding to the current frame sensor data.
[0076] Among them, the processing of the current frame sensor data and the first test sensor data may include one or more of sensor signal processing, feature extraction and perception processing, positioning and mapping processing, driving trajectory prediction, planning and decision-making, etc. The specific processing process is determined according to the specific functions of the intelligent driving system.
[0077] In some optional embodiments, the current frame sensor data and the first test sensor data may be processed separately based on the intelligent driving system to obtain a first processing result and a second processing result. Alternatively, the current frame sensor data and the first test sensor data may be fused to obtain fused sensor data, and the fused sensor data may be processed based on the intelligent driving system, that is, the current frame sensor data and the first test sensor data may be processed simultaneously to obtain the first processing result and the second processing result synchronously. The fusion method of the current frame sensor data and the first test sensor data may be, for example, splicing or other methods.
[0078] In the disclosed embodiment, the intelligent driving system processes the current frame sensor data and the first test sensor data collected in real time, so as to complete the test of the intelligent driving system when the intelligent driving system is working normally, and the overall systemic defects of the intelligent driving system can be discovered in time.
[0079] Figure 6 It is a flowchart of a monitoring method of an intelligent driving system provided by yet another exemplary embodiment of the present disclosure.
[0080] In some optional embodiments, such as Figure 6 As shown, step 2310 of processing the current frame sensor data and the first test sensor data based on the intelligent driving system to obtain a first processing result corresponding to the first test sensor data and a second processing result corresponding to the current frame sensor data may include:
[0081] Step 23110: fuse the current frame sensor data with the first test sensor data to obtain fused sensor data.
[0082] The fusion processing may be performed in a fusion manner of splicing. For example, if the current frame sensor data and the first test sensor data are images, the two images are spliced into a larger image as the fused sensor data.
[0083] In some optional embodiments, the current frame sensor data and the first test sensor data may be spliced in a preset splicing order to obtain fused sensor data. The preset splicing order may be, for example, the current frame sensor data in front and the first test sensor data in the back, so that the first processing result and the second processing result can be determined from the processing result of the intelligent driving system according to the positional relationship between the current frame sensor data and the first test sensor data in the fused sensor data.
[0084] In some optional embodiments, the data size of the current frame sensor data may be the same as or different from the data size of the first test sensor data. Optionally, in order to reduce the amount of calculation, the data size of the first test sensor data may be smaller than the current frame sensor data. For example, when the sensor data is an image, the image size of the first test sensor data may be smaller than the image size of the current frame sensor data. It can be understood that in order to achieve the fusion of the current frame sensor data and the first test sensor data, the specific sizes of the current frame sensor data and the first test sensor data need to meet the fusion conditions. For example, if the fusion method is image stitching, the current frame sensor data may be a first image with a larger size, and the first test sensor data may be a second image with a smaller size. The size of the first image and the second image in at least one direction of height and width is the same, so that the first image and the second image can be stitched in the height or width direction. For example, the size of the first image is H1*W1=512*1024, and the size of the second image is H2*W2=128*1024, where H represents the image height and W represents the image width. The first image and the second image are stitched in the height direction to obtain a 640*1024 fused image (i.e., fused sensor data).
[0085] Step 23120: Based on the intelligent driving system, the fused sensor data is processed to obtain a first processing result and a second processing result.
[0086] The fused sensor data includes the current frame sensor data and the first test sensor data, and the fused sensor data is transmitted to the intelligent driving system as a frame of sensor data. The intelligent driving system performs signal processing, feature extraction and perception, positioning and mapping, trajectory prediction, planning and decision-making on the fused sensor data to obtain a processing result. According to the corresponding relationship between the output result data and the fused sensor data, and the position relationship between the current frame sensor data and the first test sensor data in the fused sensor data, the first processing result and the second processing result are determined from the processing result.
[0087] In some optional embodiments, a sensor data fusion function may be added to the intelligent driving system. The current frame sensor data and the first test sensor data may be fused through the sensor data fusion function to obtain fused sensor data. The fused sensor data may then be transmitted to a functional part of the intelligent driving system for processing the fused sensor data, such as a signal processor. The fused sensor data may then be processed in sequence including signal processing, feature extraction and perception, positioning and mapping, trajectory prediction, planning and decision-making, etc. to obtain a processing result.
[0088] In the disclosed embodiment, the current frame sensor data is fused with the first test sensor data to obtain fused sensor data, so that the intelligent driving system can simultaneously complete the processing of the current frame sensor data and the first test sensor data through one processing process, and synchronously obtain the first processing result and the second processing result, which can reduce the number of processing times of the intelligent driving system and thus improve processing efficiency.
[0089] Figure 7 It is a flowchart of a monitoring method of an intelligent driving system provided by another exemplary embodiment of the present disclosure.
[0090] In some optional embodiments, such as Figure 7 As shown, after determining the monitoring result of the intelligent driving system based on the first result label data and the first processing result in step 240, the method of the embodiment of the present disclosure may further include:
[0091] Step 250: Determine whether the second processing result is available based on the monitoring result.
[0092] Among them, the monitoring result characterizes whether the intelligent driving system is abnormal, and the second processing result is the processing result of the intelligent driving system on the current frame sensor data. If the intelligent driving system is abnormal, the reliability of the second processing result is affected. Therefore, the monitoring result can determine whether the second processing result is available, and the state of whether the second processing result is available can be determined based on the monitoring result. The state of whether the second processing result is available can include two states: available and unavailable. If the monitoring result shows that the intelligent driving system is abnormal, it can be determined that the second processing result is unavailable. If the monitoring result shows that the intelligent driving system is normal, it can be determined that the second processing result is available, and the second processing result can be used for downstream functions. For example, the second processing result is a planning and decision-making result. The planning and decision-making result can be a generated driving path, which can be used to control the driving state of a movable device.
[0093] Step 260: Control the state of the movable device where the intelligent driving system is located according to whether the second processing result is available.
[0094] The state of the mobile device may include the driving state of the mobile device, the state of the output device on the mobile device and other related devices. For example, the driving state of the mobile device is controlled according to the second processing result, or the output device on the mobile device is controlled to output prompt information.
[0095] In the disclosed embodiment, whether the second processing result corresponding to the current frame sensor data is available can be determined through the monitoring result of the intelligent driving system. The state of the movable device is controlled according to the state of whether the second processing result is available, thereby effectively improving the safety of the movable device.
[0096] In some optional embodiments, step 260 of controlling the state of a movable device where the intelligent driving system is located according to whether the second processing result is available may include: in response to the state of whether the second processing result is available being an available state, controlling the driving state of the movable device according to the second processing result; or, in response to the state of whether the second processing result is available being an unavailable state, controlling the movable device to output system defect prompt information.
[0097] Among them, if the status of whether the second processing result is available is an available state (that is, the second processing result is available), it means that the second processing result can be used for downstream functions, and therefore, the driving state of the movable device can be controlled according to the second processing result. For example, if the second processing result is a driving path obtained by planning decisions, the driving state of the movable device can be controlled according to the driving path. If the status of whether the second processing result is available is an unavailable state (that is, the second processing result is unavailable), it means that an abnormality has occurred in the intelligent driving system, and the movable device can be controlled to output system defect prompt information to prompt the user to take timely measures, such as suspending the use of the intelligent driving function to ensure the driving safety of the movable device.
[0098] In the embodiment of the present disclosure, when it is determined that the second processing result of the current frame is unavailable, system defect prompt information can be output in a timely manner, so that the user can be informed in a timely manner that a system defect has occurred in the intelligent driving system and take countermeasures in time to avoid accidents of the movable device and further improve the driving safety of the movable device.
[0099] Figure 8 It is a flowchart of a monitoring method of an intelligent driving system provided by yet another exemplary embodiment of the present disclosure.
[0100] In some optional embodiments, such as Figure 8 As shown, the determination of the current environment information corresponding to the intelligent driving system in step 210 may include:
[0101] Step 2120: In response to the current frame being a start frame, the preset environment information is determined as the current environment information.
[0102] The preset environmental information is a type of environmental information in the test data set. The current frame being the starting frame indicates that the current frame sensor data is the first frame sensor data, that is, there is no historical environmental information obtained at the previous moment. In this case, the preset environmental information can be determined as the current environmental information for monitoring the intelligent driving system. The preset environmental information can be a type of environmental information determined in advance from the test data set. For example, a type of environmental information can be randomly obtained from the test data set as the current environmental information.
[0103] Determining the first test sensor data corresponding to the current environment information and the first result label data corresponding to the first test sensor data based on the preconfigured test data set in step 220 may include:
[0104] Step 22a0: determine the test sensor data corresponding to the preset environmental information in the test data set as the first test sensor data, and determine the result label data of the test sensor data corresponding to the preset environmental information as the first result label data.
[0105] Among them, the preset environmental information is a type of environmental information in the test data set. According to the preset environmental information, the test sensor data corresponding to the preset environmental information and the result label data corresponding to the test sensor data can be extracted from the test data set to obtain the first test sensor data corresponding to the current environmental information and the corresponding first result label data.
[0106] In the embodiment of the present disclosure, the preset environmental information in the test data set can be used as the current environmental information in the starting frame, so as to monitor the starting frame sensor data processing process of the intelligent driving system and ensure the reliability of the intelligent driving system.
[0107] In some optional embodiments, if the current frame is the starting frame, the monitoring process of the intelligent driving system may not be performed, and the intelligent driving system performs signal processing, feature extraction and perception, planning and decision-making, etc. according to the sensor data of the current frame to obtain a second processing result for controlling the driving state of the movable device. In the processing of the starting frame, the environmental information of the starting frame is obtained and stored. In the processing of the current frame as the second frame, the environmental information of the starting frame is used as the historical environmental information at the previous moment to determine the current environmental information of the second frame, and then based on the current environmental information, the intelligent driving system is monitored according to the monitoring process of the above embodiment to obtain a monitoring result.
[0108] In some optional embodiments, the first result label data includes at least one of planning decision result label data, perception result label data, and trajectory prediction result label data; the first processing result includes at least one of the planning decision prediction result, perception prediction result, and trajectory prediction result corresponding to the first test sensor data.
[0109] Among them, the perception prediction result may include the driving environment condition information, road scene information, etc. obtained by perception. The driving environment condition information may include, for example, brightness, color temperature, noise, fog index, etc. Road scene information may include, for example, meeting scenes, ramp merging into the main road scenes, lane change scenes, road element information, traffic participants and other information. The perception result label data includes the driving environment condition label data and road scene label data marked corresponding to the first test sensor data. The trajectory prediction result may include the predicted driving trajectory information, and the trajectory prediction result label data includes the marked expected driving trajectory information. The planning and decision-making prediction results can be obtained through the planning and decision-making functions in the intelligent driving system, the perception prediction results can be obtained through the feature extraction and perception functions in the intelligent driving system, and the trajectory prediction results can be obtained through the trajectory prediction function in the intelligent driving system.
[0110] In some optional embodiments, the first result label data may also include other related result label data, such as positioning and mapping result label data. Accordingly, the first processing result may also include prediction processing results corresponding to other related result label data, such as positioning and mapping prediction results. The positioning and mapping function in the intelligent driving system can locate the posture of the movable device according to the perception results, and construct a local map within a preset range around the movable device. For example, the positioning and mapping function can obtain positioning and mapping prediction results based on the SLAM (Simultaneous Localization and Mapping) algorithm or other positioning and mapping algorithms.
[0111] Determining the monitoring result of the intelligent driving system based on the first result tag data and the first processing result in step 240 may include:
[0112] Compare the planning decision prediction result with the planning decision result label data to obtain a first comparison result; compare the perception prediction result with the perception result label data to obtain a second comparison result; compare the trajectory prediction result with the trajectory prediction result label data to obtain a third comparison result; determine the monitoring result of the intelligent driving system based on the first comparison result, the second comparison result and the third comparison result.
[0113] Among them, the comparison results of each prediction result (such as planning decision prediction result, perception prediction result, trajectory prediction result) and the corresponding result label data may include consistent and inconsistent results. Or the comparison result may include the similarity (or consistency) between the prediction result and the result label data. By combining the comparison results of various prediction results and the corresponding result label data to determine the monitoring results of the intelligent driving system, the accuracy and effectiveness of the monitoring results can be improved.
[0114] Fig. 9 It is a flowchart of a monitoring method of an intelligent driving system provided by another exemplary embodiment of the present disclosure.
[0115] In some optional embodiments, after determining the monitoring result of the intelligent driving system based on the first result tag data and the first processing result in step 230, the method of the embodiment of the present disclosure may further include:
[0116] Step 410: In response to the monitoring result satisfying the system abnormality condition, outputting system defect prompt information.
[0117] Among them, if the monitoring result includes the result of whether the intelligent driving system is normal, the system abnormality condition is that the monitoring result satisfies that the intelligent driving system is abnormal. If the monitoring result includes the result of whether the first processing result is consistent with the first result label data, the system abnormality condition is that the monitoring result satisfies that the first processing result is inconsistent with the first result label data. If the monitoring result includes the degree of consistency between the first processing result and the first result label data, the system abnormality condition is that the degree of consistency between the first processing result and the first result label data is less than a preset threshold. If the monitoring result satisfies the system abnormality condition, it means that the intelligent driving system may have a system abnormality, and the system defect prompt information can be output in time to promptly remind the user that the intelligent driving system has a system abnormality, so that the user can take countermeasures in time to ensure the safety of the movable device.
[0118] In some optional embodiments, the system defect prompt information may be output in any manner, such as voice output, display output, voice combined with display output, etc.
[0119] In some optional embodiments, Fig.10 FIG. 1 is a schematic diagram of the system architecture of an intelligent driving system provided by an exemplary embodiment of the present disclosure. Fig.10As shown, the intelligent driving system 11 may include a software part 111 and a hardware part 112. The software part 111 may include applications (software) related to intelligent driving services such as data acquisition 1111, data processing 1112, feature extraction and perception 1113, positioning and mapping 1114, prediction 1115, planning and decision-making 1116, data multimedia perception SDK (Software Development Kit) 1117 and intelligent driving software suite 1118 supporting business-related applications, and an operating system (Operating System, referred to as: OS) 1119. The hardware part 112 includes a communication interface 1121, a signal processing component 1122, a neural network processor, a central processing unit (Central Processing Unit, referred to as: CPU), a graphics processing unit (Graphics Processing Unit, referred to as: GPU), and a microcontroller (Microcontroller Unit, referred to as: MCU). The communication interface 1121 may include, for example, MIPI (Mobile Industry Processor Interface), CAN (Controller Area Network) interface, etc. The signal processing component 1122 may include, for example, an ISP (Image Signal Processor), a PYM (pyramid) component, etc. The neural network processor may include, for example, a BPU (Brain Processing Unit), a VDPS (Vision Digital Signal Processor), etc. Among them, each application may run on the operating system 1119, access each hardware in the hardware part 112 through the operating system, and the software and hardware collaborate to realize the overall business process of intelligent driving. For example, the function of data acquisition 1111 is realized through the communication interface 1121, the function of data processing 1112 is realized through the signal processing component 1122, the function of feature extraction and perception 1113 is realized through the neural network processor 1123, and so on. The sensor data transmitted to the intelligent driving system 11 may include at least one of the current frame sensor data collected in real time and the first test sensor data mentioned above.That is, the first test sensor data and the current frame sensor data collected in real time by the preset sensor will go through the same data processing process in the intelligent driving system 11, and the first test sensor data and the current frame sensor data will be processed by data collection, data processing, feature extraction and perception, positioning and mapping, prediction, planning and decision-making in combination with the software part 111 and the hardware part 112 in the intelligent driving system 11, and the first processing result corresponding to the first test sensor data is obtained. The first processing result is verified according to the first result label data corresponding to the first test sensor data, which is used to determine whether the intelligent driving system 11 is abnormal, and whether the second processing result corresponding to the current frame sensor data is available. If the intelligent driving system 11 is abnormal, the intelligent driving system defect prompt information (i.e., system defect prompt information) can be output in time, and the prompt information that the second processing result is unavailable can be output. In the case where the second processing result is unavailable, avoid using the second processing result to control the driving state of the movable device to ensure the driving safety of the movable device. The overall architecture of the driving system 11 here is only an exemplary system architecture. In practical applications, it is not limited to the intelligent driving system 11 shown in the figure, and can also be any other possible system architecture.
[0120] In some optional embodiments, Fig.10 The function of the monitoring method of the intelligent driving system of the embodiment of the present disclosure is configured in the intelligent driving system 11 shown, that is, the device (such as a processor) with corresponding processing functions in the intelligent driving system 11 is used as the monitoring device of the embodiment of the present disclosure to realize the built-in self-test of the intelligent driving system 11. For example, a sensor data fusion component can be added to the hardware part 112, and relevant software for realizing the monitoring method of the intelligent driving system of the embodiment of the present disclosure can be added to the software part 111. The sensor data fusion component can be configured under the configuration of the software to fuse the current frame sensor data with the first test sensor data to obtain fused sensor data, and then obtain the second processing result corresponding to the current frame sensor data and the first processing result corresponding to the first test sensor data through data processing 1112, feature extraction and perception 1113, positioning and mapping 1114, prediction 1115, planning and decision 1116. The output result data may include the first processing result and the second processing result.
[0121] The method of the disclosed embodiment can generate first test sensor data that conforms to the real-time environment according to real-time environmental information and pre-configured test data sets during the operation of the intelligent driving system, which is used to perform built-in self-test covering all dimensions of the intelligent driving system, and realize efficient and automated monitoring of the intelligent driving system, so as to monitor and diagnose defects or abnormal conditions of the intelligent driving system, and ensure the functional safety of the intelligent driving system. In addition, before the second processing result corresponding to the real-time collected sensor data is used for downstream control, the availability of the second processing result can be judged based on the monitoring result corresponding to the first test sensor data, so as to avoid using unreliable planning decision results to control the driving state of the movable device, and further ensure the driving safety of the movable device. In the related art, a monitoring or testing unit is usually set inside the hardware component in the intelligent driving system to monitor or test the correctness of the local functional logic inside the hardware component, or a monitoring or testing unit is set inside the software component to monitor or test the correctness of the local functional logic inside the software component. Compared with the monitoring methods of related technologies, the method of the embodiments of the present invention can achieve overall system-level monitoring of the software and hardware of the intelligent driving system by comparing the processing results of the test sensor data by the intelligent driving system with the corresponding result label data, thereby avoiding traffic accidents caused by erroneous perception results, planning decision results, etc., and effectively improving the monitoring effect.
[0122] The above-mentioned embodiments of the present disclosure may be implemented separately or in any combination without conflict. The specific configuration may be based on actual needs and the present disclosure does not limit this.
[0123] Any monitoring method of an intelligent driving system provided by the embodiments of the present disclosure can be executed by any appropriate electronic device with data processing capabilities, including but not limited to: electronic devices such as terminal devices and servers. Alternatively, any monitoring method of an intelligent driving system provided by the embodiments of the present disclosure can be executed by a processor, such as a processor executing any monitoring method of an intelligent driving system mentioned in the embodiments of the present disclosure by calling corresponding instructions stored in a memory. This will not be described in detail below.
[0124] Exemplary Devices
[0125] Fig.11 is a schematic diagram of the structure of a monitoring device for an intelligent driving system provided by an exemplary embodiment of the present disclosure. The device of this embodiment can be used to implement the corresponding method embodiment of the present disclosure, such as Fig.11 The device shown may include: a first processing module 51 , a second processing module 52 , a third processing module 53 and a fourth processing module 54 .
[0126] The first processing module 51 is used to determine the current environment information corresponding to the intelligent driving system.
[0127] The second processing module 52 is used to determine first test sensor data corresponding to the current environment information and first result label data corresponding to the first test sensor data based on a preconfigured test data set.
[0128] The third processing module 53 is used to process the first test sensor data based on the intelligent driving system to obtain a first processing result corresponding to the first test sensor data.
[0129] The fourth processing module 54 is used to determine the monitoring result of the intelligent driving system based on the first result label data and the first processing result.
[0130] Fig.12 It is a structural schematic diagram of a monitoring device for an intelligent driving system provided by another exemplary embodiment of the present disclosure.
[0131] In some optional embodiments, in the above Fig.11 Based on the embodiment shown, Fig.12 As shown, the first processing module 51 may include: a first processing unit 511, configured to determine the historical environmental information at a previous moment as the current environmental information. The historical environmental information is environmental information sensed and obtained based on sensor data collected by a preset sensor at a previous moment.
[0132] Fig.13 It is a schematic diagram of the structure of a monitoring device for an intelligent driving system provided by yet another exemplary embodiment of the present disclosure.
[0133] In some optional embodiments, based on any of the above embodiments, the current environment information includes current driving environment condition information and current road scene information.
[0134] In some optional embodiments, based on any of the above embodiments, Fig.13 As shown, the second processing module 52 may include: a second processing unit 521 , a third processing unit 522 and a first determining unit 523 .
[0135] The second processing unit 521 is used to determine target test sensor data matching the current environment information and result label data corresponding to the target test sensor data from the test data set based on the current driving environment condition information and the current road scene information.
[0136] The third processing unit 522 is used to adjust the target test sensor data based on the current driving environment condition information to obtain the first test sensor data.
[0137] The first determining unit 523 is configured to use the result label data corresponding to the target test sensor data as the first result label data.
[0138] Fig.14 It is a structural schematic diagram of a monitoring device for an intelligent driving system provided by yet another exemplary embodiment of the present disclosure.
[0139] In some optional embodiments, based on any of the above embodiments, Fig.14 As shown, the device of the embodiment of the present disclosure may further include: an acquisition module 61, which is used to acquire current frame sensor data collected by a preset sensor.
[0140] The third processing module 53 may include: a fourth processing unit 531, which is used to process the current frame sensor data and the first test sensor data based on the intelligent driving system to obtain a first processing result corresponding to the first test sensor data and a second processing result corresponding to the current frame sensor data.
[0141] In some optional embodiments, the fourth processing unit 531 may be specifically configured to:
[0142] The current frame sensor data and the first test sensor data are fused to obtain fused sensor data; based on the intelligent driving system, the fused sensor data is processed to obtain a first processing result and a second processing result.
[0143] In some optional embodiments, the fourth processing unit 531 can be specifically used to fuse the current frame sensor data with the first test sensor data to obtain fused sensor data, and transmit the fused sensor data to the intelligent driving system so that the intelligent driving system processes the fused sensor data to obtain a processing result, which includes a first processing result and a second processing result.
[0144] In some optional embodiments, the fourth processing unit 531 may be a hardware fusion unit, configured to fuse the current frame sensor data with the first test sensor data to obtain fused sensor data. Alternatively, the fourth processing unit 531 may control the hardware fusion unit to fuse the current frame sensor data with the first test sensor data to obtain fused sensor data.
[0145] In some optional embodiments, such as Fig.14 As shown, the device of the embodiment of the present disclosure may further include: a fifth processing module 62 and a sixth processing module 63 .
[0146] The fifth processing module 62 is used to determine whether the second processing result is usable according to the monitoring result.
[0147] The sixth processing module 63 is used to control the state of the movable device where the intelligent driving system is located according to whether the second processing result is available.
[0148] In some optional embodiments, the sixth processing module 63 can be specifically used to: in response to the second processing result being in an available state, control the driving state of the movable device according to the second processing result; or, in response to the second processing result being in an unavailable state, control the movable device to output system defect prompt information.
[0149] In some optional embodiments, such as Fig.14 As shown, the first processing module 51 may include:
[0150] The first processing unit 511 is configured to determine preset environment information as current environment information in response to the current frame being a start frame, wherein the preset environment information is a type of environment information in a test data set.
[0151] The second processing module 52 may include:
[0152] The second determining unit 524 is used to determine the test sensor data corresponding to the preset environmental information in the test data set as the first test sensor data, and determine the result label data of the test sensor data corresponding to the preset environmental information as the first result label data.
[0153] In some optional embodiments, the first result label data includes at least one of planning decision result label data, perception result label data, and trajectory prediction result label data; the first processing result includes at least one of the planning decision prediction result, perception prediction result, and trajectory prediction result corresponding to the first test sensor data.
[0154] Fig.15 It is a schematic diagram of the structure of a monitoring device for an intelligent driving system provided by yet another exemplary embodiment of the present disclosure.
[0155] In some optional embodiments, such as Fig.15 As shown, the device of the embodiment of the present disclosure may also include:
[0156] The output module 55 is used to output system defect prompt information in response to the monitoring result meeting the system abnormality condition.
[0157] The beneficial technical effects corresponding to the exemplary embodiment of the present device can be found in the corresponding beneficial technical effects of the above exemplary method section, which will not be repeated here.
[0158] Exemplary Electronic Devices
[0159] Fig.16 It is a structural diagram of an electronic device provided by an embodiment of the present disclosure, including at least one processor 91 and a memory 92.
[0160] The processor 91 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 90 to perform desired functions.
[0161] The memory 92 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 91 may execute one or more computer program instructions to implement the methods and / or other desired functions of the various embodiments of the present disclosure described above.
[0162] In one example, the electronic device 90 may further include: an input device 93 and an output device 94 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).
[0163] The input device 93 may also include, for example, a touch screen, a microphone, various sensors, and the like. Sensors may include, for example, image sensors (such as cameras, cameras, etc.), laser radars, millimeter-wave radars, ultrasonic radars, positioning sensors, pressure sensors, air quality sensors, temperature sensors, and the like. Image sensors, laser radars, millimeter-wave radars, ultrasonic radars, and the like may be used to sense the surrounding environment, that is, to detect dynamic and static objects in the surrounding environment. Dynamic and static objects may include, for example, static objects such as lane lines, curbs, arrows, signs, trees, and buildings, as well as dynamic objects such as surrounding vehicles, pedestrians, and cyclists. Positioning sensors are used to realize the positioning of movable devices (such as self-vehicles, robots, etc.) where electronic devices are located. Positioning sensors may include, for example, inertial measurement units (Inertial Measurement Unit, referred to as: IMU), global positioning systems (Global Positioning System, referred to as: GPS), and the like. Pressure sensors may be used to detect seat pressure. Temperature sensors may be used to detect the temperature in the vehicle cabin. Air quality sensors may be used to detect the air quality in the vehicle cabin.
[0164] The output device 94 can output various information to the outside, and may include, for example, a display, a speaker, a communication network and a remote output device connected thereto, and the like.
[0165] Of course, to simplify, Fig.16Only some of the components related to the present disclosure in the electronic device 90 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 90 may also include any other appropriate components.
[0166] An embodiment of the present disclosure also provides an electronic device, which includes the monitoring device of the intelligent driving system provided by any of the above embodiments.
[0167] In some optional embodiments, the electronic device may also include an intelligent driving system.
[0168] Exemplary computer program products and computer-readable storage media
[0169] In addition to the above methods and devices, embodiments of the present disclosure may also provide a computer program product, including computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the methods of various embodiments of the present disclosure described in the above “Exemplary Methods” section.
[0170] The computer program product may be written in any combination of one or more programming languages to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.
[0171] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the processor executes the steps in the method of various embodiments of the present disclosure described in the above “Exemplary Method” section.
[0172] Computer readable storage media can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium is, for example, but not limited to, a system, device or device including electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
[0173] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and cannot be considered as necessary for each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, rather than limitation, and the above details do not limit the present disclosure to being implemented by adopting the above specific details.
[0174] Those skilled in the art may make various changes and modifications to the present disclosure without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the present disclosure claims and their equivalents, the present disclosure is also intended to include these modifications and variations.
Claims
1. A monitoring method for an intelligent driving system, comprising: Determine current environment information corresponding to the intelligent driving system; Based on a preconfigured test data set, determining first test sensor data corresponding to the current environment information and first result label data corresponding to the first test sensor data; the test data set includes test sensor data under different environments and corresponding result label data; Based on the intelligent driving system, processing the first test sensor data to obtain a first processing result corresponding to the first test sensor data; Based on the first result label data and the first processing result, a monitoring result of the intelligent driving system is determined.
2. The method according to claim 1, wherein: The current environment information includes current driving environment condition information and current road scene information; The determining, based on a preconfigured test data set, first test sensor data corresponding to the current environment information and first result label data corresponding to the first test sensor data includes: Based on the current driving environment condition information and the current road scene information, determining target test sensor data matching the current environment information and result label data corresponding to the target test sensor data from the test data set; Based on the current driving environment condition information, adjusting the target test sensor data to obtain the first test sensor data; The result label data corresponding to the target test sensor data is used as the first result label data.
3. The method according to claim 1, wherein: The determining of current environment information corresponding to the intelligent driving system includes: The historical environmental information at a previous moment is determined as the current environmental information; the historical environmental information is environmental information sensed and obtained based on sensor data collected by a preset sensor at the previous moment.
4. The method according to claim 1, wherein: The method further comprises: Get the current frame sensor data collected by the preset sensor; The step of processing the first test sensor data based on the intelligent driving system to obtain a first processing result corresponding to the first test sensor data includes: Based on the intelligent driving system, the current frame sensor data and the first test sensor data are processed to obtain a first processing result corresponding to the first test sensor data and a second processing result corresponding to the current frame sensor data.
5. The method according to claim 4, wherein: The processing of the current frame sensor data and the first test sensor data based on the intelligent driving system to obtain a first processing result corresponding to the first test sensor data and a second processing result corresponding to the current frame sensor data includes: fusing the current frame sensor data with the first test sensor data to obtain fused sensor data; Based on the intelligent driving system, the fused sensor data is processed to obtain the first processing result and the second processing result.
6. The method according to claim 4, wherein: After determining the monitoring result of the intelligent driving system based on the first result label data and the first processing result, the method further includes: Determining, according to the monitoring result, whether the second processing result is available; The state of the movable device where the intelligent driving system is located is controlled according to whether the second processing result is available.
7. The method according to claim 6, wherein: The controlling the state of the movable device where the intelligent driving system is located according to whether the second processing result is available includes: In response to the state of whether the second processing result is available being available, controlling the driving state of the movable device according to the second processing result; or, In response to the state of whether the second processing result is available being an unavailable state, the movable device is controlled to output system defect prompt information.
8. The method according to claim 4, wherein: The determining of current environment information corresponding to the intelligent driving system includes: In response to the current frame being a start frame, determining preset environmental information as the current environmental information; the preset environmental information is a type of environmental information in the test data set; The determining, based on a preconfigured test data set, first test sensor data corresponding to the current environment information comprises: The test sensor data corresponding to the preset environmental information in the test data set is determined as the first test sensor data.
9. The method according to any one of claims 1 to 8, wherein: The first result label data includes at least one of planning decision result label data, perception result label data and trajectory prediction result label data; the first processing result includes at least one of the planning decision prediction result, perception prediction result and trajectory prediction result corresponding to the first test sensor data.
10. The method according to any one of claims 1 to 8, wherein: After determining the monitoring result of the intelligent driving system based on the first result label data and the first processing result, the method further includes: In response to the monitoring result satisfying the system abnormality condition, system defect prompt information is output.
11. A monitoring device for an intelligent driving system, comprising: A first processing module, used to determine current environment information corresponding to the intelligent driving system; A second processing module, configured to determine first test sensor data corresponding to the current environment information and first result label data corresponding to the first test sensor data based on a preconfigured test data set; A third processing module, configured to process the first test sensor data based on the intelligent driving system to obtain a first processing result corresponding to the first test sensor data; A fourth processing module is used to determine the monitoring result of the intelligent driving system based on the first result label data and the first processing result.
12. A computer-readable storage medium storing a computer program, wherein the computer program is used to execute the method according to any one of claims 1 to 10.
13. An electronic device, comprising: processor; a memory for storing instructions executable by the processor; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the method according to any one of claims 1 to 10; or The electronic device comprises the device according to claim 11.
