Data acquisition method, electronic equipment and storage medium

Abstract

The invention provides a data acquisition method, electronic equipment and a storage medium, and relates to the technical field of intelligent driving. The method comprises the following steps: acquiring vehicle data, driver state data, radar detection data and visual sensor detection data; based on at least one of the vehicle data, the driver state data, the radar detection data and the visual sensor detection data, whether the working condition belongs to a missing report working condition, a false report working condition or other preset special working conditions is judged; if it is judged that the working condition belongs to the missing report working condition, the false report working condition or other preset special working conditions, collected data corresponding to the working condition is recorded and uploaded. According to the invention, only required effective data can be recorded in a targeted and efficient manner, a large amount of data does not need to be recorded and stored in real time, the time required for data acquisition and iteration can be shortened, the data acquisition efficiency is improved, and manpower is saved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent driving technology, and in particular to a data acquisition method, electronic equipment and storage medium. Background Art

[0002] In the field of intelligent driving, data plays a decisive role in intelligent products. Therefore, data collection is very important in the field of intelligent driving, especially collecting effective data is very important for optimizing intelligent driving related functions.

[0003] In the related art, a large amount of data is usually collected and stored in real time through data collection equipment, and then the collected data is cleaned, for example, data slicing, deduplication, and manual screening are performed to obtain valid data. However, this method takes a long time to collect and iterate data, and is inefficient. Summary of the invention

[0004] The embodiments of the present invention provide a data collection method, an electronic device and a storage medium to solve the problem that data collection and iteration take a long time and have low efficiency.

[0005] In a first aspect, an embodiment of the present invention provides a data collection method, including: Acquire vehicle data, driver status data, radar detection data, and visual sensor detection data; Based on at least one of the vehicle data, the driver status data, the radar detection data and the visual sensor detection data, it is determined whether it is a missed reporting condition, a false reporting condition or other preset special conditions; If it is determined to be an underreported condition, a false alarm condition, or other special conditions, the collected data corresponding to the condition will be recorded and uploaded.

[0006] In a possible implementation, judging whether it is an underreporting condition based on at least one of vehicle data, driver status data, radar detection data, and visual sensor detection data includes: In the process of target fusion based on radar detection data and visual sensor detection data, if a first radar target exists, the confidence level of the visual sensor in target detection is reduced to obtain a first confidence level; the first radar target is a radar target for which there is no corresponding first visual target; Based on the first confidence level and the first target detection model, performing target detection by a visual sensor to obtain first visual target detection data; If there is a first visual target corresponding to the first radar target in the first visual target detection data, it is determined that it belongs to the first underreporting condition.

[0007] In one possible implementation, the visual sensor detection data includes an image; After obtaining the first visual target detection data, judging whether it is an underreporting condition based on at least one of the vehicle data, the driver state data, the radar detection data, and the visual sensor detection data, further includes: If the first visual target corresponding to the first radar target still does not exist in the first visual target detection data, obtaining coordinate data corresponding to the first radar target, and determining the position of the first visual target in the image according to the coordinate data corresponding to the first radar target; According to the position of the first visual target in the image, the image is intercepted to obtain a intercepted image; Performing target detection with a visual sensor according to the captured image and the second target detection model to obtain second visual target detection data; If there is a first visual target corresponding to the first radar target in the second visual target detection data, it is determined to belong to the second underreporting condition; If the first visual target corresponding to the first radar target still does not exist in the second visual target detection data, it is determined to belong to the third underreporting condition.

[0008] In a possible implementation, the vehicle data includes at least one of vehicle speed, deceleration, three-axis acceleration, three-axis angular velocity, and steering wheel speed; Based on at least one of the vehicle data, the driver status data, the radar detection data and the visual sensor detection data, it is determined whether it is an underreporting condition, including: If the vehicle speed, deceleration, three-axis acceleration, three-axis angular velocity and steering wheel speed meet the preset vehicle drastic change conditions, it is determined to belong to the fourth underreporting condition.

[0009] In a possible implementation, judging whether it belongs to other preset special operating conditions based on at least one of vehicle data, driver status data, radar detection data, and visual sensor detection data includes: Based on the detection data of the visual sensor, determine the farthest distance of the lane line and the farthest distance of the visual target, and determine the visual visibility according to the farthest distance of the lane line and the farthest distance of the visual target; Predicting the drivable area of ​​the vehicle based on the visual sensor detection data, and determining the farthest end distance and lateral range span of the drivable area based on the drivable area of ​​the vehicle; Determine the brightness of the image captured by the visual sensor based on the visual sensor detection data; Based on the radar detection data and the visual sensor detection data, determining whether there is a first radar target; the first radar target is a radar target for which there is no corresponding first visual target; If the first radar target is present, determining the number of consecutive images for which there is no corresponding first visual target; If the visual visibility, the farthest distance of the drivable area, the lateral range span of the drivable area, the brightness of the image and the number of consecutive images meet the preset visual sensor failure conditions, it is determined to belong to other preset special working conditions.

[0010] In a possible implementation, other special working conditions are preset including a visual sensor failure working condition; The preset sensor failure conditions include at least one of the following: visual visibility is less than a preset visibility threshold, the farthest end distance of the drivable area is less than a preset distance threshold, the lateral range span of the drivable area is less than a preset span threshold, the brightness of the image exceeds a preset normal brightness range, and the number of consecutive images is greater than a preset number.

[0011] In a possible implementation, judging whether it is a false alarm condition based on at least one of vehicle data, driver status data, radar detection data, and visual sensor detection data includes: Determine whether danger occurs based on radar detection data and visual sensor detection data; If it is determined that a danger has occurred, danger avoidance action will be performed; Determine whether the vehicle data and driver status data within a first preset time period meet the preset false alarm conditions; if so, determine that it is a false alarm condition; the first preset time period starts at a first time point and ends at a second time point, the first time point is the time point when the danger avoidance action starts, the second time point is after the time point when the danger avoidance action ends, and is separated from the time point when the danger avoidance action ends by a first preset time length.

[0012] In one possible implementation, the vehicle data includes a brake signal and a steering wheel signal; The preset false alarm conditions include that no brake signal is generated within a first preset time period, no change in the steering wheel signal within the first preset time period, and no change in the driver status data within the first preset time period.

[0013] In a second aspect, an embodiment of the present invention provides a data acquisition device, including: An acquisition module, used to acquire vehicle data, driver status data, radar detection data, and visual sensor detection data; A working condition judgment module, used to judge whether it is an underreporting working condition, a false reporting working condition or other preset special working conditions based on at least one of vehicle data, driver status data, radar detection data and visual sensor detection data; The data collection module is used to record and upload the collected data corresponding to the working condition if it is determined to be an underreported working condition, a false alarm working condition or other preset special working conditions.

[0014] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the data acquisition method described in the first aspect or any possible implementation of the first aspect.

[0015] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data acquisition method described in the first aspect or any possible implementation method of the first aspect are implemented.

[0016] An embodiment of the present invention provides a data collection method, an electronic device and a storage medium. The method determines whether it belongs to an underreporting condition, a false alarm condition or other preset special conditions by obtaining at least one of the vehicle data, driver status data, radar detection data and visual sensor detection data; if it is determined to be an underreporting condition, a false alarm condition or other preset special conditions, the collected data corresponding to the condition is recorded and uploaded, so that only the required valid data can be recorded in a targeted and efficient manner, without the need to record and store a large amount of data in real time, which can reduce the time required for data collection and iteration, improve data collection efficiency and save manpower. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0018] Figure 1 is a structural diagram of a data acquisition system provided by an embodiment of the present invention; Figure 2 It is a flow chart of a data collection method provided by an embodiment of the present invention; Figure 3 is a structural schematic diagram of a data acquisition device provided by an embodiment of the present invention; Figure 4 is a schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0019] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0020] In order to make the purpose, technical solutions and advantages of the present invention more clear, specific embodiments will be described below in conjunction with the accompanying drawings.

[0021] See also Figure 1 , which shows a schematic diagram of the structure of the data acquisition system provided by the embodiment of the present application. The data acquisition system may include a domain controller, a vehicle posture acquisition module, a global positioning module, a decision and execution module, a vehicle signal acquisition module, a radar and a camera, etc. The data acquisition system includes but is not limited to the above modules, for example, see Figure 1 , and may also include a communication module, a storage module, a map module, and the like. Among them, the camera may include an ADAS (Advanced Driving Assistance System) camera and a DMS (Driver Monitoring System) camera. The camera may include cameras installed around the outside of the vehicle and cameras installed inside the vehicle. The radar may include radars installed around the outside of the vehicle.

[0022] The domain controller is mainly used to receive signals and process, fuse, store data, make decisions and control the received data; the global positioning module is used to collect the location of the vehicle; the radar generally uses a typical millimeter wave radar, and other types of radars can also be used; the ADAS camera is a visual sensor used to capture images of the viewing angle of interest; the cloud acts as a server and transmits signals and video data to the domain controller through a communication module (4G / 5G); the DMS camera is a driver monitoring camera that can detect the driver's behavior and status, such as distracted driving, phone calls, fatigue, closed eyes, etc. The vehicle posture acquisition module is used to detect the vehicle posture, etc.; the vehicle signal acquisition module is used to collect vehicle signal data, etc.; the decision and execution module can be used to make decisions and execute vehicle control behaviors, such as when danger is detected, the vehicle can be controlled to brake, etc.; the map module can provide maps for users or other functional modules to use; the storage module is used for data storage.

[0023] Among them, the vehicle posture acquisition module, global positioning module, decision and execution module, vehicle signal acquisition module, communication module, storage module and map module can be located inside the domain controller, that is, they are part of the domain controller; they can also be located outside the domain controller and connected to the domain controller, that is, they are not part of the domain controller, but have a connection relationship with the domain controller and can perform data transmission; they can also be located partially inside the domain controller and partially outside the domain controller, and so on.

[0024] For example, the overall workflow is that the radar outputs a target list signal, the camera outputs an image, the domain controller receives the image and performs target recognition, lane line detection, and drivable area detection on the image. After detecting and recognizing the image target, the target list and lane line information can be obtained through post-processing. The visual target list obtained by the image is integrated with the radar target list obtained by the radar, and combined with the vehicle's posture, speed and other signals, forward AEB, ACC, LKA and other functions can be completed.

[0025] based on Figure 1 The structure shown, see Figure 2 The flow chart of the data collection method shown in the figure introduces the data collection method provided in the embodiment of the present application in detail. The execution subject of the data collection method can be an electronic device. The electronic device can be a controller in a vehicle, for example, it can be the above-mentioned domain controller, etc.

[0026] See also Figure 2 , the above data collection method includes: In S201 , vehicle data, driver status data, radar detection data, and visual sensor detection data are acquired.

[0027] The vehicle data is data related to the vehicle, for example, signal data of various components in the vehicle, operation data of the vehicle, etc., and may include at least one of vehicle speed, acceleration / deceleration, vehicle tilt angle, steering wheel speed, steering wheel angle, brake signal, steering wheel signal, etc. The vehicle data may be acquired through modules such as a vehicle posture acquisition module and / or a vehicle signal acquisition module.

[0028] The driver status data is data that can represent the driver's status and can be obtained through the DMS camera.

[0029] The radar detection data is data detected by the radar installed on the vehicle, and may include at least one of the point cloud data detected by the radar and the list of detected radar targets, etc. The radar target list may include data such as the number, abscissa, ordinate, lateral relative speed, longitudinal relative speed, lateral acceleration, longitudinal acceleration and height of each radar target.

[0030] The visual sensor detection data is data detected by the visual sensor installed on the vehicle, and may include at least one of the data such as the video, image detected by the visual sensor, and effective visual information identified from the video and / or image. The visual sensor may be the aforementioned camera. The effective visual information may include information such as the number, horizontal coordinate, vertical coordinate, horizontal relative speed, longitudinal relative speed, width, height and category of the visual target, and may also include information such as lane line information and drivable area.

[0031] The present application can obtain the above-mentioned vehicle data, driver status data, radar detection data and visual sensor detection data in real time, or can obtain the above-mentioned vehicle data, driver status data, radar detection data and visual sensor detection data at intervals, or can obtain the above-mentioned vehicle data, driver status data, radar detection data and visual sensor detection data when relevant events are triggered, and no specific restrictions are made here.

[0032] In S202, based on at least one of the vehicle data, the driver status data, the radar detection data and the visual sensor detection data, it is determined whether it is an underreporting condition, a false alarm condition or other preset special conditions.

[0033] Missed reporting conditions include two situations: one is that the vehicle is actually in danger, but the vehicle-mounted system does not determine the danger; the other is that the visual sensor misses the situation, for example, the radar detects the radar target, but the visual sensor does not detect the visual target that can be fused with the radar target, etc. The vehicle-mounted system can identify dangers and make decisions based on the data detected by the sensors (radar and camera), and ultimately generate warnings, braking and / or steering control and other actions to avoid danger.

[0034] A false alarm condition refers to a condition in which the vehicle system determines that the vehicle is in danger, but the vehicle is not actually in danger.

[0035] The preset other special operating conditions refer to special operating conditions other than the above-mentioned underreporting conditions and the above-mentioned false alarm conditions, for example, they may include visual sensor failure conditions, special weather conditions and / or special road conditions, etc.

[0036] The embodiment of the present application can determine whether the current operating condition belongs to a missed reporting condition, a false reporting condition or other preset special operating conditions based on at least one of the aforementioned acquired vehicle data, driver status data, radar detection data and visual sensor detection data; if it does not belong to any of them, data collection and uploading to the cloud will not be performed.

[0037] In S203, if it is determined that it is an underreported working condition, a false report working condition or other preset special working conditions, the collected data corresponding to the working condition is recorded and uploaded.

[0038] When it is determined that the current working condition is an underreported working condition, a false alarm working condition, or other special working conditions, the collected data corresponding to the working condition is obtained and stored locally and uploaded to the cloud. The data stored locally and uploaded to the cloud both carry the label of the working condition.

[0039] The collected data corresponding to the working condition may include data collected by a visual sensor (ADAS camera and / or DMS camera) and / or data collected by a radar within a second preset time before the corresponding working condition starts to a third preset time after the corresponding working condition starts, etc. Exemplarily, the collected data corresponding to the working condition may include data collected by a visual sensor and / or data collected by a radar within a second preset time before a certain working condition is determined to be a certain working condition to a third preset time after a certain working condition is determined to be a certain working condition, etc. The values ​​of the second preset time and the third preset time can be set according to actual needs, and the two can be equal or unequal, for example, both can be 10 seconds, etc.

[0040] The embodiment of the present application determines whether it belongs to an underreported operating condition, a false alarm operating condition or other preset special operating conditions by obtaining at least one of the vehicle data, driver status data, radar detection data and visual sensor detection data; if it is determined to be an underreported operating condition, a false alarm operating condition or other preset special operating conditions, the collected data corresponding to the corresponding operating condition is recorded and uploaded, so that only the required valid data can be recorded in a targeted and efficient manner, without the need to record and store a large amount of data in real time, which can reduce the time required for data collection and iteration, improve data collection efficiency, and save manpower.

[0041] In some embodiments, in the above S202, judging whether it is a missed reporting condition based on at least one of the vehicle data, the driver status data, the radar detection data, and the visual sensor detection data may include: In the process of target fusion based on radar detection data and visual sensor detection data, if a first radar target exists, the confidence level of the visual sensor in target detection is reduced to obtain a first confidence level; the first radar target is a radar target for which there is no corresponding first visual target; Based on the first confidence level and the first target detection model, performing target detection by a visual sensor to obtain first visual target detection data; If there is a first visual target corresponding to the first radar target in the first visual target detection data, it is determined that it belongs to the first underreporting condition.

[0042] Target fusion is performed based on radar detection data and visual sensor detection data, specifically including the process of fusion of radar targets and visual targets at the same location based on the radar target list in the radar detection data and the relevant information about the target in the effective visual information in the visual sensor detection data. The target fusion process is the association and comparison of the parameters of multiple targets. Typical matching methods for a single frame include the nearest neighbor method, weighted bipartite graph matching algorithm, etc. The association of a single frame will set a threshold to determine the association attributes between targets. For the determined associated targets, it is generally necessary to perform an association comparison of consecutive frames. For example, the association status is determined by comparing 10 consecutive frames of data and comparing the trend curves of each parameter.

[0043] Targets on the road are divided into static targets and dynamic targets. Radar can detect static targets as well as dynamic targets. For example, it can detect targets such as guardrails, vehicles, pedestrians, lamp poles, speed bumps, etc. However, radar is more sensitive to dynamic targets and can output both dynamic and static targets. Under real road conditions, dynamic targets are generally valid targets, but whether to use static targets as valid targets can be determined based on actual needs, and no specific restrictions are made here. For the fusion of static targets, it is necessary to consider the height information of the target given by the radar and its life cycle number (such as the typical value greater than 2S) to determine whether the target is a real obstacle.

[0044] It should be noted that when performing target fusion, the corresponding radars and visual sensors are located in the same position. For example, the radar and visual sensors located in front of the vehicle perform target fusion, the radar and visual sensors located behind the vehicle perform target fusion, the radar and visual sensors located on the left side of the vehicle perform target fusion, the radar and visual sensors located on the right side of the vehicle perform target fusion, and so on.

[0045] The underreporting conditions of this embodiment are mainly judged for the underreporting conditions in the target fusion process to collect the underreporting conditions of the visual sensor.

[0046] If, for a certain radar target, no corresponding visual target is detected at the corresponding position of the radar target, the radar target is referred to as the first radar target, and the visual target is referred to as the first visual target. The radar target may be the aforementioned valid target.

[0047] If there is a first radar target, that is, the number of first radar targets is greater than or equal to 1, the confidence of the visual sensor when detecting the target can be reduced. Exemplarily, assuming that the confidence of the visual sensor when detecting the target is the initial confidence, the first confidence is obtained by reducing the initial confidence. For example, the first confidence can be obtained by subtracting the preset confidence from the initial confidence. The preset confidence can be set according to actual needs, for example, it can be 0.2 or 0.3, etc.

[0048] Target detection by a visual sensor can also be called target screening by a visual sensor, which is to screen the detected targets and take the targets whose corresponding confidence level is greater than or equal to the set confidence level threshold (for example, the aforementioned initial confidence level or the first confidence level after the confidence level is reduced) as the visual targets detected by the visual sensor.

[0049] When the visual sensor detects a target, the image it captures is input into the first target detection model. At this time, the confidence of the first target detection model is the initial confidence, that is, the target with a confidence greater than or equal to the initial confidence is regarded as the detected visual target. When the above-mentioned first radar target exists, the confidence is lowered, and the target with a confidence greater than or equal to the first confidence is regarded as the detected visual target. At this time, the detection data corresponding to the visual target detected based on the first confidence for the same image is called the first visual target detection data. Due to the reduction in confidence, the number of visual targets in the first visual target detection data should be greater than the number of visual targets detected based on the initial confidence.

[0050] The first target detection model may be any trained model that can perform target detection, for example, a YOLO model, an EfficientDet model, a Faster R-CNN model, and the like.

[0051] The first visual target detection data may include information such as the number, horizontal coordinate, vertical coordinate, horizontal relative speed, vertical relative speed, width, height and category of the visual target detected based on the first confidence level and the first target detection model.

[0052] If there is a first visual target corresponding to the first radar target in the first visual target detection data, that is, after lowering the confidence, the first visual target corresponding to the first radar target is detected, then it is determined that the current working condition belongs to the first missed report working condition. The first missed report working condition is a working condition in which the target detection accuracy of the first target detection model is low, resulting in the visual sensor missing a valid visual target. For the first missed report working condition, it is necessary to further improve the target detection accuracy of the first target detection model to improve the target confidence.

[0053] Among them, for the case where the number of first radar targets is greater than 1, if each first radar target has a corresponding first visual target in the first visual target detection data, it is determined that there is a first visual target corresponding to the first radar target in the first visual target detection data, otherwise, it is determined that there is no first visual target corresponding to the first radar target in the first visual target detection data. Alternatively, for the case where the number of first radar targets is greater than 1, if the ratio of the number of first radar targets with corresponding first visual targets to the number of all first radar targets in the first visual target detection data is greater than a preset ratio, it is determined that there is a first visual target corresponding to the first radar target in the first visual target detection data, otherwise, it is determined that there is no first visual target corresponding to the first radar target in the first visual target detection data. The preset ratio can be a larger ratio, for example, it can be 80%, 90%, and so on.

[0054] In some possible implementations, the process of detecting whether the first radar target exists includes: For the radar target in the radar detection data, determine whether the radar target is within the field of view (FOV) of the corresponding visual sensor; If the radar target is within the field of view of the corresponding visual sensor, the image coordinates corresponding to the radar target are obtained through coordinate conversion based on the coordinate data corresponding to the radar target; If the image coordinates corresponding to the radar target do not exist in the visual sensor detection data, the radar target is determined to be the first radar target; otherwise, the radar target is determined not to be the first radar target.

[0055] Among them, if the radar target is not within the field of view of the corresponding visual sensor, the program branch of false alarm collection and analysis is entered.

[0056] The coordinate data corresponding to the radar target may be the horizontal coordinate and the vertical coordinate of the radar target in the radar detection data.

[0057] This embodiment determines whether the image coordinates corresponding to the radar target exist in the visual sensor detection data by comparing the image coordinates corresponding to the radar target with the horizontal coordinates and vertical coordinates of each visual target in the visual sensor detection data.

[0058] The formula for the above coordinate transformation can include:

[0059]

[0060] ( u , v ) is the image coordinate corresponding to the radar target; , , , They are the focal length and principal point coordinates, in pixels, and are the intrinsic parameters of the visual sensor, corresponding to the intrinsic parameter matrix of the visual sensor. ; is the coordinate of the radar target in the camera coordinate system ; is the coordinate of the radar target in the world coordinate system; is the transformation matrix; .

[0061] Since it is generally necessary to transform to the world coordinate system when cooperating with other sensors, the conversion between the world coordinate system and the camera coordinate system uses external parameters, that is, the installation height, angle, etc. The world coordinate system corresponds to the camera coordinate system, and the required external parameters are finally converted into the rotation matrix and translation matrix, that is, and The vehicle is moving in real time and its posture is changing in real time. Therefore, the rotation matrix under static calibration cannot be directly used here. This embodiment uses lane lines to obtain pitch information (through the internal parameters of the visual sensor). , , , The vehicle attitude sensor is used to obtain the roll angle information, and then the rotation matrix is ​​calculated in real time to calculate the image coordinates corresponding to the radar target.

[0062] Based on the same principle, judging whether there is a first visual target corresponding to the first radar target in the first visual target detection data may include: Based on the coordinate data corresponding to the first radar target, image coordinates corresponding to the first radar target are obtained through coordinate conversion; If there are no image coordinates corresponding to the first radar target in the first visual target detection data, it is determined that there is no first visual target corresponding to the first radar target in the first visual target detection data; otherwise, it is determined that there is a first visual target corresponding to the first radar target in the first visual target detection data.

[0063] In some embodiments, the visual sensor detection data includes images; After the first visual target detection data is obtained, judging whether it is a missed reporting condition based on at least one of the vehicle data, the driver status data, the radar detection data and the visual sensor detection data, further includes: If the first visual target corresponding to the first radar target still does not exist in the first visual target detection data, obtaining coordinate data corresponding to the first radar target, and determining the position of the first visual target in the image according to the coordinate data corresponding to the first radar target; According to the position of the first visual target in the image, the image is intercepted to obtain a intercepted image; Performing target detection with a visual sensor according to the captured image and the second target detection model to obtain second visual target detection data; If there is a first visual target corresponding to the first radar target in the second visual target detection data, it is determined to belong to the second underreporting condition; If the first visual target corresponding to the first radar target still does not exist in the second visual target detection data, it is determined to belong to the third underreporting condition.

[0064] The images included in the visual sensor detection data are images taken by the visual sensor, or images converted from videos taken by the visual sensor.

[0065] The process of determining the position of the first visual target in the image according to the coordinate data corresponding to the first radar target can be implemented by referring to the process of obtaining the image coordinates corresponding to the radar target through coordinate conversion based on the coordinate data corresponding to the radar target and the aforementioned formula. That is, the coordinate data corresponding to the radar target is replaced with the coordinate data corresponding to the first radar target, and the image coordinates calculated are the position of the first visual target in the image.

[0066] According to the position of the first visual target in the image, the image is captured to obtain a captured image. This may be the image captured when the target detection of the visual sensor based on the first confidence and the first target detection model is performed to obtain the first visual target detection data. The specific capture method may be to expand the preset pixels outward with the position of the first visual target in the image as the center to obtain a captured image including the position of the first visual target in the image. Other capture methods may also be used to obtain a captured image including the position of the first visual target in the image. No specific limitation is made here. The area of ​​the captured image is smaller than the area of ​​the image before capture.

[0067] The second target detection model can be any trained model that can perform target detection, for example, it can be a YOLO model, an EfficientDet model, a Faster R-CNN model, etc. The first target detection model is a global target detection model, that is, it performs target detection on the entire frame image; the second target detection model is a local target detection model, which performs target detection based on the position of the visual target in the image, that is, it performs target detection on a specific area (local area) of the entire frame image. The architecture of the second target detection model can be the same as or different from that of the first target detection model, and no specific restrictions are made here.

[0068] The captured image is input into the second target detection model to perform target detection by the visual sensor to obtain second visual target detection data. At this time, the first confidence level is still used for detection.

[0069] The second visual target detection data may include information such as the number, horizontal coordinate, vertical coordinate, horizontal relative speed, vertical relative speed, width, height and category of the visual target obtained by performing target detection on the captured image using the second target detection model.

[0070] The second underreporting condition is a condition where the first target detection model fails to detect the target, while the second target detection model detects the target effectively, that is, the global target detection model fails, while the local target detection model is effective. When it belongs to the second underreporting condition, it means that the target detection accuracy of the first target detection model is low, and it is necessary to supplement the training data in a targeted manner to train the first target detection model and improve its target detection accuracy.

[0071] The third underreporting condition is a condition where both the first target detection model and the second target detection model fail. The collected data corresponding to this condition is the most valuable data. In practical applications, the collected data corresponding to this condition needs to be cleaned and further confirmed based on the cleaned data. If there is no valid target in the cleaned data, it means that the radar has made a false alarm, and it is necessary to accumulate data to establish a corresponding question library and optimize the radar; if there is a valid target in the cleaned data, it is necessary to further train the first target detection model and the second target detection model to improve the accuracy of target detection by both.

[0072] The first underreporting condition, the second underreporting condition and the third underreporting condition may all be underreporting conditions of the visual sensor, that is, during the target fusion process, the radar detects a valid target, but the visual sensor does not detect a valid target, and so on.

[0073] In an embodiment of the present application, when the first visual target corresponding to the first radar target still does not exist in the first visual target detection data, a partial image (i.e., the captured image) is captured based on the position of the first visual target in the image, and target detection is performed based on the partial image and the second target detection model to obtain second visual target detection data. In the second visual target detection data, if the first visual target corresponding to the first radar target exists, it is determined that the data belongs to the second underreporting condition; if the first visual target corresponding to the first radar target still does not exist, it is determined that the data belongs to the third underreporting condition.

[0074] The embodiment of the present application can collect targeted data accordingly by distinguishing the first underreporting condition, the second underreporting condition and the third underreporting condition, so as to facilitate targeted optimization of the target detection model corresponding to the visual sensor or the radar.

[0075] In some embodiments, the vehicle data includes at least one of vehicle speed, deceleration, three-axis acceleration, three-axis angular velocity, and steering wheel speed; Based on at least one of the vehicle data, the driver status data, the radar detection data and the visual sensor detection data, it is determined whether it is an underreporting condition, including: If the vehicle speed, deceleration, three-axis acceleration, three-axis angular velocity and steering wheel speed meet the preset vehicle drastic change conditions, it is determined to belong to the fourth underreporting condition.

[0076] In the embodiment of the present application, in addition to the above-mentioned visual sensor underreporting condition, the underreporting condition may also include the underreporting condition that the vehicle is actually in danger but the on-board system has not determined the danger, that is, the fourth underreporting condition.

[0077] If the vehicle system can promptly identify the dangerous situation, the vehicle system will promptly take corresponding measures to prevent the vehicle from being in danger and prevent the vehicle from undergoing drastic changes. Therefore, the embodiment of the present application determines whether it belongs to the fourth underreporting condition by determining whether the vehicle undergoes drastic changes.

[0078] Specifically, the above-mentioned preset vehicle drastic change conditions may include a preset vehicle speed drastic reduction condition, a preset vehicle deceleration excessive condition, a preset vehicle posture abnormal condition and / or a preset steering wheel speed excessive condition, etc.

[0079] The preset vehicle speed rapid reduction condition may include the vehicle speed reducing by a preset speed within a fourth preset time period, for example, reducing by 10 km / h within 1 second, and so on.

[0080] The preset vehicle deceleration excessive condition may include the vehicle deceleration being greater than a preset deceleration threshold. For example, the preset deceleration threshold may be 3 m / s 2 ,etc.

[0081] The preset vehicle posture abnormal condition may include that the vehicle tilt angle determined based on the three-axis acceleration and three-axis angular velocity of the vehicle is greater than a preset angle threshold. For example, the preset angle threshold may be 30 degrees, and so on.

[0082] The preset steering wheel speed too fast condition may include the steering wheel speed being greater than a preset speed threshold.

[0083] The values ​​of the fourth preset duration, preset speed, preset deceleration threshold, preset angle threshold and preset speed threshold can be set according to actual needs and are not specifically limited here.

[0084] That is, the above-mentioned vehicle drastic change conditions may include at least one of the vehicle speed decreasing by a preset speed within a fourth preset time period, the vehicle's deceleration being greater than a preset deceleration threshold, the vehicle's inclination angle determined based on the vehicle's three-axis acceleration and three-axis angular velocity being greater than a preset angle threshold, and the steering wheel speed being greater than a preset speed threshold.

[0085] When it is detected that the vehicle speed decreases from a preset speed within a fourth preset time period, the absolute value of the vehicle's deceleration is greater than a preset deceleration threshold, the vehicle's inclination angle determined based on the vehicle's three-axis acceleration and three-axis angular velocity is greater than a preset angle threshold, and the steering wheel speed is greater than a preset speed threshold, it is determined to belong to the fourth underreporting condition.

[0086] Exemplarily, when it is detected that the vehicle speed decreases by a preset speed within a fourth preset time period and the vehicle's deceleration is greater than a preset deceleration threshold, it is determined to be a fourth missed report condition; when it is detected that the vehicle's tilt angle is greater than a preset angle threshold, it is determined to be a fourth missed report condition; when it is detected that the steering wheel speed is greater than a preset speed threshold and the vehicle speed decreases by a preset speed within a fourth preset time period, it is determined to be a fourth missed report condition; and so on.

[0087] After acquiring the collected data corresponding to the fourth underreporting condition, the hazard determination method of the vehicle-mounted system can be optimized based on the data.

[0088] In some embodiments, based on at least one of vehicle data, driver status data, radar detection data, and visual sensor detection data, determining whether it belongs to other preset special operating conditions includes: Based on the detection data of the visual sensor, determine the farthest distance of the lane line and the farthest distance of the visual target, and determine the visual visibility according to the farthest distance of the lane line and the farthest distance of the visual target; Predicting the drivable area of ​​the vehicle based on the visual sensor detection data, and determining the farthest end distance and lateral range span of the drivable area based on the drivable area of ​​the vehicle; Determine the brightness of the image captured by the visual sensor based on the visual sensor detection data; Based on the radar detection data and the visual sensor detection data, determining whether there is a first radar target; the first radar target is a radar target for which there is no corresponding first visual target; If the first radar target is present, determining the number of consecutive images for which there is no corresponding first visual target; If the visual visibility, the farthest distance of the drivable area, the lateral range span of the drivable area, the brightness of the image and the number of consecutive images meet the preset visual sensor failure conditions, it is determined to belong to other preset special working conditions.

[0089] The visual sensor detection data may include captured images and / or videos, and relevant information of lane lines detected based on the captured images and / or videos, such as the location information, distance information, etc. of the lane lines, and the location coordinates of visual targets detected based on the captured images and / or videos, etc.

[0090] In the embodiment of the present application, the distance between the farthest point of the lane line and the vehicle is referred to as the farthest distance of the lane line. The distance between the visual target and the vehicle can be determined based on information such as the position coordinates of the visual target. The maximum distance between each visual target and the vehicle is taken as the farthest distance of the visual target. The larger value between the farthest distance of the visual target and the farthest distance of the lane line is taken as the visual visibility.

[0091] The embodiment of the present application can also predict the vehicle's drivable area based on visual sensor detection data and a trained semantic segmentation network model.

[0092] The above-mentioned determination of the farthest distance and lateral range span of the drivable area based on the drivable area of ​​the vehicle may include: extracting a boundary sequence of the drivable area of ​​the vehicle (which may include multiple boundary points, point 1 to point n); calculating the world coordinates of each boundary point based on the internal and external parameters of the visual sensor; based on the world coordinates of each boundary point, taking the distance between the boundary point farthest from the vehicle and the vehicle as the farthest distance of the drivable area; based on the world coordinates of each boundary point, taking the difference in the horizontal coordinates of the two boundary points with the largest difference in the horizontal coordinates as the lateral range span of the drivable area.

[0093] The image captured by the visual sensor can be obtained from the visual sensor detection data, and the brightness of the image can be determined. If the image brightness is very high, such as above 180, it means that the image may be affected by interference sources such as backlight. If the image brightness is very low, such as below 50, it means that the image brightness is very low, and the detection effect of the visual sensor is likely to be affected.

[0094] Based on the relevant methods in the aforementioned embodiments, it is possible to determine whether the first radar target exists based on the radar detection data and the visual sensor detection data, and determine the number of consecutive images (number of frames) in which the first radar target exists, that is, the number of consecutive images of the first visual target that do not correspond to the first radar target. For example, if the visual target detected by the 5 consecutive frames of images taken by the visual sensor is fused with the radar target detected at the same time, the first radar target exists, but the visual target detected by the 6th frame of the image is fused with the radar target detected at the same time, the first radar target does not exist, then the number of consecutive images in which the corresponding first visual target does not exist is determined to be 5.

[0095] In this embodiment, if the visual visibility, the farthest distance of the drivable area, the lateral range span of the drivable area, the brightness of the image and the number of consecutive images meet the preset visual sensor failure conditions, it is determined to belong to other preset special working conditions.

[0096] In some embodiments, other special working conditions are preset including visual sensor failure conditions; The preset sensor failure conditions include at least one of the following: visual visibility is less than a preset visibility threshold, the farthest end distance of the drivable area is less than a preset distance threshold, the lateral range span of the drivable area is less than a preset span threshold, the brightness of the image exceeds a preset normal brightness range, and the number of consecutive images is greater than a preset number.

[0097] The values ​​of the preset visibility threshold, the preset distance threshold, the preset span threshold, the preset normal brightness range and the preset number can be set according to actual needs and are not specifically limited here.

[0098] Among them, at least one of the preset visibility threshold, the preset distance threshold, the preset span threshold, the preset normal brightness range and the preset number can be associated with the current vehicle speed, the current weather and / or the current time.

[0099] If the current vehicle speed is high, that is, the vehicle is traveling at a high speed, it means that the driver's field of vision is acceptable and the visibility is high. At this time, the preset visibility threshold, preset distance threshold, preset span threshold and preset normal brightness range can be increased accordingly, and the preset number can be reduced accordingly, that is, the preset visibility threshold, preset distance threshold, preset span threshold and preset normal brightness range are positively correlated with the current vehicle speed, and the preset number is negatively correlated with the current vehicle speed.

[0100] If the current weather is bad or the current time is night, the visibility will be reduced, and accordingly, the preset visibility threshold, preset distance threshold, preset span threshold and preset normal brightness range can be reduced accordingly, and the preset number can be increased; otherwise, the preset visibility threshold, preset distance threshold, preset span threshold and preset normal brightness range can be increased accordingly, and the preset number can be reduced.

[0101] The visual sensor failure condition may be a condition in which the visual sensor fails due to a fault or interference from other factors (eg, strong light, bad weather, etc.).

[0102] In some possible implementations, grading thresholds can be set for a preset visibility threshold, a preset distance threshold, a preset span threshold, a preset normal brightness range, and a preset number, and the failure level of the visual sensor is determined based on the different grading ranges of the above-mentioned visual visibility, the farthest distance of the drivable area, the lateral range span of the drivable area, the brightness of the image, and the number of continuous images.

[0103] In some possible implementations, the preset other special working conditions may also include special weather working conditions and / or special road working conditions. The special weather working condition may be a working condition corresponding to the preset special weather, such as heavy rain, strong wind, typhoon, hail, heavy snow and other special weather. The special road working condition may be a working condition corresponding to the preset special road, such as special roads such as snow, mud, sand and the like.

[0104] In some embodiments, judging whether it is a false alarm condition based on at least one of vehicle data, driver status data, radar detection data, and visual sensor detection data includes: Determine whether danger occurs based on radar detection data and visual sensor detection data; If it is determined that a danger has occurred, danger avoidance action will be performed; Determine whether the vehicle data and driver status data within a first preset time period meet the preset false alarm conditions; if so, determine that it is a false alarm condition; the first preset time period starts at a first time point and ends at a second time point, the first time point is the time point when the danger avoidance action starts, the second time point is after the time point when the danger avoidance action ends, and is separated from the time point when the danger avoidance action ends by a first preset time length.

[0105] In some embodiments, the vehicle data includes brake signals and steering wheel signals; The preset false alarm conditions include that no brake signal is generated within a first preset time period, no change in the steering wheel signal within the first preset time period, and no change in the driver status data within the first preset time period.

[0106] The above-mentioned judging whether a danger occurs based on the radar detection data and the visual sensor detection data may include: performing target fusion based on the radar detection data and the visual sensor detection data to obtain a fused target; judging whether there are obstacles that affect driving on the driving route based on the fused target; if so, judging that a danger occurs, otherwise, judging that no danger occurs.

[0107] The danger avoidance action may include a steering avoidance action and / or a braking action. The vehicle-mounted system can determine whether a danger will occur during driving based on the above process. When a danger is determined to occur, the vehicle system will make a corresponding decision and ultimately generate actions such as early warning, braking and / or steering control.

[0108] The embodiment of the present application monitors the vehicle and driver behaviors during and within a certain period of time after the vehicle-mounted system performs a danger avoidance action, and performs analysis to determine whether it is a false alarm condition. Normally, during or after the vehicle-mounted system performs a danger avoidance action, the driver will brake, turn the steering wheel, and / or change the driver's state (for example, from a distracted state to an undistracted state, from a fatigued state to a non-fatigued state, etc.). When none of the above behaviors occur, it is determined to be a false alarm condition.

[0109] The fact that no brake signal is generated within the first preset time period indicates that the driver has not stepped on the brake pedal within the first preset time period.

[0110] The steering wheel signal is used to represent the angle of the steering wheel. If the steering wheel signal does not change, it means that the steering wheel angle does not change.

[0111] The driver status data is used to characterize the driver's status. If the driver status data does not change within the first preset time period, it means that the driver's status does not change within the first preset time period.

[0112] In some possible implementations, after the collected data corresponding to the working condition is recorded and uploaded, it can be manually reviewed to determine whether the collected data belongs to the corresponding working condition.

[0113] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0114] The following is an embodiment of the device of the present invention. For details not described in detail therein, reference may be made to the corresponding method embodiment described above.

[0115] Figure 3 The following is a schematic diagram showing the structure of a data acquisition device provided by an embodiment of the present invention. For ease of explanation, only the parts related to the embodiment of the present invention are shown, which are described in detail as follows: like Figure 3 As shown, the data acquisition device 30 may include: an acquisition module 31 , a working condition judgment module 32 and a data acquisition module 33 .

[0116] An acquisition module 31, used to acquire vehicle data, driver status data, radar detection data and visual sensor detection data; The working condition judgment module 32 is used to judge whether it is a missed working condition, a false working condition or other preset special working conditions based on at least one of the vehicle data, the driver status data, the radar detection data and the visual sensor detection data; The data collection module 33 is used to record and upload the collected data corresponding to the working condition if it is determined to be an underreported working condition, a false report working condition or other preset special working conditions.

[0117] In a possible implementation, in the working condition determination module 32, based on at least one of the vehicle data, the driver status data, the radar detection data, and the visual sensor detection data, it is determined whether it is an underreported working condition, including: In the process of target fusion based on radar detection data and visual sensor detection data, if a first radar target exists, the confidence level of the visual sensor in target detection is reduced to obtain a first confidence level; the first radar target is a radar target for which there is no corresponding first visual target; Based on the first confidence level and the first target detection model, performing target detection by a visual sensor to obtain first visual target detection data; If there is a first visual target corresponding to the first radar target in the first visual target detection data, it is determined that it belongs to the first underreporting condition.

[0118] In a possible implementation, in the working condition determination module 32, the visual sensor detection data includes an image; After obtaining the first visual target detection data, judging whether it is an underreporting condition based on at least one of the vehicle data, the driver state data, the radar detection data, and the visual sensor detection data, further includes: If the first visual target corresponding to the first radar target still does not exist in the first visual target detection data, obtaining coordinate data corresponding to the first radar target, and determining the position of the first visual target in the image according to the coordinate data corresponding to the first radar target; According to the position of the first visual target in the image, the image is intercepted to obtain a intercepted image; Performing target detection with a visual sensor according to the captured image and the second target detection model to obtain second visual target detection data; If there is a first visual target corresponding to the first radar target in the second visual target detection data, it is determined to belong to the second underreporting condition; If the first visual target corresponding to the first radar target still does not exist in the second visual target detection data, it is determined to belong to the third underreporting condition.

[0119] In a possible implementation, in the working condition determination module 32, the vehicle data includes at least one of vehicle speed, deceleration, three-axis acceleration, three-axis angular velocity, and steering wheel speed; Based on at least one of the vehicle data, the driver status data, the radar detection data and the visual sensor detection data, it is determined whether it is an underreporting condition, including: If the vehicle speed, deceleration, three-axis acceleration, three-axis angular velocity and steering wheel speed meet the preset vehicle drastic change conditions, it is determined to belong to the fourth underreporting condition.

[0120] In a possible implementation, in the working condition determination module 32, based on at least one of the vehicle data, the driver status data, the radar detection data, and the visual sensor detection data, it is determined whether it belongs to other preset special working conditions, including: Based on the detection data of the visual sensor, determine the farthest distance of the lane line and the farthest distance of the visual target, and determine the visual visibility according to the farthest distance of the lane line and the farthest distance of the visual target; Predicting the drivable area of ​​the vehicle based on the visual sensor detection data, and determining the farthest end distance and lateral range span of the drivable area based on the drivable area of ​​the vehicle; Determine the brightness of the image captured by the visual sensor based on the visual sensor detection data; Based on the radar detection data and the visual sensor detection data, determining whether there is a first radar target; the first radar target is a radar target for which there is no corresponding first visual target; If the first radar target is present, determining the number of consecutive images for which there is no corresponding first visual target; If the visual visibility, the farthest distance of the drivable area, the lateral range span of the drivable area, the brightness of the image and the number of consecutive images meet the preset visual sensor failure conditions, it is determined to belong to other preset special working conditions.

[0121] In a possible implementation, other special working conditions are preset including a visual sensor failure working condition; The preset sensor failure conditions include at least one of the following: visual visibility is less than a preset visibility threshold, the farthest end distance of the drivable area is less than a preset distance threshold, the lateral range span of the drivable area is less than a preset span threshold, the brightness of the image exceeds a preset normal brightness range, and the number of consecutive images is greater than a preset number.

[0122] In a possible implementation, in the working condition determination module 32, based on at least one of the vehicle data, the driver status data, the radar detection data, and the visual sensor detection data, it is determined whether it is a false alarm working condition, including: Determine whether danger occurs based on radar detection data and visual sensor detection data; If it is determined that a danger has occurred, danger avoidance action will be performed; Determine whether the vehicle data and driver status data within a first preset time period meet the preset false alarm conditions; if so, determine that it is a false alarm condition; the first preset time period starts at a first time point and ends at a second time point, the first time point is the time point when the danger avoidance action starts, the second time point is after the time point when the danger avoidance action ends, and is separated from the time point when the danger avoidance action ends by a first preset time length.

[0123] In one possible implementation, the vehicle data includes a brake signal and a steering wheel signal; The preset false alarm conditions include that no brake signal is generated within a first preset time period, no change in the steering wheel signal within the first preset time period, and no change in the driver status data within the first preset time period.

[0124] Figure 4 Schematic diagram of an electronic device provided by an embodiment of the present invention. Figure 4 As shown, the electronic device 4 of this embodiment includes: a processor 40 and a memory 41. The memory 41 is used to store a computer program 42, and the processor 40 is used to call and run the computer program 42 stored in the memory 41 to perform the steps in the above-mentioned various data collection method embodiments, such as Figure 2 Alternatively, the processor 40 is used to call and run the computer program 42 stored in the memory 41 to implement the functions of each module / unit in the above-mentioned device embodiments, for example Figure 3 The functions of each module are shown.

[0125] Exemplarily, the computer program 42 may be divided into one or more modules / units, which are stored in the memory 41 and executed by the processor 40 to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, which are used to describe the execution process of the computer program 42 in the electronic device 4. For example, the computer program 42 may be divided into Figure 3 The modules shown.

[0126] The electronic device 4 may be a computing device such as a computer or a server, or may be a device such as an ECU (Electronic Control Unit) on a vehicle. The electronic device 4 may include, but is not limited to, a processor 40 and a memory 41. Those skilled in the art may understand that Figure 4It is only an example of the electronic device 4 and does not constitute a limitation of the electronic device 4. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0127] The processor 40 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0128] The memory 41 may be an internal storage unit of the electronic device 4, such as a hard disk or memory of the electronic device 4. The memory 41 may also be an external storage device of the electronic device 4, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 4. Further, the memory 41 may also include both an internal storage unit and an external storage device of the electronic device 4. The memory 41 is used to store the computer program and other programs and data required by the electronic device. The memory 41 may also be used to temporarily store data that has been output or is to be output.

[0129] Corresponding to the above-mentioned electronic device, an embodiment of the present application further provides a vehicle, comprising the above-mentioned electronic device.

[0130] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0131] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0132] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0133] In the embodiments provided by the present invention, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0134] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0135] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0136] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned data acquisition method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device that can carry the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium.

[0137] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A data collection method, characterized in that: include: Acquire vehicle data, driver status data, radar detection data, and visual sensor detection data; Based on at least one of the vehicle data, the driver status data, the radar detection data, and the visual sensor detection data, determining whether it is a missed reporting condition, a false reporting condition, or other preset special conditions; If it is determined to be the underreported working condition, the false alarm working condition or the other preset special working condition, the collected data corresponding to the working condition is recorded and uploaded.

2. The data collection method according to claim 1, characterized in that: Based on at least one of the vehicle data, the driver status data, the radar detection data, and the visual sensor detection data, determining whether it is an underreporting condition includes: In the process of target fusion based on the radar detection data and the visual sensor detection data, if there is a first radar target, the confidence level of the visual sensor in target detection is reduced to obtain a first confidence level; the first radar target is a radar target for which there is no corresponding first visual target; Based on the first confidence level and the first target detection model, performing target detection by a visual sensor to obtain first visual target detection data; If a first visual target corresponding to the first radar target exists in the first visual target detection data, it is determined that the first underreporting condition exists.

3. The data collection method according to claim 2, characterized in that: The visual sensor detection data includes images; After obtaining the first visual target detection data, judging whether it is an underreporting condition based on at least one of the vehicle data, the driver state data, the radar detection data, and the visual sensor detection data, further includes: If the first visual target corresponding to the first radar target still does not exist in the first visual target detection data, obtaining coordinate data corresponding to the first radar target, and determining the position of the first visual target in the image according to the coordinate data corresponding to the first radar target; According to the position of the first visual target in the image, performing image interception to obtain a intercepted image; Performing target detection with a visual sensor according to the captured image and the second target detection model to obtain second visual target detection data; If a first visual target corresponding to the first radar target exists in the second visual target detection data, it is determined to belong to a second underreporting condition; If the first visual target corresponding to the first radar target still does not exist in the second visual target detection data, it is determined that it belongs to the third underreporting condition.

4. The data collection method according to claim 1, characterized in that: The vehicle data includes at least one of vehicle speed, deceleration, three-axis acceleration, three-axis angular velocity and steering wheel speed; Based on at least one of the vehicle data, the driver status data, the radar detection data, and the visual sensor detection data, determining whether it is an underreporting condition includes: If the vehicle speed, the deceleration, the three-axis acceleration, the three-axis angular velocity and the steering wheel speed meet the preset vehicle drastic change condition, it is determined to belong to the fourth underreporting condition.

5. The data collection method according to claim 1, characterized in that: Based on at least one of the vehicle data, the driver status data, the radar detection data, and the visual sensor detection data, determining whether it belongs to other preset special working conditions includes: Determine the farthest distance of the lane line and the farthest distance of the visual target based on the detection data of the visual sensor, and determine the visual visibility according to the farthest distance of the lane line and the farthest distance of the visual target; Predicting a drivable area of ​​the vehicle based on the visual sensor detection data, and determining a farthest end distance and a lateral range span of the drivable area based on the drivable area of ​​the vehicle; Determining the brightness of an image captured by the visual sensor based on the visual sensor detection data; Based on the radar detection data and the visual sensor detection data, determining whether there is a first radar target; the first radar target is a radar target for which there is no corresponding first visual target; If the first radar target exists, determining the number of consecutive images for which there is no corresponding first visual target; If the visual visibility, the farthest distance of the drivable area, the lateral range span of the drivable area, the brightness of the image and the number of consecutive images meet the preset visual sensor failure conditions, it is determined to belong to other preset special operating conditions.

6. The data collection method according to claim 5, characterized in that: The preset other special working conditions include visual sensor failure conditions; The preset sensor failure conditions include at least one of the following: the visual visibility is less than a preset visibility threshold, the farthest end distance of the drivable area is less than a preset distance threshold, the lateral range span of the drivable area is less than a preset span threshold, the brightness of the image exceeds a preset normal brightness range, and the number of continuous images is greater than a preset number.

7. The data collection method according to any one of claims 1 to 6, characterized in that: Based on at least one of the vehicle data, the driver status data, the radar detection data, and the visual sensor detection data, determining whether it is a false alarm condition includes: Determining whether danger occurs according to the radar detection data and the visual sensor detection data; If it is determined that a danger has occurred, danger avoidance action will be performed; Determine whether the vehicle data and driver status data within a first preset time period meet a preset false alarm condition; if so, determine that it belongs to a false alarm condition; the first preset time period starts at a first time point and ends at a second time point, the first time point is the time point when the danger avoidance action starts, the second time point is after the time point when the danger avoidance action ends, and is separated from the time point when the danger avoidance action ends by a first preset time length.

8. The data collection method according to claim 7, characterized in that: The vehicle data includes a brake signal and a steering wheel signal; The preset false alarm condition includes that the brake signal is not generated within the first preset time period, the steering wheel signal does not change within the first preset time period, and the driver status data does not change within the first preset time period.

9. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the data collection method according to any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the data collection method according to any one of claims 1 to 8 are implemented.

Images