Long-tail scene recognition method and device based on vehicle supervision data, medium and product

Abstract

The embodiment of the invention relates to the technical field of image processing, and discloses a long-tail scene recognition method and device based on vehicle supervision data, a medium and a product. The method comprises the following steps: acquiring monitoring data acquired by a vehicle-mounted monitoring terminal; wherein the monitoring data comprises structured CAN (Controller Area Network) data and image data; when the event that the vehicle is separated from the automatic driving is identified, determining the monitoring data in a reference time period of the automatic driving separation time point as identification data; and inputting the identification data into a pre-trained long-tail scene identification model, and identifying whether the vehicle is in a long-tail scene based on the long-tail scene identification model. By adopting the scheme, the accuracy and efficiency of long-tail scene recognition are improved, the labor cost is saved, batch output can be realized, and the real-time performance of a long-tail scene recognition result is improved.

Description

Technical Field

[0001] The present application relates to the field of vehicle autonomous driving technology, and in particular to a long-tail scene recognition method, device, medium and product based on vehicle supervision data. Background Art

[0002] In recent years, with the rapid development of science and technology, the fields of intelligent driving and unmanned driving of vehicles have developed rapidly. Long-tail scenarios refer to situations with low frequency but high potential impact or risk in the field of autonomous driving. Their uncertainty and high risk are common reasons for the disengagement of autonomous driving in driverless intelligent connected vehicles during open road testing. For third-party regulators, identifying long-tail scenarios is conducive to evaluating the technical level of the tested object, discovering common technical problems in the industry, and proposing road traffic risk improvement measures to the competent authorities. At present, the main methods for identifying long-tail scenarios are: test simulation summary of open source data sets such as KITTI and City Scapes; case summary of on-road testing of a single car company. However, due to the difference in the sensitivity of vehicle-side perception data, it is difficult for third-party regulators to obtain image data from on-board sensors, and identifying long-tail scenarios from regulatory image data has become another path. However, manual methods have low accuracy and are time-consuming and labor-intensive in identifying long-tail scene categories from massive regulatory image data. Therefore, how to accurately and efficiently identify long-tail scenarios has become a technical problem that needs to be solved in this field. Summary of the invention

[0003] One purpose of the present application is to provide a long-tail scene recognition method, device, medium and product based on vehicle supervision data, at least to solve the problems of poor accuracy and low recognition efficiency of traditional long-tail scene recognition. The present application obtains monitoring data collected by the on-board monitoring terminal; when it is identified that the vehicle has left the automatic driving event, the structured CAN data and image data within the reference time period are input into a pre-trained long-tail scene recognition model, and the vehicle is identified based on the long-tail scene recognition model to determine whether it is in a long-tail scene. By adopting this solution, the accuracy and efficiency of long-tail scene recognition can be improved, labor costs can be saved, and batch output can be achieved to improve the real-time performance of long-tail scene recognition results.

[0004] To achieve the above objectives, some embodiments of the present application provide the following aspects:

[0005] In a first aspect, some embodiments of the present application further provide a long-tail scene recognition method based on vehicle supervision data, including:

[0006] Acquire monitoring data collected by the vehicle-mounted monitoring terminal; wherein the monitoring data includes structured CAN data and image data;

[0007] When an event of the vehicle disengaging from the autonomous driving is identified, monitoring data within a reference period at the time point of the autonomous driving disengagement is determined as identification data;

[0008] The recognition data is input into a pre-trained long-tail scene recognition model, and whether the vehicle is in a long-tail scene is recognized based on the long-tail scene recognition model.

[0009] In a second aspect, some embodiments of the present application further provide an electronic device, comprising: one or more processors; and a memory storing computer program instructions, wherein the computer program instructions, when executed, cause the processor to perform the steps of the method described above.

[0010] In a third aspect, some embodiments of the present application further provide a computer-readable medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the method as described above.

[0011] In a fourth aspect, some embodiments of the present application further provide a computer program product, comprising a computer program / instruction, which implements the steps of the method described above when executed by a processor.

[0012] Compared with the related art, the solution provided in the embodiment of the present application obtains the monitoring data collected by the vehicle monitoring terminal; wherein the monitoring data includes structured CAN data and image data; when the vehicle is identified to have disengaged from the autonomous driving event, the monitoring data within the reference period of the autonomous driving disengagement time point is determined as the identification data; the identification data is input into the pre-trained long-tail scene recognition model, and the vehicle is identified as being in a long-tail scene based on the long-tail scene recognition model. By adopting this solution, the accuracy and efficiency of long-tail scene recognition can be improved, labor costs can be saved, and batch output can be performed to improve the real-time performance of long-tail scene recognition results. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0014] Figure 1 An exemplary flow chart of a long-tail scene recognition method based on vehicle supervision data provided according to some embodiments of the present application;

[0015] Figure 2 An exemplary flow chart of a training process of a long-tail scene recognition model provided according to some embodiments of the present application;

[0016] Figure 3 A schematic diagram of an automated long-tail scene type recognition system based on vehicle supervision data provided according to some embodiments of the present application;

[0017] Figure 4 A flowchart of an automated long-tail scene type identification method based on vehicle supervision data provided according to some embodiments of the present application;

[0018] Figure 5 An exemplary structural diagram of the electronic device is disclosed. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0020] First embodiment

[0021] The first embodiment of the present application relates to a long-tail scene recognition method based on vehicle supervision data. Figure 1 As shown, the method may include the following steps:

[0022] Step S101, acquiring monitoring data collected by a vehicle-mounted monitoring terminal; wherein the monitoring data includes structured CAN data and image data;

[0023] The vehicle-mounted monitoring terminal can be a terminal device on the vehicle for obtaining monitoring data. It can obtain data from the vehicle's CAN bus, such as structured CAN data, and can also obtain image data, such as obtaining image data outside the vehicle through a camera above the vehicle, or it can also obtain image data inside the vehicle through a camera inside the vehicle.

[0024] In this solution, the structured CAN data mainly includes vehicle speed, acceleration, latitude and longitude, heading angle, lighting, vehicle control mode, etc. The image data can be color data or infrared data, which can be determined according to the relevant parameters of the camera.

[0025] Step S102, when an event of vehicle disengagement from the automatic driving is identified, monitoring data within a reference period of the automatic driving disengagement time point is determined as identification data;

[0026] When a vehicle is in the process of disengaging from autonomous driving, it can be determined as a vehicle disengaging from autonomous driving. There are many types of disengagement from autonomous driving, such as disengagement due to manual intervention or automatic disengagement due to a long-tail driving scenario.

[0027] In this solution, after the event is identified, the time point when the autonomous driving is disengaged can be recorded, and the monitoring data within the reference period can be determined as the identification data.

[0028] Among them, the time point when autonomous driving is disengaged is, for example, 10:00:00 am, then the reference time period can be 9:58:00 am - 10:00:00 am, or 9:58:00 am - 10:02:00 am, or can be a longer time range spanning forwards and backwards.

[0029] Step S103: input the recognition data into a pre-trained long-tail scene recognition model, and identify whether the vehicle is in a long-tail scene based on the long-tail scene recognition model.

[0030] After the recognition data is determined, it can be recognized based on a pre-trained long-tail scene recognition model and a corresponding recognition result can be output, such as outputting that the vehicle is in a long-tail scene or that the vehicle is not in a long-tail scene.

[0031] The solution provided in this embodiment obtains monitoring data collected by the vehicle monitoring terminal; wherein the monitoring data includes structured CAN data and image data; when the vehicle is identified to have disengaged from the autonomous driving event, the monitoring data within the reference period of the autonomous driving disengagement time point is determined as the identification data; the identification data is input into the pre-trained long-tail scene recognition model, and the vehicle is identified based on the long-tail scene recognition model as to whether it is in a long-tail scene. By adopting this solution, the accuracy and efficiency of long-tail scene recognition can be improved, labor costs can be saved, and batch output can be achieved to improve the real-time performance of long-tail scene recognition results.

[0032] Second embodiment

[0033] The second embodiment of the present application relates to a long-tail scene recognition method based on vehicle supervision data. The above embodiment is specifically optimized as follows: the training process of the long-tail scene recognition model includes: obtaining monitoring data collected by the vehicle-mounted monitoring terminal; wherein the monitoring data includes structured CAN data and image data; preprocessing the monitoring data to obtain preprocessing results; after identifying the vehicle's departure from the autonomous driving event, determining the reference time period of the autonomous driving departure time point from the preprocessing results; obtaining the preprocessing results within the reference time period as training data, and labeling the training data based on the actual vehicle scene; supervised training of the long-tail scene recognition model based on the labeled training data, and obtaining the long-tail scene recognition model after the training is completed.

[0034] like Figure 2 As shown, the method may include the following steps:

[0035] Step S201, acquiring monitoring data collected by the vehicle-mounted monitoring terminal; wherein the monitoring data includes structured CAN data and image data;

[0036] Specifically, long-tail scene data can be collected through the on-board monitoring terminal. The collected data mainly includes: structured CAN data, which mainly contains vehicle speed, acceleration, longitude and latitude, heading angle, lights, vehicle control mode and other information; image data.

[0037] Step S202, preprocessing the monitoring data to obtain a preprocessing result;

[0038] Among them, the preprocessing of vehicle monitoring data is an important step to ensure data quality and availability. Preprocessing methods include data cleaning, such as removing outliers, checking obvious outliers in the data, such as data points with extremely high or low speeds, unreasonable positions, etc. Outliers can be identified by setting a reasonable threshold range and deleted or specially processed. For example, if the normal speed range of a vehicle is between 0 and 120 km / h, then values ​​outside this range may be outliers. Processing missing values, check whether there are missing values ​​in the data. If there are few missing values, you can consider using interpolation methods (such as linear interpolation, mean interpolation, etc.) to fill in the missing values. If there are many missing values, more complex processing can be performed according to the specific situation, such as using a regression model to predict missing values ​​or directly deleting records containing a large number of missing values. Preprocessing methods also include data standardization, such as unit unification, ensuring that the units of all data are consistent. For example, the speed unit is unified as kilometers per hour, and the time unit is unified as seconds. This facilitates subsequent data analysis and processing. Data format standardization, check whether the format of the data is consistent, such as date and time format, coordinate format, etc. Unify the data format into a standard format to facilitate data reading and processing. Preprocessing methods also include data compression and time series compression. For time series data, appropriate compression algorithms can be used to reduce the amount of data without losing important information. For example, data from adjacent time points can be compared, and if the changes are small, the data from these time points can be merged. Spatial data compression: For vehicle location data, spatial compression algorithms such as the Douglas-Peucker algorithm can be used to reduce data storage space. It also includes data conversion, such as coordinate conversion. If vehicle monitoring data uses different coordinate systems, coordinate conversion is required to unify them into a standard coordinate system. This makes it easy to perform geospatial analysis. Data type conversion: convert data types as needed. For example, convert string type time data to timestamp format for time-related calculations.

[0039] In this embodiment, optionally, the monitoring data is preprocessed to obtain a preprocessing result, including:

[0040] The monitoring data is subjected to time synchronization processing and time serialization processing to obtain a preprocessing result; wherein, in the preprocessing result, each image data is associated with at least one structured CAN data of vehicle speed, acceleration, longitude and latitude, light, and vehicle control mode.

[0041] Specifically, vehicle speed reflects the distance traveled by the vehicle in a unit of time. It is one of the important indicators for measuring the driving status of the vehicle. A higher speed may mean that the vehicle is driving fast, requiring more reaction time and braking distance. Speed ​​data can be used to analyze the driving efficiency of the vehicle, evaluate the impact of road conditions on vehicle driving, and monitor whether the vehicle is speeding. Acceleration indicates how fast the vehicle speed changes. Positive acceleration means that the vehicle is accelerating, and negative acceleration means that the vehicle is decelerating. Acceleration data is very important for understanding the dynamic performance of the vehicle. For example, when accelerating or decelerating sharply, the handling performance and safety of the vehicle will be affected. Acceleration data can also be used to analyze the driving behavior of autonomous vehicle safety officers and evaluate the performance of the vehicle under different road conditions. Longitude and latitude are the geographical coordinates of the vehicle's location. With longitude and latitude information, the vehicle's location on the earth can be accurately determined. This is critical for vehicle navigation, tracking and monitoring. Using longitude and latitude data, you can understand the vehicle's driving route in real time, determine whether the vehicle deviates from the planned route, and perform emergency rescue operations when necessary. Vehicle lights include headlights, taillights, turn signals, etc. The lighting status can reflect the vehicle's driving intention and current operating conditions. For example, turning on the turn signal means that the vehicle is about to turn, and turning on the brake light means that the vehicle is braking. Light data is very important for other road users to understand the vehicle's movements and helps improve road traffic safety. Vehicle control mode refers to the driving mode or operating mode of the vehicle. Different vehicles may have different control modes, such as automatic mode, manual mode, sports mode, economy mode, etc. In each mode, the performance and driving experience of the vehicle may be different. The choice of vehicle control mode can be adjusted according to the needs of the autonomous vehicle safety officer and driving conditions to achieve the best driving effect and fuel economy.

[0042] In this solution, the vehicle monitoring terminal retrieves and preprocesses the data. The retrieved data is time-synchronized, that is, the image data and structured CAN data are connected by aligning the timestamps to associate the vehicle speed, acceleration, longitude and latitude, lights, vehicle control mode and other information under each image data. Finally, the data is time-series processed.

[0043] Step S203, after identifying the vehicle disengagement event from the automatic driving, determining a reference period of the automatic driving disengagement time point from the preprocessing result;

[0044] Filter the data range of long-tail scenarios, that is, filter the data of the corresponding time period according to the time point when autonomous driving is disengaged.

[0045] In one embodiment, optionally, after identifying an event that the vehicle disengages from the autonomous driving, the method further includes:

[0046] Eliminate incidents where the vehicle disengages from autonomous driving in scenarios where human intervention occurs and incidents where the autonomous driving vehicle safety officer accidentally touches the vehicle's autopilot.

[0047] In this solution, the types of autonomous driving disengagement situations can be manually counted. For example, the data of the screening period can be classified and statistically analyzed to determine whether the scenario belongs to the long-tail scenario or the conventional manual takeover scenario. Specifically, the scenarios that need to be excluded are: takeovers caused by fixed corporate policies, such as manual takeovers in and out of parking stations; meaningless manual intervention by the safety officer of the autonomous driving vehicle, illegal operations such as hand slips, etc.

[0048] Through such a setting, this solution can further filter out the data of vehicle deviation from autonomous driving events, thereby improving the accuracy of model training using data.

[0049] Step S204, obtaining the preprocessing results within the reference period as training data, and labeling the training data based on the actual vehicle scenario;

[0050] In this solution, the training data can be labeled according to the actual vehicle scenario, such as long-tail scenarios, or non-long-tail scenarios, to improve the accuracy of the output results of the subsequent long-tail scenario recognition model through supervised training.

[0051] In one embodiment, optionally, labeling the training data based on the actual vehicle scenario includes:

[0052] The training data is classified into long-tail scenarios and labeled with long-tail scenario labels based on actual vehicle scenarios.

[0053] Classify and annotate long-tail scenes. Analyze the vehicle data and visual environment data of long-tail scenes, and classify the vehicle's own status and surrounding road environment information. According to the scene classification, summarize the long-tail scene types and manually annotate the key scene information.

[0054] Through such classification, this solution can improve the accuracy of model training, and at the same time, it can perform training at different granularities to improve the recognition accuracy of the model.

[0055] In one embodiment, optionally, the training data is classified into long-tail scenarios based on actual vehicle scenarios, including:

[0056] Acquire the driving data and environmental data of the vehicle; wherein the driving data includes: acceleration or deceleration state data, left turn, right turn or straight direction data; the environmental data includes: road data of intersections or sections, the number or type of surrounding traffic participants, and traffic facilities data of traffic lights or variable signs;

[0057] The training data is classified into long-tail scenarios according to the driving data and the environmental data.

[0058] Specifically, during the training process, driving data and environmental data can be combined to classify long-tail scenarios, which can help subsequent third parties to further segment long-tail scenarios and improve recognition accuracy.

[0059] In one embodiment, optionally, after labeling the training data based on the actual vehicle scenario, the method further includes:

[0060] The results based on label annotation are summarized into a long-tail scene dataset according to the format of long-tail scene recognition model training.

[0061] In this solution, during the model training process, a training data set can be formed, that is, the annotated scene data is extracted and summarized into a long-tail scene data set in a format that is convenient for model training.

[0062] Step S205 , performing supervised training on the long-tail scene recognition model based on the labeled training data, and obtaining the long-tail scene recognition model after the training is completed.

[0063] Build an automated type recognition tool based on the long-tail scene database. Specifically, train an automated long-tail scene recognition model. Train an automated long-tail scene recognition model based on the long-tail scene database, and train the corresponding scene recognition model based on the input data content and scene labels. Form a batch processing flow. Check and fill in the gaps in the recognition results, and improve the long-tail scene recognition model and database content.

[0064] Figure 3 Schematic diagram of an automated long-tail scene type recognition system based on vehicle supervision data provided according to some embodiments of the present application. Figure 3 As shown in the figure, the system is mainly composed of a data management module, a data processing module and an automatic scene recognition module. The data management module is responsible for the access, storage and output of CAN data and image data. The data processing module is responsible for the preprocessing, annotation and scene database construction of CAN data and image data. The automatic recognition module is used to automatically identify the long-tail scene type and optimize the long-tail scene recognition model and data set based on the recognition results.

[0065] (1) Resource Management Module

[0066] The resource management module is responsible for accessing and storing CAN data and image data from the vehicle monitoring terminal. The module is also used to store data in the long-tail scene database after data processing.

[0067] (2) Data processing module

[0068] This module is mainly used for:

[0069] ① Automatically filter out images and CAN data within a period of time before and after the autonomous driving is disengaged.

[0070] ② Connect image data and structured CAN data by aligning timestamps to associate information such as vehicle speed, acceleration, longitude and latitude, lights, and vehicle control mode under each image data.

[0071] ③ Quickly mark out traffic participants, traffic lights / variable signs and other traffic facility information to build a long-tail scene dataset.

[0072] (3) Automated long-tail scene recognition module

[0073] This module can train an automated long-tail scene recognition model based on a long-tail scene database, and train a corresponding scene recognition model based on the input data content and scene labels. This model can batch process image data of the corresponding time period before and after the autonomous driving is disengaged due to long-tail scenes, and can also continuously update and optimize based on the input long-tail scene data to maintain the efficiency and accuracy of the model.

[0074] This embodiment provides an automated long-tail scene type recognition system based on vehicle regulatory data, which opens up the data link between the long-tail scene recognition system and the vehicle-mounted monitoring terminal. At the same time, it helps third-party regulatory agencies to efficiently identify long-tail scene types based on regulatory image data and CAN data, which is beneficial for third-party regulatory agencies to analyze and mine the collected long-tail scene data, thereby promoting the development of driverless intelligent connected vehicles.

[0075] Figure 4 Flow chart of an automatic long-tail scene type identification method based on vehicle supervision data provided according to some embodiments of the present application. Figure 4 As shown, the process includes the following steps:

[0076] (1) Vehicle-mounted monitoring terminals collect long-tail scene data

[0077] ①Structured CAN data, mainly including vehicle speed, acceleration, latitude and longitude, heading angle, lights, vehicle control mode and other information;

[0078] ②Image data.

[0079] (2) Building a long-tail scenario database based on regulatory data

[0080] ① Data retrieval and preprocessing of the vehicle monitoring terminal. The retrieved data is time-synchronized, that is, the image data and structured CAN data are connected by aligning the timestamps to associate the vehicle speed, acceleration, longitude and latitude, lights, vehicle control mode and other information under each image data. Finally, the data is time-serialized.

[0081] ② Filter the data range of long-tail scenarios. Filter the data of the corresponding time period according to the time point when the autonomous driving is disengaged.

[0082] ③ Manual statistics of autonomous driving disengagement types. Classify and statistically analyze the data during the screening period to determine whether the scenario belongs to a long-tail scenario or a regular manual takeover scenario. Scenarios to be excluded: takeovers caused by fixed corporate policies (such as manual takeovers in and out of parking sites); meaningless manual intervention by autonomous driving vehicle safety officers (illegal operations such as hand slips).

[0083] ④ Classify and annotate long-tail scenarios. Analyze the vehicle data and visual environment data of long-tail scenarios, analyze the vehicle's own status (acceleration / deceleration, left turn / right turn / straight ahead) and surrounding road environment information (intersections / road sections, number / type of surrounding traffic participants, traffic lights / variable signs and other traffic facilities information) and classify them. According to the scenario classification, summarize the long-tail scenario types and manually annotate the key scenario information.

[0084] ⑤ Form a training data set. After the annotated scene data is extracted, it is summarized into a long-tail scene data set in a format that is convenient for model training.

[0085] (3) Build an automated type recognition tool based on the long-tail scenario database

[0086] ① Train the automated long-tail scene recognition model. According to the long-tail scene database, train the automated long-tail scene recognition model, and train the corresponding scene recognition model according to the input data content and scene labels;

[0087] ②Form a batch processing process;

[0088] ③ Check and fill in the gaps in the recognition results, and improve the long-tail scene recognition model and database content.

[0089] Through such a setting, this solution can help third-party regulatory agencies to quickly and effectively analyze long-tail scenario types based on regulatory image data and CAN data, and determine the reasons for the disengagement of unmanned vehicles.

[0090] In addition, some embodiments of the present application also provide an electronic device. The electronic device may be a digital computer in various forms, such as a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, etc. The electronic device may also be a mobile device in various forms, such as a personal digital processing, a cellular phone, a smart phone, a wearable device, and other similar computing devices.

[0091] The electronic device includes: one or more processors; and a memory storing computer program instructions, wherein when the computer program instructions are executed, the processor executes the steps of the method provided in any one or more of the above embodiments. Figure 5 An exemplary structural diagram of the electronic device is disclosed. Figure 5 As shown, the electronic device includes: one or more processors 501, a memory 502, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Among them, the components shown in this article, their connections and relationships, and their functions are only used as examples, and are not intended to limit the implementation of the present application described and / or required herein.

[0092] The electronic device may further include: an input device 503 and an output device 504. The processor 501, the memory 502, the input device 503 and the output device 504 may be connected via a bus or other means. Figure 5 The example of connecting through bus is taken in the following.

[0093] The input device 503 can receive input digital or character information, and generate key signal input related to the user settings and function control of the electronic device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick and other input devices. The output device 504 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display and a plasma display. In some embodiments, the display device may be a touch screen.

[0094] To provide interaction with a user, the electronic device may be a computer. The computer has: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball), through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0095] In the embodiments of the present application, a computer program / instruction is stored on a computer-readable medium, and when the computer program / instruction is executed by a processor, the steps of the method provided by any one or more of the above embodiments are implemented. The computer-readable medium may be included in the electronic device described in the above embodiments; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more computer-readable instructions.

[0096] The memory 502 can be used as a non-transient computer-readable storage medium, which can be used to store non-transient software programs, non-transient computer executable programs and modules. The processor 501 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 502, so as to implement the program instructions / modules corresponding to the method provided by any one or more embodiments in the embodiments of the present application.

[0097] The memory 502 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 502 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage devices. In some embodiments, the memory 502 may optionally include a memory remotely arranged relative to the processor 501, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0098] It should be noted that the computer-readable medium described in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device.

[0099] Computer readable media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0100] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet).

[0101] In the above-described embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. For example, it can be implemented by using an application specific integrated circuit (ASIC), a general-purpose computer or any other similar hardware device. In certain embodiments, the software program of the present application can be executed by a processor to implement the above steps or functions. Similarly, the software program of the present application (including relevant data structures) can be stored in a computer-readable recording medium, for example, a RAM memory, a magnetic or optical drive or a floppy disk and similar devices. In addition, some steps or functions of the present application can be implemented by hardware, for example, as a circuit that cooperates with a processor to perform each step or function.

[0102] The computer program product provided in the embodiment of the present application includes one or more computer programs / instructions, and when the computer program / instructions are executed by the processor, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive so lid state di sk (SSD)), etc.

[0103] The flow chart or block diagram in the accompanying drawings shows the possible architecture, function and operation of the equipment, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated system for hardware that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0104] The scope of the present application is defined by the appended claims rather than the above description, and it is intended that all changes that fall within the meaning and scope of the equivalent elements of the claims are included in the present application. Any figure mark in the claims should not be regarded as limiting the claims involved. In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the device claim may also be implemented by one unit or device through software or hardware. The words "first", "second", etc. are only used to distinguish the description, and do not indicate any particular order, nor can they be understood as indicating or implying relative importance.

[0105] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily mention changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims, and the above embodiments should be regarded as exemplary and non-restrictive.

Claims

1. A long-tail scene recognition method based on vehicle supervision data, characterized in that: The method comprises: Acquire monitoring data collected by the vehicle-mounted monitoring terminal; wherein the monitoring data includes structured CAN data and image data; When an event of the vehicle disengaging from the autonomous driving is identified, monitoring data within a reference period at the time point of the autonomous driving disengagement is determined as identification data; The recognition data is input into a pre-trained long-tail scene recognition model, and whether the vehicle is in a long-tail scene is recognized based on the long-tail scene recognition model.

2. The method according to claim 1, characterized in that The training process of the long-tail scene recognition model includes: Acquire monitoring data collected by the vehicle-mounted monitoring terminal; wherein the monitoring data includes structured CAN data and image data; Preprocessing the monitoring data to obtain a preprocessing result; After identifying the vehicle disengagement event, determining a reference period of the disengagement time point of the autonomous driving from the preprocessing results; Obtaining preprocessing results within a reference period as training data, and labeling the training data based on actual vehicle scenarios; The long-tail scene recognition model is supervisedly trained based on the labeled training data, and the long-tail scene recognition model is obtained after the training is completed.

3. The method according to claim 2, characterized in that The monitoring data is preprocessed to obtain a preprocessing result, including: The monitoring data is subjected to time synchronization processing and time serialization processing to obtain a preprocessing result; wherein, in the preprocessing result, each image data is associated with at least one structured CAN data of vehicle speed, acceleration, longitude and latitude, light, and vehicle control mode.

4. The method according to claim 2, characterized in that: The training data is labeled based on the actual vehicle scenario, including: The training data is classified into long-tail scenarios and labeled with long-tail scenario labels based on actual vehicle scenarios.

5. The method according to claim 4, characterized in that The training data is classified into long-tail scenarios based on actual vehicle scenarios, including: Acquire the driving data and environmental data of the vehicle; wherein the driving data includes: acceleration or deceleration state data, left turn, right turn or straight direction data; the environmental data includes: road data of intersections or sections, the number or type of surrounding traffic participants, and traffic facilities data of traffic lights or variable signs; The training data is classified into long-tail scenarios according to the driving data and the environmental data.

6. The method according to claim 2, characterized in that After identifying an event in which the vehicle disengages from the autonomous driving, the method further includes: Eliminate incidents where the vehicle disengages from autonomous driving in scenarios where human intervention occurs and incidents where the autonomous driving vehicle safety officer accidentally touches the vehicle's autopilot.

7. The method according to claim 2, characterized in that: After labeling the training data based on the actual vehicle scenario, the method further includes: The results based on label annotation are summarized into a long-tail scene dataset according to the format of long-tail scene recognition model training.

8. An electronic device, characterized in that: The electronic device comprises: one or more processors; and A memory storing computer program instructions, which, when executed, cause the processor to perform the steps of the method as claimed in any one of claims 1 to 7.

9. A computer readable medium having a computer program / instructions stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images