Hard-case recognition method, system and apparatus

Abstract

The present application provides a hard-case recognition method, system and apparatus, which are used for mining potential hard-case scenarios during vehicle driving, and realizing precise mining of hard-case scenarios. The method comprises: first acquiring a perception data set collected during vehicle driving, wherein the perception data set is obtained by means of collection by at least one sensor in a vehicle in a current scenario, and the perception data comprises a plurality of frames of perception data; then inputting the plurality of frames of perception data into a perception model to obtain a plurality of perception results, wherein the perception model is used for detecting elements in the inputted data, and can output information of the detected elements, for example, the category, position, shape or size of the elements; and then, on the basis of the similarity between the plurality of perception results, determining a difficulty coefficient, wherein the difficulty coefficient is used for measuring the difficulty of determining a vehicle driving decision in the current scenario, or measuring the perception capability of the vehicle or perception model in the current scenario.

Description

A difficult example identification method, system and device

[0001] The present application claims priority to the Chinese patent application No. 202410657825.4, filed on May 24, 2024, and entitled "A difficult example identification method, system and device", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the field of intelligent driving, in particular to a difficult example identification method, system and device. BACKGROUND

[0003] With the popularization of intelligent driving functions, intelligent driving will become a very common driving mode, and accurate understanding of complex road structures is crucial for intelligent driving, which will directly affect the safety of vehicles and user experience.

[0004] The perception ability of the vehicle side is relatively reliable for general road conditions, but in complex scenarios, the real-time perception ability of a single vehicle still needs to be further improved. Generally, to improve such ability, a large amount of rich true value data is needed to train the vehicle side network, thereby enhancing its perception of complex environments. However, for some difficult example scenarios of road structures, how to improve the cognitive ability of the vehicle side to complex road structures becomes a problem to be solved. SUMMARY

[0005] The present application provides a difficult example identification method, system and device for mining difficult example scenarios that may exist during vehicle driving, and realizing accurate mining of difficult example scenarios.

[0006] Therefore, in a first aspect, the present application provides a difficult example identification method, comprising: first, obtaining a set of perception data collected during vehicle driving, the set of perception data being collected by at least one sensor in the vehicle for the current scene, and the perception data including multiple frames of perception data; then inputting the multiple frames of perception data into a perception model to obtain multiple perception results, the perception model being used for detecting elements in the input data and can output information of the detected elements, such as the category, position, shape or size of the elements; and then determining a difficulty coefficient according to the similarity between the multiple perception results, the difficulty coefficient being used to measure the difficulty of determining a vehicle driving decision in the current scene, or to measure the perception ability of the vehicle or the perception model in the current scene.

[0007] In the embodiments of the present application, the difficulty coefficient can be calculated by using the difference between different perception data frames, so as to identify the difficult example scene based on the difficulty coefficient. The difference between different perception data frames can be used to represent the perception difference of the perception model for different perception data frames. If the difference is too large, it means that the perception model may have errors in the perception of the current scene, resulting in inaccurate perception results. In this case, the current scene can be considered as a difficult example scene. If the perception difference is not large, it means that the output of the perception model is stable, and the elements in the perception data can be stably identified, which means that the current scene is a non-difficult example scene. Therefore, by using the perception ability of the perception model for different perception data, the difficulty coefficient corresponding to the current scene is determined, so that the difficult example scene is accurately identified.

[0008] In a possible implementation, the multi-frame perception data includes first-frame perception data and second-frame perception data, the first-frame perception data and the second-frame perception data are collected by the same sensor at different time points for the current scene, that is, the first-frame perception data and the second-frame perception data are perception data with different time domain dimensions; the inputting of the multi-frame perception data into the perception model to obtain the plurality of perception results can include: inputting the first-frame perception data and the second-frame perception data into the perception model respectively to output first perception result and second perception result, the first perception result includes information of at least one first element extracted from the first-frame perception data, and the second perception result includes information of at least one second element extracted from the second-frame perception data; the determining of the difficulty coefficient according to the similarity between the plurality of perception results can include: determining the difficulty coefficient according to the similarity between the first perception result and the second perception result.

[0009] In the embodiments of the present application, the perception difference of the perception model in the time domain dimension can be used to calculate the difficulty coefficient, so as to accurately identify the difficult example scene from the time domain dimension.

[0010] In a possible implementation, the method further includes: calculating the similarity between the information of the at least one first element and the information of the at least one second element; and obtaining the similarity between the first perception result and the second perception result according to the similarity between the information of the at least one first element and the information of the at least one second element. In the embodiments of the present application, the difficulty coefficient can be calculated from the dimension of the elements, so as to calculate the difficulty coefficient in a finer granularity dimension, so as to more accurately identify the difficult example scene.

[0011] In a possible implementation, the calculating the similarity between the information of the at least one first element and the information of the at least one second element includes: calculating the similarity between the position and / or attribute of each of the at least one first element and the information of the corresponding second element, that is, calculating the similarity between the elements in different time domains in the position and / or attribute.

[0012] In the implementation, the difficulty coefficient can be calculated from the dimensions of the position and / or attribute of the element. The similarity between the position and the attribute of the element can accurately indicate whether the perception model outputs an element with a position or attribute jump, so that the perception ability of the perception model in the current scene can be accurately measured, and the corresponding difficulty coefficient can be generated, so that the difficult example scene can be accurately identified subsequently.

[0013] In a possible implementation, the multiple frames of perception data include third frame perception data and fourth frame perception data, and the third frame perception data and the fourth frame perception data are collected by different sensors, that is, the perception data set can include different perception frames in the spatial dimension. The inputting the multiple frames of perception data into the perception model to obtain multiple perception results can further include: inputting the third frame perception data and the fourth frame perception data into the perception model respectively to output third perception results and fourth perception results, the third perception results including information of at least one third element extracted from the third frame perception data, and the fourth perception results including information of at least one fourth element extracted from the fourth frame perception data, that is, the perception data including the same scene collected by different sensors is taken as the input of the perception model, and the perception results of the perception model for the perception data of different sensors are output. Correspondingly, the determining the difficulty coefficient according to the similarity between the multiple perception results can further include: determining the difficulty coefficient according to the similarity between the third perception results and the fourth perception results. In the implementation, the perception ability of the perception model for the perception data of different sensors can be calculated, and the difficulty coefficient is measured according to the difference between the perception abilities, so that the difficult example scene can be identified subsequently, and the difficult example scene can be mined in the spatial dimension.

[0014] In a possible implementation, the method further includes: calculating the similarity between the information of the at least one third element and the information of the at least one fourth element; and obtaining the similarity between the third perception results and the fourth perception results according to the similarity between the information of the at least one third element and the information of the at least one fourth element.

[0015] In the embodiment of the present application, similar to the similarity calculation method in the time domain, the similarity can also be calculated from the dimension of the element, so as to measure the difference in the perception ability of the perception model in the space domain from the dimension of the element, identify whether the perception result in the space domain produces a jump, and accurately excavate the difficult example scene in the space domain.

[0016] In a possible implementation, the aforementioned difficulty coefficient is calculated based on information of static elements in the plurality of perception results, and the static element is an element with a moving speed lower than a preset moving speed in the current scene. In the embodiment of the present application, when calculating the difficulty coefficient, the static element can be mainly referred to for calculation. Generally, the dynamic element can change in the scene, so the dynamic element has little reference significance for subsequent model training, and therefore the dynamic element can be filtered out, and the difficulty coefficient is calculated based on only the perception result of the static element, so that the obtained difficulty coefficient has more reference value.

[0017] In a possible implementation, the aforementioned method further includes: in the case where the difficulty coefficient is greater than the first threshold value, sending difficult example information to the cloud end, the difficult example information including the difficulty coefficient and the perception data collected in the current scene, so that the cloud end can label the difficult example information and train the model using the labeled data. Therefore, in the embodiment of the present application, the difference between the perception results corresponding to the data collected by the vehicle end sensor can be used to excavate the difficult example scene, so as to provide the cloud end with usable training data.

[0018] In a possible implementation, the aforementioned method further includes: obtaining a difficult example duration, the difficult example duration being a duration during which the difficulty coefficient is greater than the first threshold value, and the difficult example information further including the difficult example duration. In the embodiment of the present application, when uploading the difficult example information, the difficult example duration can also be uploaded, so that the cloud end can also obtain the duration of the difficult example scene, so as to facilitate subsequent labeling of the difficult example data.

[0019] In a possible implementation, the aforementioned method further includes: receiving a perception model sent by the cloud end, the perception model being a model obtained by the cloud end using the difficult example information for training. In addition, after receiving the perception model sent by the cloud end, the aforementioned perception model can be replaced by the perception model, or the perception model can be deployed locally on the vehicle end, so as to improve the perception ability of the vehicle.

[0020] In a possible implementation, the aforementioned method further includes: determining a driving decision of the vehicle according to the difficulty coefficient. Therefore, in the embodiment of the present application, the driving decision of the vehicle can also be determined based on the difficulty coefficient, such as reducing the intelligent driving level of the vehicle, or reminding the user to take over the vehicle, so as to improve the driving safety of the vehicle.

[0021] In a second aspect, the present application provides a model training method, comprising: receiving difficult example information sent by at least one vehicle, the generation manner of the difficult example information can refer to the introduction of the first aspect or any optional implementation manner of the first aspect; labeling the difficult example information to obtain corresponding labeled data; performing model training according to the difficult example information and the labeled data to obtain a perception model; and sending the perception model to the at least one vehicle, so that the vehicle can locally deploy the received perception model.

[0022] In the implementation manner of the present application, the vehicle end can accurately identify the difficult example scene, so that the mined difficult example information is uploaded to the cloud end for model training, so as to obtain a model that can perceive more difficult example scenes, and the model is distributed to the vehicle end, so as to improve the perception ability of the vehicle end.

[0023] In a third aspect, the present application provides a difficult example identification device, comprising:

[0024] A data acquisition module is configured to acquire a perception data set, the perception data set comprising perception data collected by at least one sensor in a vehicle for a current scene, and the perception data comprising multiple frames of perception data.

[0025] A perception module is configured to input the multiple frames of perception data into a perception model to obtain multiple perception results, and the perception model is configured to detect elements in the input data.

[0026] A difficult example detection module is configured to determine a difficulty coefficient according to the similarity between the multiple perception results, and the difficulty coefficient is configured to measure the difficulty of determining a driving decision of the vehicle in the current scene.

[0027] The effects of the third aspect and any optional implementation manner of the third aspect can refer to the effects of the first aspect or any optional implementation manner of the first aspect, and will not be described hereinafter.

[0028] In a possible implementation manner, the multiple frames of perception data comprise first frame perception data and second frame perception data, and the first frame perception data and the second frame perception data are collected by the same sensor at different time points for the current scene.

[0029] The perception module is specifically configured to input the first frame perception data and the second frame perception data into the perception model respectively, and output a first perception result and a second perception result, wherein the first perception result comprises information of at least one first element extracted from the first frame perception data, and the second perception result comprises information of at least one second element extracted from the second frame perception data.

[0030] The difficult example detection module is specifically configured to determine the difficulty coefficient according to the similarity between the first perception result and the second perception result.

[0031] In a possible implementation, the difficult case detection module is further configured to: calculate a similarity between the information of the at least one first element and the information of the at least one second element; and obtain a similarity between the first perception result and the second perception result according to the similarity between the information of the at least one first element and the information of the at least one second element.

[0032] In a possible implementation, the difficult case detection module is specifically configured to: calculate a similarity between a position and / or an attribute of the information of each of the at least one first element and the information of the corresponding second element.

[0033] In a possible implementation, the multi-frame perception data includes third-frame perception data and fourth-frame perception data, and the third-frame perception data and the fourth-frame perception data are obtained by different sensors.

[0034] The perception module is further configured to input the third-frame perception data and the fourth-frame perception data into the perception model respectively, and output a third perception result and a fourth perception result, wherein the third perception result includes information of at least one third element extracted from the third-frame perception data, and the fourth perception result includes information of at least one fourth element extracted from the fourth-frame perception data.

[0035] The difficult case detection module is specifically configured to determine the difficulty coefficient according to a similarity between the third perception result and the fourth perception result.

[0036] In a possible implementation, the difficult case detection module is further configured to: calculate a similarity between the information of the at least one third element and the information of the at least one fourth element; and obtain a similarity between the third perception result and the fourth perception result according to the similarity between the information of the at least one third element and the information of the at least one fourth element.

[0037] In a possible implementation, the difficulty coefficient is calculated based on information of a static element in the plurality of perception results, and the static element is an element with a moving speed lower than a preset moving speed in a current scene.

[0038] In a possible implementation, the apparatus further includes a transceiver module configured to: in a case where the difficulty coefficient is greater than a first threshold, send, to a cloud, difficult case information, wherein the difficult case information includes the difficulty coefficient.

[0039] In a possible implementation, the difficult case detection module is further configured to: obtain a difficult case duration, wherein the difficult case duration is a duration during which the difficulty coefficient is greater than the first threshold, and the difficult case information further includes the difficult case duration.

[0040] In a possible implementation, the transceiver module is further configured to receive a perception model sent by the cloud, wherein the perception model is a model trained using the difficult case information.

[0041] In a possible implementation, the device further includes a driving decision module configured to determine a driving decision of the vehicle according to the difficulty coefficient.

[0042] In a fourth aspect, the present application provides a cloud platform, comprising:

[0043] The transceiver module is configured to receive the difficult example information sent by the at least one vehicle, and the generation of the difficult example information can refer to the description of the first aspect or any optional implementation of the first aspect.

[0044] The labeling module is configured to label the difficult example information to obtain corresponding labeled data.

[0045] The training module is configured to train a model according to the difficult example information and the labeled data to obtain a perception model.

[0046] The transceiver module is further configured to send the perception model to the at least one vehicle, and the vehicle can locally deploy the received perception model.

[0047] In a fifth aspect, the embodiments of the present application provide a difficult example identification device, comprising a processor and a memory, wherein the processor and the memory are interconnected through a circuit, the processor calls program codes in the memory to perform functions related to processing in the difficult example identification method shown in any one of the first aspects. Optionally, the difficult example identification device can be a chip.

[0048] In a sixth aspect, the embodiments of the present application provide a cloud platform, comprising a processor and a memory, wherein the processor and the memory are interconnected through a circuit, the processor calls program codes in the memory to perform functions related to processing in the method shown in any one of the second aspects. Optionally, the cloud platform can also be a chip.

[0049] In a seventh aspect, the embodiments of the present application provide an intelligent driving vehicle, comprising a processor and a memory, wherein the processor and the memory are interconnected through a circuit, the processor calls program codes in the memory to perform functions related to processing in the difficult example identification method shown in any one of the first aspects.

[0050] In an eighth aspect, the present application provides a cloud service system, comprising a vehicle and a cloud platform, the vehicle can include the vehicle mentioned in any implementation of the third aspect, and the cloud platform can include the cloud platform mentioned in any implementation of the fourth aspect.

[0051] In a ninth aspect, an embodiment of the present application provides a digital processing chip or chip, which includes a processing unit and a communication interface, the processing unit acquires program instructions through the communication interface, the program instructions are executed by the processing unit, and the processing unit is configured to perform the processing-related functions in any of the embodiments of the first aspect to the second aspect.

[0052] In a tenth aspect, an embodiment of the present application provides a computer-readable storage medium, which includes instructions, when executed on a computer, cause the computer to perform the method in any of the embodiments of the first aspect or the second aspect.

[0053] In an eleventh aspect, an embodiment of the present application provides a computer program product including computer programs / instructions, when executed by a processor, cause the processor to perform the method in any of the embodiments of the first aspect or the second aspect. BRIEF DESCRIPTION OF DRAWINGS

[0054] FIG. 1 is a schematic diagram of an architecture of a cloud service system provided by the present application;

[0055] FIG. 2 is a schematic diagram of a structure of a vehicle provided by the present application;

[0056] FIG. 3 is a schematic diagram of a flow of a difficult example identification method provided by the present application;

[0057] FIG. 4 is a schematic diagram of an architecture of another cloud service system provided by the present application;

[0058] FIG. 5 is a schematic diagram of a flow of another difficult example identification method provided by the present application;

[0059] FIG. 6 is a schematic diagram of a segmentation result provided by the present application;

[0060] FIG. 7 is a schematic diagram of a flow of another difficult example identification method provided by the present application;

[0061] FIG. 8 is a schematic diagram of a flow of another difficult example identification method provided by the present application;

[0062] FIG. 9 is a schematic diagram of a structure of a difficult example identification apparatus provided by the present application;

[0063] FIG. 10 is a schematic diagram of a structure of a cloud platform provided by the present application;

[0064] FIG. 11 is a schematic diagram of a structure of a computing device provided by the present application. DETAILED DESCRIPTION

[0065] With reference to the drawings, the technical solutions in the embodiments of the present application will be described below. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all the other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0066] The method provided in the present application can be applied to a vehicle, and can also be applied to a system architecture formed by the vehicle and the cloud.

[0067] For example, the cloud service system architecture to which the method provided in the present application is applied can be as shown in FIG. 1. As shown in FIG. 1, the cloud service system 10 can include a cloud end 11 and a vehicle end 14.

[0068] The cloud end 11 can specifically include a processing server 12 and a data center 13,

[0069] The processing server 12 can specifically include a server cluster or an independently deployed computing device, or the computing device can also be other computing devices. Optionally, the processing server 12 can cooperate with other computing devices, such as data storage, routers, load balancers and the like. The processing server 12 can use data in the data storage system or call program code in the data storage system to implement the method steps provided in the embodiments of the present application.

[0070] The data center 13 can be used to store data, for the processing server 12 to query or write data and the like.

[0071] The vehicle end 14 is connected to the processing server 12, and one or more sensors are arranged in the vehicle end 14, which can be used to collect perception data of the environment where the vehicle is located.

[0072] The vehicle end 14 can send difficult example information to the processing server 12, the processing server 12 can label the difficult example information, and use the labeled difficult example information for model training, and distribute the trained model to the vehicle end 14. In other words, for the difficult example road perceived by the vehicle end, the processing server 12 can train specifically, so as to improve the output effect of the model on the difficult example road part, so that the vehicle end 14 can use the trained model to perceive the difficult example road part, and improve the intelligent driving function of the vehicle end 14, such as automatic driving or auxiliary driving function and the like.

[0073] The vehicle structure provided in the present application will be introduced below.

[0074] Referring to FIG. 2, FIG. 2 is a schematic diagram of a structure of a vehicle 100 according to an embodiment of the present application. The vehicle 100 can be configured to operate in an autonomous driving mode. For example, the vehicle 100 can control itself while in the autonomous driving mode, determine whether there is an obstacle in a surrounding environment, and control the vehicle 100 based on information of the obstacle. The vehicle 100 can also be configured to operate without human interaction while in the autonomous driving mode.

[0075] FIG. 2 is a functional block diagram of the vehicle 100 according to an embodiment of the present application. The vehicle 100 can be configured to operate in a full or partial autonomous driving mode. For example, the vehicle 100 can obtain surrounding environment information through a perception system 120, and obtain an autonomous driving strategy based on analysis of the surrounding environment information to achieve full autonomous driving, or present the analysis result to a user to achieve partial autonomous driving.

[0076] The vehicle 100 can include various subsystems, such as an infotainment system 110, the perception system 120, a decision control system 130, a drive system 140, and a computing platform 150. Alternatively, the vehicle 100 can include more or fewer subsystems, and each subsystem can include multiple components. In addition, each subsystem and component of the vehicle 100 can be interconnected through wired or wireless means.

[0077] In some embodiments, the infotainment system 110 can include a communication system 111, an entertainment system 112, and a navigation system 113.

[0078] The communication system 111 can include a wireless communication system 111 that can wirelessly communicate with one or more devices directly or via a communication network. For example, the wireless communication system 111 can use 3G cellular communication, such as CDMA, EVDO, GSM / GPRS, or 4G cellular communication, such as LTE, or 5G cellular communication. The wireless communication system 111 can communicate with a wireless local area network (WLAN) using WiFi. In some embodiments, the wireless communication system 146 can communicate directly with devices using an infrared link, Bluetooth, or ZigBee. The wireless communication system 111 can include one or more dedicated short range communications (DSRC) devices that can include public and / or private data communication between vehicles and / or roadside stations.

[0079] The entertainment system 112 can include a center screen, a microphone and a sound system, based on which a user can listen to the radio or play music in the vehicle, or connect the mobile phone with the vehicle and realize the mobile phone screen projection on the center screen. The center screen can be touchable, and the user can operate through the touch screen. In some cases, the user's voice signal can be obtained through the microphone, and some control of the vehicle 100 by the user can be realized according to the analysis of the user's voice signal, such as adjusting the temperature in the vehicle. In other cases, music can be played to the user through the sound system.

[0080] The navigation system 113 can include a map service, thereby providing the vehicle 100 with a navigation of a driving route. The navigation system 113 can be used in cooperation with the global positioning system 121 and the inertial measurement unit 122 of the vehicle. The map can be a two-dimensional map, a high-precision map, or a map constructed based on data collected during the driving of the vehicle.

[0081] The perception system 120 can include several sensors that sense information about the environment surrounding the vehicle 100. For example, the perception system 120 can include a global positioning system 121 (which can be a GPS system, a Beidou system or other positioning system), an inertial measurement unit (IMU) 122, a laser radar 123, a millimeter wave radar 124, an ultrasonic radar 125 and a camera 126. The perception system 120 can also include sensors that monitor the internal systems of the vehicle 100 (e.g., an in-vehicle air quality monitor, a fuel gauge, an oil temperature gauge, etc.). Sensor data from one or more of these sensors can be used to detect objects and their respective characteristics (position, shape, direction, speed, etc.). Such detection and identification are key functions for the safe operation of the vehicle 100. The perception data collected by the sensors in the vehicle mentioned below in this application can include information collected by each unit in the perception system 120.

[0082] The global positioning system 121 can be used to determine the geographical position of the vehicle 100.

[0083] The inertial measurement unit 122 is used to sense the position and orientation changes of the vehicle 100 based on inertial acceleration. In some embodiments, the inertial measurement unit 122 can be a combination of an accelerometer and a gyroscope.

[0084] The laser radar 123 can use laser to sense objects in the environment in which the vehicle 100 is located. In some embodiments, the laser radar 123 can include one or more laser sources, a laser scanner and one or more detectors, as well as other system components.

[0085] The millimeter wave radar 124 can utilize radio signals to sense objects within the surrounding environment of the vehicle 100. In some embodiments, in addition to sensing objects, the millimeter wave radar 124 can also be used to sense the speed and / or heading of the objects.

[0086] The ultrasonic radar 125 can utilize ultrasonic signals to sense objects around the vehicle 100.

[0087] The camera 126 can be used to capture image information of the surrounding environment of the vehicle 100. The camera 126 can include a monocular camera, a binocular camera, a structured light camera, a panoramic camera, etc. The image information captured by the camera 126 can include still image information or video stream information.

[0088] The decision control system 130 includes a computing system 131 that makes analytical decisions based on the information obtained by the perception system 120. The decision control system 130 also includes a vehicle controller 132 that controls the power system of the vehicle 100, as well as a steering system 133, a throttle 134, and a braking system 135 that control the vehicle 100.

[0089] The computing system 131 can process and analyze various information obtained by the perception system 120 in order to identify targets, objects, and / or features in the surrounding environment of the vehicle 100. The targets can include pedestrians or animals, and the objects and / or features can include traffic signals, road boundaries, and obstacles. The computing system 131 can use object recognition algorithms, Structure from Motion (SFM) algorithms, video tracking, etc. In some embodiments, the computing system 131 can be used to map the environment, track objects, estimate the speed of objects, etc. The computing system 131 can analyze the various information obtained and derive a control strategy for the vehicle.

[0090] The vehicle controller 132 can be used to coordinate the control of the power battery and the engine 141 of the vehicle in order to improve the power performance of the vehicle 100.

[0091] The steering system 133 can be used to adjust the heading of the vehicle 100. For example, in one embodiment, the steering system 133 can be a steering wheel system.

[0092] The throttle 134 is used to control the operating speed of the engine 141 and, in turn, the speed of the vehicle 100.

[0093] The braking system 135 is used to control the deceleration of the vehicle 100. The braking system 135 can use friction to slow down the rotation of the wheels 144. In some embodiments, the braking system 135 can convert the kinetic energy of the wheels 144 into electrical current. The braking system 135 can also take other forms to slow down the rotation of the wheels 144 in order to control the speed of the vehicle 100.

[0094] The drive system 140 includes components that provide motive movement for the vehicle 100. In one embodiment, the drive system 140 can include an engine 141, an energy source 142, a drivetrain 143, and wheels 144. The engine 141 can be an internal combustion engine, an electric motor, an air compression engine, or other types of engine combinations, such as a hybrid engine composed of a gasoline engine and an electric motor, a hybrid engine composed of an internal combustion engine and an air compression engine. The engine 141 converts the energy source 142 into mechanical energy.

[0095] Examples of the energy source 142 include gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, and other sources of electrical power. The energy source 142 can also provide energy for other systems of the vehicle 100.

[0096] The drivetrain 143 can transmit the mechanical power from the engine 141 to the wheels 144. The drivetrain 143 can include a transmission, a differential, and drive axles. In one embodiment, the drivetrain 143 can also include other devices, such as a clutch. The drive axles can include one or more axles that can be coupled to one or more wheels 144.

[0097] Some or all of the functions of the vehicle 100 are controlled by a computing platform 150. The computing platform 150 can include at least one processor 151 that can execute instructions 153 stored in a non-transitory computer readable medium, such as a memory 152. In some embodiments, the computing platform 150 can also be a plurality of computing devices that control individual components or subsystems of the vehicle 100 in a distributed manner.

[0098] The processor 151 can be any conventional processor, such as commercially available CPUs. Alternatively, the processor 151 can also include a graphics processing unit (GPU), a field programmable gate array (FPGA), a system on chip (SOC), an application-specific integrated circuit (ASIC), or a combination thereof. The processor 151 can be located on a device that is remote from the vehicle and in wireless communication with the vehicle.

[0099] In some embodiments, the memory 152 can include instructions 153 (e.g., program logic) that can be executed by the processor 151 to perform various functions of the vehicle 100. The memory 152 can also include additional instructions, including instructions to send data to, receive data from, interact with, and / or control one or more of the infotainment system 110, the perception system 120, the decision control system 130, and the drive system 140.

[0100] In addition to the instructions 153, the memory 152 can store data, such as road maps, route information, the location, orientation, speed, and other similar vehicle data of the vehicle, as well as other information. Such information can be used by the vehicle 100 and the computing platform 150 during operation of the vehicle 100 in autonomous, semi-autonomous, and / or manual modes.

[0101] The computing platform 150 can control the functions of the vehicle 100 based on inputs received from various subsystems, such as the drive system 140, the perception system 120, and the decision control system 130. For example, the computing platform 150 can utilize inputs from the decision control system 130 in order to control the steering system 133 to avoid obstacles detected by the perception system 120. In some embodiments, the computing platform 150 can be operable to provide control over many aspects of the vehicle 100 and its subsystems.

[0102] Optionally, one or more of the above-described components can be installed separately from or associated with the vehicle 100. For example, the memory 152 can exist partially or entirely separately from the vehicle 100. The above-described components can be communicatively coupled together in a wired and / or wireless manner.

[0103] Optionally, the above-described components are only an example, and in actual applications, components in each module described above can be added or deleted according to actual needs, and FIG. 2 should not be understood as a limitation on the embodiments of the present application.

[0104] The vehicle 100 described above can be a car, a truck, a motorcycle, a bus, a ship, an airplane, a helicopter, an entertainment vehicle, an amusement park vehicle, construction equipment, a trolley, a golf cart, or a train, etc. that can achieve intelligent driving or a vehicle terminal, etc. The embodiments of the present application are not particularly limited.

[0105] In the intelligent driving function of a vehicle, accurate recognition of complex road structures is crucial. For example, in automatic driving in intelligent driving, the process of automatic driving can be divided into multiple steps, including perception, decision planning, and driving control, etc. In the perception stage, the vehicle itself is positioned and the surrounding environment of the vehicle is analyzed, such as vehicles, pedestrians, roads, trees, walls, and road signs around the ego vehicle. The perception stage usually relies on the fusion perception of multiple sensors in the vehicle, such as visual sensors (such as image sensors) and radars (such as millimeter waves, ultrasonic waves, lidar, etc.). The vehicle can achieve intelligent driving based on the perception of the surrounding environment and the road and the reference map. Therefore, accurate perception of road structures is crucial for the intelligent driving function of the vehicle.

[0106] Generally, the perception ability of the vehicle end is relatively reliable for general road conditions, but in complex scenarios, the real-time perception ability of a single vehicle still needs to be further improved. Generally, to improve this ability, a large amount of rich true value data is needed to train the vehicle end network, thereby enhancing its perception of complex environments. In addition, high-definition maps (HD) can also be used as reference data for vehicle intelligent driving functions, but due to high production costs, maintenance and updating difficulties, they have gradually failed to meet the needs of the development of intelligent driving. Therefore, how to quickly mine and produce road structure difficult example data to improve the recognition ability of the vehicle end network for complex road structures is a major challenge currently faced by the intelligent driving field.

[0107] For example, in an existing scheme, in order to assist the vehicle to accurately recognize road difficult example scenarios, a special collection vehicle equipped with high-precision sensors (such as lidar, camera) is used to collect road difficult example data, and a high-definition map is made in the cloud and sent to the vehicle end. However, in this scheme, the perception of the road environment by the vehicle end is very dependent on the warning map made by the cloud, and the production cost is very high. Moreover, the production of the high-definition map requires the use of a special collection vehicle equipped with high-precision sensors to collect road difficult example data, and the cost of data collection by humans is also very high. In addition, the maintenance cost of the map is very high, and if the map update cycle is long, it may lead to the timeliness invalidation of the map information.

[0108] For example, in an existing scheme, in the manned state, the system includes sensors that are still running but do not participate in vehicle control, only to verify the decision algorithm. The algorithm of the system makes continuous simulation decisions in the "shadow mode", and compares the decisions with the behavior of the driver. Once the two are inconsistent, the scene is determined as an "extreme working condition", which triggers data backhaul, and the training of difficult example scenes in the cloud based on the backhaul data. However, since the difficult example data collection is based on the inconsistency of the decisions, it is not possible to directly locate where the error occurs, and it is necessary to analyze the cloud to determine whether it is a road structure cognition difficult example. And the length / content of the collected data cannot be accurately specified according to the road structure cognition training requirements, which may cause waste of traffic due to excessive data collection, or insufficient data collection to support training.

[0109] Therefore, the present application provides a difficult example identification method, which compares different perception data collected, calculates the difficulty coefficient based on the difference between different perception data, so as to accurately identify difficult examples.

[0110] Referring to FIG. 3, the present application provides a flowchart of a difficult example identification method, as follows.

[0111] 301, obtain a perception data set.

[0112] The perception data includes at least one sensor in the vehicle, and the perception data set can include multiple frames of perception data. For example, the perception data set can include at least one sensor in the vehicle collecting the current scene of the vehicle.

[0113] It should be noted that the frame of perception data in the embodiment of the present application can be the data collected by a sensor in a unit time. The data collected by the same sensor in different unit times is different frames, and the data collected by different sensors in the same unit time is also different frames.

[0114] The at least one sensor can include an image sensor or a radar, for example, and can include the sensors in the perception system 120 in FIG. 2, as described above.

[0115] Correspondingly, the data in the perception data set can include image or point cloud data. Further, the perception data set can include data collected by different sensors at the same time, or data collected by the same sensor at adjacent times, etc.

[0116] ​For example, the set of perception data can include images captured by the image sensor at the t-th second and images captured by the image sensor at the t+1-th second; or the set of perception data can include images captured by the image sensor and the radar respectively under the same scene.

[0117] 302. inputting the multiple frames of perception data into the perception model to obtain multiple perception results.

[0118] The perception model can be used to detect elements in the input data, such as performing target detection, segmentation tasks, or other tasks that can be used to identify elements. The input data of the multi-modal perception data can include multiple modalities, such as image or radar perception data.

[0119] The perception model can include one or more models, such as a corresponding perception model for different types of input to detect elements in different types of data. The perception model can also use a multi-modal perception model, i.e., the input of the perception model can include multi-modal data, and the perception model can detect elements in different modalities of data and output corresponding perception results.

[0120] After obtaining the multiple frames of perception data, the multiple frames of perception data can be input into the perception model to output the perception results corresponding to each frame of perception data, and obtain multiple perception results. The perception results can include the position and attribute information of the identified elements, such as the position, size, shape, or type of the elements.

[0121] Specifically, the elements can correspond to entities in the scene covered by the perception data, such as vehicles, lane lines, road edges, traffic signs, or pedestrians in the environment.

[0122] In one possible scenario, the multiple frames of perception data can include, but are not limited to, point cloud data captured by a laser radar, image data captured by an image sensor, or data captured by an ultrasonic radar. The calibration parameters of each sensor can be used to map each frame of perception data to an aligned matrix, which can be input into the perception model.

[0123] For example, in the case that there is point cloud data and image data in the perception data, the point cloud data and the image data are mapped into a matrix according to the calibration parameters of the lidar and the camera, to obtain single-channel or multi-channel input data, which is input into the perception model. For the point cloud data, the perception model can output the perception result in the 3D space. For example, in the road environment collection scene, the spatial position information and the element attribute information of the vehicles, pedestrians, static obstacles, road signs and other elements in the environment can be recognized. For the image data, the perception model can output the visual perception result in the 2D space. For example, in the road environment collection scene, the position or attribute information of the vehicles, pedestrians, static obstacles, road signs and other elements in the environment can be recognized.

[0124] 303. Determine the difficulty coefficient according to the similarity between the plurality of perception results.

[0125] After obtaining the plurality of perception results, the similarity between the plurality of perception results can be calculated, and the difficulty coefficient can be calculated based on the similarity. The difficulty coefficient can be used to measure the difficulty of determining the driving decision of the vehicle in the current scene, or the perception ability of the perception system of the vehicle for the current scene. The difficulty coefficient can be used to determine whether the current scene is a difficult example scene. The difficult example scene is a scene with high complexity, or a scene with high difficulty for the vehicle to make a driving decision.

[0126] In addition, the algorithm for calculating the similarity can use cosine similarity, Euclidean distance or Manhattan distance, which can be adjusted according to the actual application scenario. Of course, whether the position or attribute of the elements changes can also be directly output as the similarity.

[0127] Therefore, in the embodiments of the present application, the similarity between different perception data collected by the sensor can be used to determine the difficulty of making a driving decision in the current scene, so as to identify whether the current scene is a difficult example scene based on the difficulty. Generally, for the difficult example scene, the perception ability of the model for the perception result of the data collected by the sensor is poor, and the element perception difference may be generated for different data frames in the same scene, so the current scene can be identified as a difficult example scene based on the difference, that is, a scene with poor perception effect, so that the difficult example scene can be identified.

[0128] Optionally, the elements identified in the perception result can be divided into dynamic elements and static elements. The static elements are elements that can not change in the scene or change within a period of time, such as lane lines, road edges, road signs, or guide signs. The dynamic elements are elements with a certain speed, such as pedestrians and vehicles. Generally, dynamic elements can change in the scene, such as pedestrians or vehicles, so the position or attributes of dynamic elements in different perception data can jump, and even interfere with the perception ability of the vehicle end. Therefore, in the embodiments of the present application, when calculating the similarity, the similarity between static elements can be calculated to measure the perception ability or difficulty of the vehicle for the current scene, so as to more accurately measure the perception difficulty of the vehicle in the current scene and accurately identify the difficult examples.

[0129] Optionally, when calculating the difficulty coefficient, the difficulty coefficient can be calculated in various ways. For example, the similarity or difference between the data collected by the same sensor at adjacent time points, or the similarity or difference between the different data collected by different sensors in the same scene, or a combination of the similarity or difference between the data collected by the same sensor at adjacent time points and the similarity or difference between the different data collected by different sensors in the same scene, etc. can be used to calculate the difficulty coefficient. The following will be introduced respectively.

[0130] Method one, similarity or difference between data collected by the same sensor at adjacent time points

[0131] In one possible embodiment, the difficulty coefficient can be calculated based on the similarity or difference between the data collected by the same sensor at adjacent time points. Specifically, the multiple frames of perception data include first frame perception data and second frame perception data, which are collected by the same sensor at different time points. The first frame perception data and the second frame perception data are input into the perception model respectively, and the first perception result and the second perception result are output. The first perception result includes information of at least one first element extracted from the first frame perception data, and the second perception result includes information of at least one second element extracted from the second perception data. Then, the difficulty coefficient is determined according to the similarity between the first perception result and the second perception result. The similarity and the difficulty coefficient are usually positively correlated.

[0132] The different time points mentioned above can be the time points between the adjacent frames collected by the same sensor, or different time points when the same sensor collects the same scene, etc. That is, the data collected by the same sensor at different time points contains the same scene.

[0133] Generally, the first element and the second element correspond to the same or similar entities in the environment, i.e., the data collected by the same sensor at different time instants for the same entity can be compared, so as to represent whether the perception of the same sensor for the same entity at different time instants jumps or drifts, etc., by the similarity between the perception results of the entity at different time instants, and measure the difficulty of the perception of the entity in the environment. Therefore, in the embodiments of the present application, the difficulty coefficient in the current scene can be measured based on the similarity or difference between the perception data collected by the same sensor at different time instants in the same scene, so as to accurately identify the difficult example scene.

[0134] In a possible implementation, the similarity between the information of the at least one first element and the information of the at least one second element can be calculated; and the similarity between the first perception result and the second perception result is obtained according to the similarity between the information of the at least one first element and the information of the at least one second element. Therefore, in the embodiments of the present application, the similarity or difference between the elements detected by the perception model from different frames can be calculated, so as to calculate the difficulty coefficient from the dimension of the perception elements.

[0135] Specifically, the similarity between the information of each of the at least one first element and the information of the corresponding second element can be calculated. In the embodiments of the present application, the similarity or difference can be calculated from the dimensions of the positions or attributes of the elements, so as to determine whether there is a position jump or attribute jump of the elements in the data collected by the same sensor at different time instants for the same scene, so as to subsequently determine the difficult example scene.

[0136] Mode two, based on the similarity or difference between the different data collected by different sensors in the same scene

[0137] In a possible implementation, the difficulty coefficient can also be calculated based on the similarity or difference between the data collected by different sensors in the same scene. For example, the third frame of perception data and the fourth frame of perception data can be input into the perception model respectively, and the third perception result and the fourth perception result are output, the third perception result includes the information of at least one third element extracted from the third frame of perception data, and the fourth perception result includes the information of at least one fourth element extracted from the fourth frame of perception data, the difficulty coefficient is determined according to the similarity between the third perception result and the fourth perception result, and the third frame of perception data and the fourth frame of perception data are the data collected by different sensors in the same scene.

[0138] Generally, the third element corresponds to the same or similar entity in the environment corresponding to the fourth element. In the embodiments of the present application, the similarity or difference between the elements of different data collected by different sensors in the same scene can be calculated, which can represent the perception ability in the current scene. For example, the perception result of the data collected by some sensors can be inaccurate, resulting in a low similarity or a large difference between the perception of the data collected by different sensors, which is a difficult example scene. Thus, the difficult example scene can be identified based on the similarity or difference between the different data collected by different sensors in the same scene.

[0139] In a possible implementation, the similarity between the information of the at least one third element and the information of the at least one fourth element is calculated; and the similarity between the third perception result and the fourth perception result is obtained according to the similarity between the information of the at least one third element and the information of the at least one fourth element. Thus, in the embodiments of the present application, the perception similarity or difference of the data collected by different sensors can be calculated from the dimension of the perception elements, so as to calculate the difficulty coefficient to identify the difficult example scene subsequently.

[0140] Method three, combining the similarity or difference between the data collected by the same sensor at adjacent time and the similarity or difference between the different data collected by different sensors in the same scene

[0141] In this method, the difficulty coefficients corresponding to the two kinds of differences can be calculated. For convenience of distinction, the difficulty coefficient calculated based on the similarity or difference between the data collected by the same sensor at adjacent time is referred to as a first difficulty coefficient, and the difficulty coefficient calculated based on the similarity or difference between the different data collected by different sensors in the same scene is referred to as a second difficulty coefficient. The first difficulty coefficient and the second difficulty coefficient are weighted and fused, and the final output difficulty coefficient is obtained.

[0142] Thus, in the embodiments of the present application, the difficulty coefficient of the current scene can be measured from multiple dimensions, such as the similarity or difference between the data collected by the same sensor at adjacent time and the similarity or difference between the different data collected by different sensors in the same scene, so as to more accurately identify the difficult example scene.

[0143] Optionally, in the case of identifying the difficult example scene, the difficult example information can be sent to the cloud platform, so that the cloud platform trains the model based on the difficult example information sent by one or more vehicles, obtains a model that can output more accurate detection results in the difficult example scene, and distributes the model to the vehicle end, so that the vehicle end can realize more accurate detection in the difficult example scene based on the received model, and improve the intelligent driving effect of the vehicle end in the difficult example scene.

[0144] Optionally, in a possible scenario, the vehicle end can also adjust the driving decision of the vehicle according to the calculated difficulty coefficient. For example, in the process of intelligent driving of the vehicle, if the difficulty coefficient calculated in a certain scene is high, such as higher than a preset value, the intelligent driving function of the vehicle can be adjusted, such as reminding the user to take over the vehicle or downgrading the intelligent driving level, so as to improve the driving safety of the vehicle.

[0145] The foregoing introduces the method flow provided by the present application. In the following, the method flow provided by the present application is further introduced in detail in combination with the system architecture and specific application scenarios provided by the present application.

[0146] Referring to FIG. 4, the application scenario provided by the present application is shown.

[0147] The scenario includes a vehicle end and a cloud end. The vehicle end deploys a perception model and a difficult example detection module. The perception model can detect elements of the input data and output information of the detected elements. The difficult example detection module can be used to identify whether the current scene is a difficult example scene based on the element detection result output by the perception model.

[0148] Vehicle end side: One or more sensors, such as image sensors (or cameras), lidar, or ultrasonic radar, can be used to collect data. The collected data is used as the input of the perception model to obtain the output result. Then, the difficult example detection module set by the vehicle end identifies whether the current scene is a difficult example scene based on the perception result output by the perception model. After identifying the difficult example scene, the difficult example information, i.e., the information containing the difficult example scene, is uploaded to the cloud end.

[0149] Cloud end side: receiving difficult example information sent by at least one vehicle; labeling the difficult example information to obtain corresponding labeled data; training a model according to the difficult example information and the labeled data to obtain a perception model; and sending the perception model to at least one vehicle. That is, when the cloud end receives one or more difficult example information uploaded by one or more vehicles, it can perform model inference on the received difficult example information, and perform between the output result of the model inference, to obtain the true value corresponding to the difficult example. And use the difficult example information and the corresponding true value to train the model to obtain the trained perception model, and publish the trained perception model to the vehicle end.

[0150] In addition, the vehicle end side: after receiving the perception model from the cloud end, the perception model can be deployed locally. For example, the perception model can replace the existing perception model, so that a perception model with stronger perception capability is used for environment perception.

[0151] In the embodiments of the present application, a module for mining difficult examples is arranged at the vehicle end. Road structure difficult example mining and production functions are important links in the intelligent driving data platform. Active road structure difficult example identification can improve the efficiency of data use, save a large amount of labeling cost, and quickly improve the identification ability of road structure difficult examples. That is, valuable road structure difficult example data can be automatically mined, the cost of manpower and labeling for difficult example construction is significantly reduced, and the vehicle-end perception ability of road structure difficult examples is quickly improved.

[0152] The method provided in the present application can calculate the difficulty coefficient from the time domain and / or the spatial domain.

[0153] Referring to FIG. 5, the flowchart of another difficult example identification method provided in the present application is as follows.

[0154] 501, data acquisition.

[0155] Data acquisition can be performed by the sensors arranged in the vehicle, which can specifically include data collected by one or more sensors at the same or different time points. For example, it can include data collected by the same sensor at different time points, and can also include data collected by different sensors.

[0156] The sensors arranged in the vehicle can include but are not limited to lidar, image sensor, or radar, etc. In the following, the laser radar and image sensor arranged in the vehicle are taken as examples to illustrate the data collection. The laser radar and image sensor mentioned below can also be replaced by other sensors that can be used to perceive the environment, such as ultrasonic radar, infrared sensor, or depth sensor, etc. The specific sensor can be determined according to the actual application scenario.

[0157] For example, during the driving of the vehicle, the laser radar and image sensor arranged in the vehicle can be used to collect the perception data of the current scene, such as collecting image or point cloud data containing the current scene of the vehicle.

[0158] Optionally, after the data collected by each sensor is obtained, the calibration parameters of each sensor can be used to map the data collected by each sensor into a matrix, denoted as M.

[0159] 502, road element extraction.

[0160] Taking the extraction of elements from the collected image and point cloud data as an example, various types of perception models can be set accordingly, such as setting a visual perception model for the image, setting a road structure recognition model for the point cloud data, etc. A multi-modal perception model can also be directly deployed, with the image and point cloud data serving as inputs to the multi-modal perception model, so as to output one or more elements extracted from the image and one or more elements extracted from the point cloud data.

[0161] Specifically, the position and attribute information of the extracted elements can be output, and the attribute information can specifically include the size, shape, type, color, etc. of the elements.

[0162] Generally, elements in a 2D space can be extracted from an image, and elements in a 3D space can be extracted from point cloud data. Of course, if the image carries depth information, elements in a 3D space can also be extracted from the image.

[0163] Exemplarily, the road structure recognition capability of the multi-modal perception model can be used to analyze the road area and other areas in the current scene. Specifically, the multi-modal perception model can perform point cloud segmentation or visual segmentation to segment the point cloud and image into one or more elements, as shown in FIG. 6. For example, the segmented elements can include but are not limited to movable vehicles and pedestrians on the road, static environmental obstacles, and road markings. Generally, dynamic targets have less impact on difficult example scenarios for the vehicle end, such as pedestrians and vehicles in the environment that can leave, so they have less reference significance for the next driving decision. Therefore, dynamic elements in the scene can be filtered out, and static elements in the scene can be retained. For example, dynamic elements such as pedestrians and vehicles can be filtered out, and static environmental obstacles (such as trees or fences, etc.) and road markings (lane lines, road edges, or road arrows, etc.) can be retained to identify the road object position area and object attribute of the static elements.

[0164] 503, time domain difficulty coefficient calculation.

[0165] Specifically, for difficulty calculation in the time domain dimension, the input data can include perception results corresponding to data collected by the same sensor at different times and containing the current scene.

[0166] As the t-th frame and the t+n-th frame collected by a sensor can be taken as the input of the multi-modal perception model, the t-th frame perception result and the t+n-th frame perception result are output. The positions and attributes of road elements are extracted from the t-th frame perception result and the t+n-th frame perception result respectively. The elements extracted from the t-th frame perception result and the t+n-th frame perception result are analyzed, such as calculating the similarity or difference, and the difficulty coefficient is calculated based on the similarity or difference. For example, the difficulty coefficient and the similarity are negatively correlated, that is, the higher the similarity, the lower the jump rate of the perception result, the more accurate the result of recognition, and the higher the similarity, the more likely the perception result jumps, and the result of element perception may be inaccurate. For another example, the difficulty coefficient and the difference are positively correlated, the greater the difference, the greater the difficulty coefficient, indicating that the perception of the element is less accurate; the smaller the difference, the smaller the difficulty coefficient, indicating that the perception of the element is more accurate.

[0167] For example, the difficulty coefficient calculation process of the time domain channel can be as shown in FIG. 7.

[0168] Among them, the input is the t-th time road cognition result and the t+n-th time road cognition result, that is, the perception result output after taking the t-th frame perception data and the t+n-th frame perception data collected by the same sensor as the input of the multi-modal perception model.

[0169] Subsequently, road elements are extracted from the t-th time road cognition result and the t+n-th time road cognition result respectively, including the positions and attributes of the elements. In addition, for the detection area, it is determined whether there is a static road element to be analyzed, and the attributes of the static road element are obtained when there is a static road element, such as the shape, size, length and other information of the road yellow solid line, left arrow, etc.

[0170] Subsequently, the positions and attributes of the road elements extracted at the t-th time and the t+n-th time are analyzed. The similarity between the perception results of the same position at different times can be analyzed, such as comparing the similarity between the elements in a certain area in the perception results at different times.

[0171] Optionally, for long static elements such as lane lines and road edges in the road, tracking recognition can be added to identify whether there is a jump in position or attribute of the long static element, such as a jump rate or a confidence level of generating a jump. Generally, if there is a jump in position or attribute of the element, it may indicate that the perception result of the data collected by the sensor is inaccurate.

[0172] Subsequently, the difficulty coefficient can be calculated based on the analysis result, such as taking the difficulty coefficient as h1. The difficulty coefficient can be calculated based on the jump rate, the higher the jump rate, the higher the difficulty coefficient, and the lower the jump rate, the lower the difficulty coefficient.

[0173] Therefore, in the embodiments of the present application, the perception difficulty of the vehicle-end perception system for the current scene can be calculated from the time dimension, or the difficulty of the vehicle to make a driving decision in the current scene, so that the difficult example scene can be accurately identified subsequently.

[0174] In addition, the difficult example duration can also be calculated. The difficult example duration can include the duration when the difficulty coefficient is lower than the preset value, that is, the perception effect of the vehicle for the current environment is poor in this duration, so that a series of difficult example scenes are determined from the time dimension.

[0175] 504, spatial difficulty coefficient calculation.

[0176] The input of the difficulty coefficient calculation in the spatial dimension can include the data collected by different sensors for the same scene. As in the step 502 described above, the data collected by different sensors can be used as the input of the multi-modal perception model, and the perception results corresponding to different sensors are output. The difficulty coefficient is calculated based on the similarity or matching degree between the perception results corresponding to different sensors, and the difficulty coefficient is represented as h2.

[0177] For example, the difficulty coefficient calculation process of the spatial channel can be as shown in FIG. 8.

[0178] Here, taking a plurality of sensors as an example, including image data collected by cameras (cameral1 to cameral x) and point cloud data collected by a laser radar. Among them, cameral1 corresponds to 2D space static element perception result 1, cameral x corresponds to 2D space static element perception result 2, and point cloud data corresponds to static element position area perception result. Since the data based on the perception is usually an element in a 3D space, in order to facilitate comparison, the static element in the 3D space can be projected into a 2D space, such as based on the position corresponding to the static element in the 3D space and the calibration parameters of the laser radar, the static element is projected into the 2D space.

[0179] Then the similarity between the perception results corresponding to different sensors can be calculated, which can be specifically calculating the similarity between the same static elements, and calculating the corresponding difficulty coefficient based on the similarity between the static elements.

[0180] 505, difficult example identification.

[0181] After calculating the difficulty coefficient from the time dimension and / or the spatial dimension respectively, the difficult example scene can be identified based on the difficulty coefficient.

[0182] Optionally, if the difficulty coefficient h1 is calculated only from the time domain dimension, h1 can be directly used as the perception difficulty coefficient; if the difficulty coefficient h2 is calculated only from the space domain dimension, h2 can be directly used as the perception difficulty coefficient; if the difficulty coefficients h1 and h2 are calculated from both the time domain dimension and the space domain dimension, h1 and h2 can be fused by weighting, and the specific weight can be an empirical value, a pre-set fixed value, etc., and the final perception difficulty coefficient is output.

[0183] In the embodiments of the present application, the perception result similarity or difference between different perception data is used to represent the perception ability of the vehicle end, so that the perception ability of the vehicle end can be accurately represented, and the difficult example scene can be accurately identified, so as to make corresponding driving decisions for the difficult example scene. Moreover, the perception ability of the vehicle end can be measured from the time domain dimension and / or the space domain dimension, so that the difficult example scene can be identified from the time domain and / or the space domain dimension. In the time domain dimension, the information of time sequence and the static road stable information are fully utilized to mine the scenes with unstable perception and error scenes. In the space domain dimension, the detection results of different sensors are mutually verified to locate the difficult example scene of static perception missing and mis-detection from more dimensions.

[0184] Moreover, compared with the use of a high-precision map collection vehicle to collect difficult example scenes and the use of a shadow mode to collect difficult example scenes, the method provided in the embodiments of the present application actively mines and reports the difficult example scenes of the vehicle end based on the static road inherent characteristics (with stability) combined with the inconsistency and instability of the time sequence output of the model and the mutual verification of different sensors, so that the difficult examples can be located from a large amount of unlabeled point cloud data, the complex road scene that cannot be accurately perceived by the model can be actively discovered in advance, and the data of this scene can be reported after automatic labeling and used for model training. Meanwhile, the true value can be directly issued to the vehicle end for perception identification assistance.

[0185] The foregoing describes the method flow provided by the present application, and the structure of the device for executing the foregoing method flow is described below.

[0186] Referring to FIG. 9, the structure of a difficult example identification device provided by the present application is shown in the figure. The difficult example identification device includes:

[0187] The data acquisition module 901 is configured to acquire a perception data set, the perception data set including at least one sensor in the vehicle collecting a current scene, and the perception data including multiple frames of perception data.

[0188] The perception module 902 is configured to input the multiple frames of perception data into a perception model to obtain multiple perception results, and the perception model is configured to detect elements in the input data.

[0189] The difficulty instance detection module 903 is configured to determine a difficulty coefficient according to the similarity between the plurality of perception results, and the difficulty coefficient is used to measure the difficulty of determining the vehicle driving decision in the current scene.

[0190] In a possible implementation, the multi-frame perception data includes first-frame perception data and second-frame perception data, and the first-frame perception data and the second-frame perception data are collected by the same sensor at different time points in the current scene.

[0191] The perception module 902 is specifically configured to input the first-frame perception data and the second-frame perception data into the perception model respectively, and output first perception results and second perception results, wherein the first perception results include information of at least one first element extracted from the first-frame perception data, and the second perception results include information of at least one second element extracted from the second-frame perception data.

[0192] The difficulty instance detection module 903 is specifically configured to determine the difficulty coefficient according to the similarity between the first perception results and the second perception results.

[0193] In a possible implementation, the difficulty instance detection module 903 is further configured to calculate the similarity between the information of the at least one first element and the information of the at least one second element, and obtain the similarity between the first perception results and the second perception results according to the similarity between the information of the at least one first element and the information of the at least one second element.

[0194] In a possible implementation, the difficulty instance detection module 903 is specifically configured to calculate the similarity between the information of each of the at least one first element and the information of the corresponding second element.

[0195] In a possible implementation, the multi-frame perception data includes third-frame perception data and fourth-frame perception data, and the third-frame perception data and the fourth-frame perception data are collected by different sensors.

[0196] The perception module 902 is further configured to input the third-frame perception data and the fourth-frame perception data into the perception model respectively, and output third perception results and fourth perception results, wherein the third perception results include information of at least one third element extracted from the third-frame perception data, and the fourth perception results include information of at least one fourth element extracted from the fourth-frame perception data.

[0197] The difficulty instance detection module 903 is specifically configured to determine the difficulty coefficient according to the similarity between the third perception results and the fourth perception results.

[0198] In a possible implementation, the difficulty example detection module 903 is further configured to: calculate a similarity between the information of the at least one third element and information of at least one fourth element; and obtain a similarity between a third perception result and a fourth perception result according to the similarity between the information of the at least one third element and the information of the at least one fourth element.

[0199] In a possible implementation, the difficulty coefficient is calculated based on information of a static element in the plurality of perception results, and the static element is an element with a moving speed lower than a preset moving speed in the current scene.

[0200] In a possible implementation, the apparatus further includes a transceiver module 904 configured to: in a case where the difficulty coefficient is greater than a first threshold, send, to a cloud, difficulty example information, where the difficulty example information includes the difficulty coefficient.

[0201] In a possible implementation, the difficulty example detection module 903 is further configured to: obtain a difficulty example duration, where the difficulty example duration is a duration during which the difficulty coefficient is greater than the first threshold, and the difficulty example information further includes the difficulty example duration.

[0202] In a possible implementation, the transceiver module 904 is further configured to receive a perception model sent by the cloud, where the perception model is a model trained using the difficulty example information.

[0203] In a possible implementation, the apparatus further includes a driving decision module 905 configured to determine a driving decision of the vehicle according to the difficulty coefficient.

[0204] Referring to FIG. 10, the present application provides a schematic diagram of a cloud platform structure, which includes:

[0205] A transceiver module 1001 is configured to receive difficulty example information sent by at least one vehicle, and the difficulty example information can be generated in the manner described in the first aspect or any possible implementation of the first aspect.

[0206] A labeling module 1002 is configured to label the difficulty example information to obtain corresponding labeled data.

[0207] A training module 1003 is configured to train a model according to the difficulty example information and the labeled data to obtain a perception model.

[0208] The transceiver module 1001 is further configured to send the perception model to the at least one vehicle, and the vehicle can locally deploy the received perception model.

[0209] As shown in FIG. 11, the present application provides a hardware structure schematic diagram of a computing device 110. The computing device 110 can be used to implement the steps of the method or the cloud platform in FIGS. 3 to 8.

[0210] The computing device 110 shown in FIG. 11 can include a processor 1101, a memory 1102, a communication interface 1103, and a bus 1104. The processor 1101, the memory 1102, and the communication interface 1103 can be connected through the bus 1104.

[0211] The processor 1101 is a control center of the computing device 110, and can be a general central processing unit (CPU), or other general-purpose processors, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc., and can specifically include a GPU or an NPU, etc., and can be adaptively set according to an actual application scenario.

[0212] As an example, the processor 1101 can include one or more CPUs, and can also include other processors, such as the CPU, NPU, or GPU shown in FIG. 11, etc.

[0213] The memory 1102 can be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, and can also be an electrically erasable programmable read-only memory (EEPROM), a magnetic disk storage medium or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.

[0214] In one possible implementation, the memory 1102 can exist independently of the processor 1101. The memory 1102 can be connected to the processor 1101 through the bus 1104, and is used to store data, instructions, or program codes. When the processor 1101 invokes and executes the instructions or program codes stored in the memory 1102, the method provided by the embodiments of the present application can be implemented, for example, the methods shown in FIGS. 3 to 8.

[0215] In another possible implementation, the memory 1102 can also be integrated with the processor 1101.

[0216] The communication interface 1103 is configured to connect the computing device 110 to other devices through a communication network, which can be an Ethernet network, a radio access network (RAN), a wireless local area network (WLAN), or the like. The communication interface 1103 can include a receiving unit for receiving data and a transmitting unit for transmitting data.

[0217] The bus 1104 can be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, an extended industry standard architecture (EISA) bus, or the like. The bus can be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, only one thick line is shown in FIG. 11, but it does not mean that there is only one bus or only one type of bus.

[0218] It should be noted that the structure shown in FIG. 11 does not constitute a limitation on the computing device 110. In addition to the components shown in FIG. 11, the computing device 110 can include more or fewer components than shown, or combine certain components, or different arrangement of components.

[0219] From the above description of the embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software and the necessary general hardware, and of course can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, and the like. Generally, functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure for implementing the same function can also be various, such as analog circuits, digital circuits, or special circuits. However, for the present application, software program implementation is a better embodiment. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc., including a plurality of instructions for causing an apparatus (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application.

[0220] In the above embodiments, all or part can be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part can be implemented in the form of a computer program product.

[0221] The computer readable storage medium in the embodiments of the present application stores a program for training a model or performing an inference task, and when the program is run on a computer, the computer is caused to perform all or part of the steps of the method described in the foregoing embodiments of FIG. 3 to FIG. 8.

[0222] The digital processing chip in the embodiments of the present application integrates a circuit for implementing the processor or the function of the processor and one or more interfaces. When the digital processing chip integrates a memory, the digital processing chip can complete the method steps of any one or more of the foregoing embodiments. When the digital processing chip does not integrate a memory, the digital processing chip can be connected with an external memory through a communication interface. The digital processing chip implements the method steps of any one or more of the foregoing embodiments according to the program code stored in the external memory.

[0223] The computer program product in the embodiments of the present application includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer readable storage medium can be any available medium that a computer can store or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)), etc.

[0224] The data comparison device provided by the embodiments of the present application can be a chip, which comprises a processing unit, for example, a processor, and a communication unit, for example, an input / output interface, a pin, a circuit or the like. The processing unit can execute computer execution instructions stored in a storage unit, so that the chip in the server executes the method described in the embodiments shown in FIG. 3 to FIG. 8. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache or the like. The storage unit can also be a storage unit outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or the like.

[0225] Specifically, the foregoing processing unit or processor can be a central processing unit (CPU), a neural-network processing unit (NPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, a discrete gate or transistor logic device, a discrete hardware component, or the like. The general-purpose processor can be a microprocessor or any conventional processor or the like.

[0226] In addition, it should be noted that the apparatus embodiments described above are merely illustrative, and the units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. According to actual needs, part or all of the modules can be selected to achieve the purpose of the embodiments. In addition, in the apparatus embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which can be implemented as one or more communication buses or signal lines.

[0227] Those skilled in the art can clearly understand that the application can be implemented by means of software plus necessary universal hardware, and of course can also be implemented by means of dedicated hardware including special-purpose integrated circuits, special-purpose CPUs, special-purpose memories, special-purpose components, etc. Generally, any function completed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure for implementing the same function can also be various, such as analog circuits, digital circuits, or special-purpose circuits, etc. However, for the present application, software program implementation is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a readable storage medium, such as a floppy disk, a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc., and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments of the present application.

[0228] In the above embodiments, all or part can be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part can be implemented in the form of a computer program product.

[0229] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)), etc.

[0230] The terms "first", "second", and the like in the description and in the claims of the present application and above drawings are used for distinguishing between similar objects and not necessarily for describing a specific sequential or chronological order. It is to be understood that the terms so used are interchangeable under appropriate circumstances such that the embodiments of the application described herein are, for example, capable of orderly or inverse order, depending upon the circumstances. The term "and / or" in the present application is merely used to represent an association between associated objects, and it is possible that three relationships exist, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone. In addition, the character " / " in the present application generally represents an "or" relationship between the associated objects. Furthermore, the terms "comprising" and "having" and any variations thereof are intended to cover a non-exclusive inclusion, for example, a process, method, system, product or device that includes a list of steps or modules as an example is not necessarily limited to those steps or modules that are clearly listed, but can include other steps or modules that are not clearly listed or inherent to such process, method, product or device. The naming or numbering of steps in the present application does not mean that the steps in the method flow must be executed in the time / logical order indicated by the naming or numbering, and the named or numbered flow steps can be executed in a different order according to the technical purpose to be achieved, as long as the same or similar technical effects can be achieved. The division of modules in the present application is a logical division, and in actual application, there can be another division manner, for example, multiple modules can be combined or integrated in another system, or some features can be ignored or not executed, in addition, the coupling or direct coupling or communication connection between the displayed or discussed modules can be through some ports, and the indirect coupling or communication connection between the modules can be electrical or other similar forms, which are not limited in the present application. Furthermore, the modules or sub-modules described as separate components can or can not be physically separated, and can or can not be physical modules, or can be distributed to multiple circuit modules, and part or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of the present application.

Claims

1. A difficult case identification method characterized by comprising: The method comprises: obtaining a set of perception data, the set of perception data comprising a plurality of frames of perception data collected by at least one sensor in a vehicle from a current scene; inputting the plurality of frames of perception data into a perception model to obtain a plurality of perception results, the perception model being configured to detect elements in the input data; determining a difficulty coefficient based on a similarity between the plurality of perception results, the difficulty coefficient being used to measure a perception ability of the vehicle in the current scene.

2. The method of claim 1, wherein, The plurality of frames of perception data comprises a first frame of perception data and a second frame of perception data, the first frame of perception data and the second frame of perception data being collected by a same sensor at different time points from the current scene; The inputting the plurality of frames of perception data into the perception model to obtain the plurality of perception results comprises: inputting the first frame of perception data and the second frame of perception data into the perception model respectively to output a first perception result and a second perception result, the first perception result comprising information of at least one first element extracted from the first frame of perception data, and the second perception result comprising information of at least one second element extracted from the second frame of perception data; The determining the difficulty coefficient based on the similarity between the plurality of perception results comprises: determining the difficulty coefficient based on a similarity between the first perception result and the second perception result.

3. The method of claim 2, wherein, The method further comprises: calculating a similarity between the information of the at least one first element and the information of the at least one second element; obtaining the similarity between the first perception result and the second perception result based on the similarity between the information of the at least one first element and the information of the at least one second element.

4. The method of claim 3, wherein, The calculating the similarity between the information of the at least one first element and the information of the at least one second element comprises: calculating a similarity between a position and / or an attribute of the information of each of the at least one first element and the information of a corresponding second element.

5. The method according to any one of claims 1-4, characterized in that, The plurality of frames of perception data comprises a third frame of perception data and a fourth frame of perception data, the third frame of perception data and the fourth frame of perception data being collected by different sensors; The inputting the plurality of frames of perception data into the perception model to obtain the plurality of perception results further comprises: inputting the third frame of perception data and the fourth frame of perception data into the perception model respectively to output a third perception result and a fourth perception result, the third perception result comprising information of at least one third element extracted from the third frame of perception data, and the fourth perception result comprising information of at least one fourth element extracted from the fourth frame of perception data; The determining the difficulty coefficient based on the similarity between the plurality of perception results comprises: determining the difficulty coefficient based on a similarity between the third perception result and the fourth perception result.

6. The method of claim 5, wherein, The method further comprises: calculating a similarity between the information of the at least one third element and the information of the at least one fourth element; The similarity between the third perception result and the fourth perception result is obtained according to the similarity between the information of the at least one third element and the information of the at least one fourth element.

7. The method according to any one of claims 1 to 6, characterized in that, The difficulty coefficient is calculated based on information of static elements in the plurality of perception results, the static elements being elements in the current scene with a moving speed lower than a preset moving speed.

8. The method according to any one of claims 1 to 7, characterized in that, The method further comprises: When the difficulty coefficient is greater than a first threshold, sending, to a cloud, difficulty example information, the difficulty example information including the difficulty coefficient.

9. The method of claim 8, wherein, The method further comprises: Obtaining a difficulty example duration, the difficulty example duration being a duration during which the difficulty coefficient is greater than the first threshold, the difficulty example information further including the difficulty example duration.

10. The method according to claim 8 or 9, characterized in that, The method further comprises: Receiving a perception model sent by the cloud, the perception model being a model trained using the difficulty example information.

11. The method according to any one of claims 1-10, characterized in that, The method further comprises: Determining a driving decision of the vehicle according to the difficulty coefficient.

12. A difficult case identification apparatus characterized by comprising: Comprise: A data collection module configured to obtain a perception data set, the perception data set including a plurality of frames of perception data collected by at least one sensor in a vehicle on a current scene; A perception module configured to input the plurality of frames of perception data into a perception model to obtain a plurality of perception results, the perception model being configured to detect elements in the input data; A difficulty example detection module configured to determine a difficulty coefficient according to the similarity between the plurality of perception results, the difficulty coefficient being configured to measure the perception ability of the vehicle in the current scene.

13. The apparatus of claim 12, wherein, The plurality of frames of perception data include first frame of perception data and second frame of perception data, the first frame of perception data and the second frame of perception data being collected by a same sensor at different time points on the current scene; The perception module is specifically configured to: Input the first frame of perception data and the second frame of perception data into the perception model respectively to output first perception result and second perception result, the first perception result including information of at least one first element extracted from the first frame of perception data, the second perception result including information of at least one second element extracted from the second frame of perception data; The difficulty example detection module is specifically configured to: Determine the difficulty coefficient according to the similarity between the first perception result and the second perception result.

14. The apparatus of claim 13, wherein, The difficulty example detection module is further configured to: Calculate the similarity between the information of the at least one first element and the information of the at least one second element; Obtain the similarity between the first perception result and the second perception result according to the similarity between the information of the at least one first element and the information of the at least one second element.

15. The apparatus of claim 14, wherein, The difficulty example detection module is specifically configured to: Calculate the similarity between the information of each first element in the at least one first element and the information of a corresponding second element.

16. The apparatus of any one of claims 12-15, wherein, The plurality of frames of perception data include third frame of perception data and fourth frame of perception data, the third frame of perception data and the fourth frame of perception data being collected by different sensors; The perception module is further configured to: inputting the third frame of perception data and the fourth frame of perception data into the perception model respectively, and outputting a third perception result and a fourth perception result, wherein the third perception result comprises information of at least one third element extracted from the third frame of perception data, and the fourth perception result comprises information of at least one fourth element extracted from the fourth frame of perception data; The difficulty detection module is specifically configured to: determine the difficulty coefficient according to a similarity between the third perception result and the fourth perception result.

17. The apparatus of claim 16, wherein, The difficulty detection module is further configured to: calculate a similarity between the information of the at least one third element and the information of the at least one fourth element; obtain the similarity between the third perception result and the fourth perception result according to the similarity between the information of the at least one third element and the information of the at least one fourth element.

18. The apparatus of any one of claims 12-17, wherein, The difficulty coefficient is calculated based on information of a static element in the plurality of perception results, and the static element is an element with a moving speed lower than a preset moving speed in the current scene.

19. The apparatus of any one of claims 12-18, wherein, The device further comprises: a transceiver module configured to send, to a cloud, difficulty information when the difficulty coefficient is greater than a first threshold value, wherein the difficulty information comprises the difficulty coefficient.

20. The apparatus of claim 19, wherein, The difficulty detection module is further configured to: obtain a difficulty duration, wherein the difficulty duration is a duration during which the difficulty coefficient is greater than the first threshold value, and the difficulty information further comprises the difficulty duration.

21. The device of claim 19 or 20, wherein the transceiver module is further configured to receive a perception model sent by the cloud, wherein the perception model is a model trained using the difficulty information.

22. The apparatus of any one of claims 12-21, wherein, The device further comprises: a driving decision module configured to determine a driving decision of the vehicle according to the difficulty coefficient.

23. A difficult case detection apparatus characterized by comprising: The device comprises a processor and a memory coupled to the processor, and the memory stores a program, and when the program stored in the memory is executed by the processor, the steps of the method in any one of claims 1-11 are implemented.

24. A computer readable storage medium comprising a program which, when executed by a processing unit, performs the steps of the method of any one of claims 1-11.

25. A computer program product, characterised in that, The computer program product comprises software code for performing the steps of the method of any one of claims 1-11. The computer program product comprises software code for performing the steps of the method of any one of claims 1-11.

Images