Information collection method and device, computer device and storage medium

Abstract

The application relates to an information collection method and device, computer equipment and a storage medium. First sensor data of a road object collected by a first sensor is obtained. The first sensor data is input into a preset deep learning model to obtain second feature information of the road object. The deep learning model is obtained by training data collected by the first sensor and a second sensor in the same time period and the same scene. The types of the first feature information and the second feature information are different. According to the method, the computer equipment can obtain the second feature information of the road object through the first sensor data and the deep learning model, and the system reliability is improved. The first feature information and the second feature information of the road object are obtained by deploying one first sensor, and the flexibility of sensor deployment is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of sensors, in particular to an information collection method and device, computer equipment and a storage medium. BACKGROUND

[0002] With the development of automatic driving, vehicle-road cooperation and other technologies, sensors have been more and more widely used. In an automatic driving system or a vehicle-road cooperation system, different types of feature information of a target object need to be collected so that the system can make more accurate judgments based on the feature information.

[0003] In order to collect different types of feature information, the above system can deploy multiple sensors, for example, a camera can be deployed in the system to collect color features of the target object; a laser radar can also be deployed in the system to collect distance features of the target object, etc.

[0004] However, when one of the sensors fails, the system cannot obtain the corresponding type of feature information through the sensor, which leads to abnormal operation of the automatic driving system or the vehicle-road cooperation system and even causes safety accidents. SUMMARY

[0005] Therefore, it is necessary to provide an information collection method, device, computer equipment and storage medium capable of improving the reliability of the system to solve the above technical problems.

[0006] An information collection method, the method comprising:

[0007] obtaining first sensor data of a road object collected by a first sensor;

[0008] inputting the first sensor data into a preset deep learning model to obtain second feature information of the road object; the deep learning model is obtained by training data collected by the first sensor and a second sensor in the same time period and the same scene; and the first feature information and the second feature information are of different types.

[0009] In one embodiment, before obtaining the first sensor data of the road object collected by the first sensor, the method further comprises:

[0010] obtaining first historical data collected by the first sensor and second historical data collected by the second sensor in the same time period and the same scene;

[0011] performing spatio-temporal synchronization processing on the first historical data collected by the first sensor and the second historical data collected by the second sensor to obtain a spatio-temporal mapping relationship between the first historical data and the second historical data;

[0012] The deep learning model is established based on the spatiotemporal mapping relationship between the first historical data and the second historical data.

[0013] In one of the embodiments, the first historical data includes a plurality of first data frames, and the second historical data includes a plurality of second data frames. The spatiotemporal mapping relationship between the first historical data collected by the first sensor and the second historical data collected by the second sensor is obtained through spatiotemporal synchronization processing, and includes:

[0014] The first historical data and the second historical data are time-synchronized to obtain a plurality of pairs of time-synchronized data frames. Each pair of data frames includes a first data frame and a second data frame that are synchronized in sampling time.

[0015] The first data frame and the second data frame in each pair of data frames are subjected to coordinate system conversion to obtain a plurality of pairs of spatially synchronized data. Each pair of data includes first data in the first data frame and second data in the second data frame.

[0016] In one of the embodiments, the relative positions of the first sensor and the second sensor are fixed. The deep learning model is established based on the spatiotemporal mapping relationship between the first historical data and the second historical data, and includes:

[0017] The second historical data is subjected to feature extraction to obtain second feature information corresponding to the second data.

[0018] The first data is used as a training input sample, and the second feature information associated with the second data corresponding to the first data is used as a sample label of the training input sample to obtain a training data set.

[0019] The first data is used as an input of an initial deep learning model, and the second feature information associated with the second data is used as a reference output of the initial deep learning model. The initial deep learning model is trained to obtain the deep learning model.

[0020] In one of the embodiments, the first sensor is a laser radar, the second sensor is an image sensor, the first sampling point data is point cloud data of a road object, and the second sampling point data is image data of the road object. The second feature information is an RGB value of the road object. The first data is used as an input of an initial deep learning model, and the second feature information associated with the second data is used as a reference output of the initial deep learning model. The initial deep learning model is trained to obtain the deep learning model, and includes:

[0021] The point cloud data is used as an input of an initial deep learning model, and the RGB value corresponding to the image data is used as a reference output of the initial deep learning model. The initial deep learning model is trained to obtain a first deep learning model.

[0022] In one of the embodiments, the first sensor is an image sensor, the second sensor is a laser radar, the first sampling point data is image data of a road object, and the second sampling point data is point cloud data of the road object; the second feature information is depth information of the road object, and the depth information is used to represent a distance between the road object and the laser radar; the first data is taken as an input of an initial deep learning model, the second feature information associated with the second data is taken as a reference output of the initial deep learning model, the initial deep learning model is trained, and the deep learning model is obtained, including:

[0023] The image data is taken as an input of an initial deep learning model, the depth information corresponding to the point cloud data is taken as a reference output of the initial deep learning model, the initial deep learning model is trained, and a second deep learning model is obtained.

[0024] In one of the embodiments, the first historical data and the second historical data are time-synchronized to obtain a plurality of data frame pairs in time synchronization, including:

[0025] The first historical data and the second historical data are converted to the same time axis;

[0026] In the time axis, a first sampling time of each first data frame in the first historical data and a second sampling time of each second data frame in the second historical data are obtained;

[0027] A difference between the first sampling time and the second sampling time is calculated;

[0028] If the difference is less than a preset threshold, it is determined that the first data frame corresponding to the first sampling time and the second data frame corresponding to the second sampling time are a data frame pair.

[0029] In one of the embodiments, the sampling frequencies of the first historical data and the second historical data are in a multiple relationship.

[0030] In one of the embodiments, the first data frame and the second data frame in each data frame pair are converted in a coordinate system to obtain a spatially synchronized data pair, including:

[0031] According to a preset conversion matrix, the second data in the data frame pair is converted into the first sensor coordinate system to obtain a mapping coordinate of each second data in the first sensor coordinate system;

[0032] In the data frame pair, the first data corresponding to the mapping coordinate is obtained;

[0033] The second data corresponding to each mapping coordinate and the first data corresponding to the mapping coordinate are determined as a data pair.

[0034] An information collection method, the method comprising:

[0035] According to the current sensor state, determine whether the sensor switching condition is met;

[0036] If the sensor switching condition is met, the current sensor is taken as the second sensor, and the steps of the information collection method according to any one of claims 1-9 are executed to collect information.

[0037] An information collection device, the device comprising:

[0038] An acquisition module, configured to acquire first sensor data of a road object collected by a first sensor;

[0039] An input module, configured to input the first sensor data into a preset deep learning model to obtain second feature information of the road object; the deep learning model is obtained based on data collected by the first sensor and a second sensor in the same time period and the same scene; the first feature information and the second feature information are of different types.

[0040] A computer device, comprising a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0041] The first sensor data is input into a preset deep learning model to obtain second feature information of the road object; the deep learning model is obtained based on data collected by the first sensor and a second sensor in the same time period and the same scene; the first feature information and the second feature information are of different types.

[0042] A computer readable storage medium, having a computer program stored thereon, the computer program is executed by a processor to implement the following steps:

[0043] The first sensor data is input into a preset deep learning model to obtain second feature information of the road object; the deep learning model is obtained based on data collected by the first sensor and a second sensor in the same time period and the same scene; the first feature information and the second feature information are of different types.

[0044] The aforementioned information acquisition method, apparatus, computer equipment, and storage medium involve the computer equipment acquiring first sensor data of a road object collected by a first sensor, inputting the first sensor data into a preset deep learning model to obtain second feature information of the road object. The deep learning model is trained based on data collected by the first and second sensors within the same time period and scenario. The first and second feature information are of different types. Because the deep learning model is trained on data collected by the first and second sensors within the same time period and scenario, it possesses the ability to correlate the first sensor data with the second feature information. Furthermore, since the computer equipment inputs the first sensor data into the deep learning model, it can obtain the second feature information of the road object through the deep learning model, thereby avoiding system malfunctions caused by the second sensor's inability to obtain the second feature information, thus improving system reliability. Alternatively, when there are limitations on the number or weight of sensors in the system, the first and second feature information of the road object can be obtained by deploying a single first sensor, improving the flexibility of sensor deployment in the system. Attached Figure Description

[0045] Figure 1 This is an application environment diagram of the information acquisition method in one embodiment;

[0046] Figure 2 This is a flowchart illustrating an information acquisition method in one embodiment;

[0047] Figure 3 This is a flowchart illustrating an information acquisition method in one embodiment;

[0048] Figure 4 This is a flowchart illustrating the information acquisition method in another embodiment;

[0049] Figure 5 This is a flowchart illustrating the information acquisition method in another embodiment;

[0050] Figure 6 This is a flowchart illustrating the information acquisition method in another embodiment;

[0051] Figure 7 This is a flowchart illustrating the information acquisition method in another embodiment;

[0052] Figure 8 This is a flowchart illustrating the information acquisition method in another embodiment;

[0053] Figure 9 This is a flowchart illustrating the information acquisition method in another embodiment;

[0054] Figure 10 A structural block diagram of an information collection device in one embodiment;

[0055] Figure 11 A structural block diagram of an information collection device in another embodiment;

[0056] Figure 12 A structural block diagram of an information collection device in another embodiment;

[0057] Figure 13 A structural block diagram of an information collection device in another embodiment;

[0058] Figure 14 An internal structural diagram of a computer device in one embodiment. DETAILED DESCRIPTION

[0059] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0060] The information collection method provided by the present application can be applied in an application environment as shown in Figure 1 The computer device 100 is connected with the first sensor 200. The computer device 100 can be arranged in the first sensor 200 device or can be an independent device outside the first sensor 200. The first sensor 200 and the second sensor 300 can be sensors in a roadside system or sensors in a vehicle-mounted system, which are not limited herein. The first sensor 200 and the second sensor 300 can be, but are not limited to, a laser radar, a camera and a millimeter wave radar. The computer device 100 can be realized by an independent server or a server cluster composed of multiple servers.

[0061] In one embodiment, as shown in Figure 2 , an information collection method is provided. Taking the computer device 100 in Figure 1 as an example, the method comprises the following steps:

[0062] S101, acquiring first sensor data of a road object collected by a first sensor.

[0063] The first sensor can be a laser radar, a camera or a millimeter wave radar, and the type of the first sensor is not limited herein. The first sensor can be a sensor arranged in a roadside system or a sensor in a vehicle-mounted system, and the application scenario of the first sensor is not limited herein.

[0064] The first sensor data is data collected by the first sensor in the current scene and contains information of the road object. Different types of first sensor data can be obtained by different types of first sensors. The first sensor data can be point cloud data, image data, or video data, which is not limited herein.

[0065] The first sensor has the ability to obtain first feature information of the road object. Different types of first feature information can be obtained by different types of first sensors. The first feature information can be color features of the road object, distance features of the road object, speed features of the road object, pose features of the road object, surface texture features of the road object, etc. The type of the first feature information is not limited herein. For example, when the first sensor is an image sensor, it has the ability to obtain color features of the road object. When the first sensor is a laser radar, it has the ability to obtain distance features of the road object.

[0066] The road object can be a traffic participant in the road, such as a vehicle or a pedestrian. In addition, the road object can also be an obstacle in the road. The type of the road object is not limited herein.

[0067] The first feature information can also contain multiple types of features, including color features and surface texture features, or speed features and distance features, which are not limited herein. For example, when the first sensor is a laser radar, it can have the ability to obtain distance features and speed features of the road object at the same time.

[0068] Specifically, the computer device obtains first sensor data of the road object collected by the first sensor. The first sensor data can be received by the first sensor through wired connection or wireless connection, which is not limited herein.

[0069] The computer device can obtain the first sensor data sent by the first sensor in real time, or can obtain the first sensor data returned by the first sensor based on the collection instruction sent by the computer device, which is not limited herein.

[0070] Further, after obtaining the first sensor data, the computer device can perform feature extraction on the first sensor data to obtain first feature information of the road object. Specifically, when the first sensor data is point cloud data, the computer device can input the point cloud data into a target recognition model to obtain the contour of the road object, and then obtain the distance features of the road object. When the first sensor data is image data, the computer device can obtain the RGB value of the road object according to the image data.

[0071] S102, input the first sensor data into a preset deep learning model to obtain second feature information of the road object; wherein the deep learning model is obtained by training data collected by the first sensor and the second sensor in the same time period and the same scene; and the first feature information and the second feature information are of different types.

[0072] The second sensor can be a laser radar, a camera or a millimeter wave radar, and the type of the second sensor is not limited herein. Correspondingly, the second historical data collected by the second sensor can be image data or point cloud data, and the type of the second feature information is not limited herein.

[0073] In the present application, the types of the first sensor and the second sensor are different, and the types of the first feature information and the second feature information are also different. The first sensor itself does not have the ability to obtain the second feature information of the road object, and the second sensor itself also does not have the ability to obtain the second feature information of the road object. For example, the first sensor is a laser radar, and the second sensor is an image sensor. Generally, the point cloud data obtained by the laser radar cannot directly obtain the color feature of the road object, and the image data obtained by the image sensor cannot directly obtain the distance feature of the road object.

[0074] For a conventional roadside system or vehicle-mounted system, when the first sensor and the second sensor are used to collect the first feature information and the second feature information of the road object respectively, if the second sensor fails, the system cannot obtain the second feature information of the road object. In the present application, the computer device trains the deep learning model based on the first historical data collected by the first sensor and the second historical data collected by the second sensor, so that the deep learning model obtains the correlation between the first historical data and the second historical data, and thus the second feature information of the road object can be obtained according to the first sensor data.

[0075] The deep learning model can be a supervised learning model, an unsupervised learning model or a semi-supervised learning model, and the type of the deep learning model is not limited herein.

[0076] The deep learning model can obtain the second feature information according to multiple features in the first feature information, or obtain the second feature information according to a single feature in the first feature information, which is not limited herein. The first historical data and the second historical data can be data collected at the same scene at the same time, or data collected in sequence and having an overlapping time period, which is not limited herein. The deep learning model can be applied to the same scene only, or can be migrated to other scenes for use.

[0077] Specifically, after the computer device obtains the first sensor data, the computer device can directly input the first sensor data into the deep learning model, or determine whether to input the first sensor data into the deep learning model according to the working state of the second sensor. For example, when there is a limit on the number or weight of sensors in the system, only one first sensor can be deployed to obtain the first feature information and the second feature information of the road object through the first sensor. Alternatively, the computer device can first determine whether the second sensor is failed, and if so, execute the step of inputting the first sensor data into the preset deep learning model to obtain the second feature information of the road object.

[0078] The failure can be an abnormal connection state between the second sensor and the computer device, or abnormal data returned by the second sensor, such as missing data or distorted data, which is not limited herein.

[0079] When determining whether the second sensor is failed, the computer device can determine according to the state monitoring log of the second sensor, or detect whether the data returned by the second sensor is received. If the data returned by the second sensor is not received, it can be determined that the second sensor is failed, which is not limited herein.

[0080] The above information collection method, the computer device obtains first sensor data of a road object collected by a first sensor, inputs the first sensor data into a preset deep learning model, and obtains second feature information of the road object; wherein the deep learning model is obtained by training data collected by the first sensor and the second sensor in the same time period and the same scene; the types of the first feature information and the second feature information are different. Since the deep learning model is trained based on the data collected by the first sensor and the second sensor in the same time period and the same scene, the deep learning model has the ability to associate the first sensor data with the second feature information; further, since the computer device inputs the first sensor data into the deep learning model, the computer device can obtain the second feature information of the road object through the deep learning model, thereby avoiding the system abnormal working caused by the second sensor unable to obtain the second feature information, and improving the reliability of the system; or when the number or weight of the sensors in the system is limited, the first sensor can be deployed to obtain the first feature information and the second feature information of the road object, thereby improving the flexibility of the sensor deployment of the system. In addition, it should be noted that the second feature information obtained in this embodiment can be directly used for multi-sensor data fusion, target detection, target tracking, etc., without the need for a corresponding feature extraction process. Of course, if the format of the second feature information does not meet the subsequent multi-sensor data fusion, target detection, target tracking data operation, a corresponding format conversion needs to be performed.

[0081] Figure 3 For another flowchart of the information collection method in another embodiment, the embodiment relates to a method for establishing a deep learning model, based on the above embodiment, as shown in Figure 3 the above method further comprises:

[0082] S201, obtaining first historical data collected by a first sensor and second historical data collected by a second sensor in the same time period and the same scene.

[0083] Specifically, the computer device can obtain the first historical data and the second historical data collected by the first sensor and the second sensor in the same time period and the same scene in the database according to the collection time period and the collection scene.

[0084] S202, performing spatio-temporal synchronization processing on the first historical data collected by the first sensor and the second historical data collected by the second sensor, to obtain a spatio-temporal mapping relationship between the first historical data and the second historical data.

[0085] After obtaining the first historical data and the second historical data, the computer device can perform spatio-temporal synchronization processing on the first historical data and the second historical data.

[0086] Specifically, the computer device can first perform time synchronization on the first historical data and the second historical data, and then perform space synchronization; or first perform space synchronization on the first historical data and the second historical data, and then perform time synchronization; the above-mentioned time-space synchronization processing manner is not limited herein. After the time-space synchronization processing on the first historical data and the second historical data, the time-space mapping relationship obtained contains the corresponding relationship between the first historical data and the second historical data collected at the same time for the same target.

[0087] S203, establishing a deep learning model based on the time-space mapping relationship of the first historical data and the second historical data.

[0088] After obtaining the time-space corresponding relationship of the second historical data and the historical point cloud data, the computer device can obtain a training data set of the deep learning model based on the time-space corresponding relationship. The above-mentioned training data set can include positive samples and can also include negative samples. Based on obtaining the training data set, the computer device can perform model training based on the training data set to obtain the deep learning model. The computer device can perform data cleaning on each sample in the training data set, or can process each sample in the training data set to increase the number of samples, and the sample processing manner is not limited herein.

[0089] The above-mentioned information collection method, the computer device performs time-space synchronization processing on the first historical data and the second historical data, and establishes a deep learning model based on the established time-space mapping relationship of the first historical data and the second historical data, so that the deep learning model has the ability to associate the first sensor data with the second feature information, thereby the second feature information of the road object can be obtained according to the first sensor data, and the stability and reliability of the system are improved.

[0090] Figure 4 For another flowchart of the information collection method in another embodiment, the present embodiment relates to a deep learning type of establishment manner, based on the above-mentioned embodiment, the first historical data includes a plurality of first data frames, each first data frame includes a plurality of first data, the second historical data includes a plurality of second data frames, each second data frame includes a plurality of second data, as Figure 4 shown, the above-mentioned S202 includes:

[0091] S301, time synchronization is performed on the first historical data and the second historical data to obtain a plurality of data frame pairs that are time synchronized; each data frame pair includes a first data frame and a second data frame that are synchronized in sampling time.

[0092] The computer device can synchronize the first historical data and the second historical data in time when establishing the deep learning model based on the first historical data and the second historical data, so as to correspond the first historical data and the second historical data collected at the same time.

[0093] The first historical data can include a plurality of first data frames, and the second historical data can include a plurality of second data frames. The computer device can determine whether the first data frames and the second data frames are synchronized in time according to the collection time of the first data frames and the second data frames, and then determine the first data frames and the second data frames synchronized in sampling time as a data frame pair.

[0094] Specifically, when the computer device synchronizes in time according to the sampling time, the sampling time of each first data frame and each second data frame can be obtained, and then the obtained sampling time is compared to determine the synchronized data frames. In addition, the computer device can first determine the second data frame synchronized in time with the first first data frame, and then calculate the position of the second data frame synchronized in time with other first data frames according to the sampling frequency of the first historical data and the second historical data. The above time synchronization method is not limited.

[0095] Optionally, the sampling frequency of the first historical data and the second historical data can be the same. After the computer device determines the first data frame pair synchronized in time, the data frames in the first historical data after the first data frame in the data pair and the data frames in the second historical data after the second data frame in the data pair are determined as the second data frame pair, and the time synchronization of each data frame is sequentially completed, which can improve the time synchronization efficiency of the first historical data and the second historical data.

[0096] S302, coordinate system conversion is performed on the first data frame and the second data frame in each data frame pair to obtain a space-synchronized data pair, and each data pair includes first data in the space-synchronized first data frame and second data in the second data frame.

[0097] Each first data frame can include a plurality of first data, and the first data can be the data of a sampling point in the first data frame, or the data of the sampling points in a certain region in the first data frame. In addition, the first data can also be the first historical data corresponding to a certain road object in the first data frame, which is not limited herein. Each second data frame can include a plurality of second data, and the second data can be the data of a sampling point in the second data frame, or the data of the sampling points in a certain region in the second data frame. In addition, the second data can also be the second historical data corresponding to a certain road object in the second data frame, which is not limited herein.

[0098] After the computer device obtains the plurality of data frame pairs synchronized at the plurality of sampling moments, the first data and the second data in each data frame pair can be corresponded to obtain the first data and the second data collected by different sensor pairs on the same target.

[0099] Specifically, the computer device can perform spatial calibration through coordinate conversion, so that the data collected by the first sensor and the second sensor can be labeled in the same coordinate system, thereby obtaining the correspondence between the first data and the second data. The computer device can convert the first data to the coordinate of the second data, or convert the second data to the coordinate of the first data. In addition, the computer device can also convert the first data and the second data to other coordinate systems, such as the earth coordinate system. The above spatial calibration manner is not limited herein.

[0100] Correspondingly, the computer device can establish a deep learning model based on the correspondence between the first feature information corresponding to the first data and the second feature information corresponding to the second data in each data pair.

[0101] Specifically, the computer device can extract features of the second data to obtain the second feature information corresponding to the second data, and establish a deep learning model of the association relationship between the second feature information associated with the first data and the second data based on the correspondence between the first data and the second data.

[0102] The above information collection method can synchronize the first historical data and the second historical data in time, and then perform spatial correspondence, so that the spatio-temporal mapping relationship between the first historical data and the second historical data can be quickly established, and the acquisition efficiency of the deep learning model is improved.

[0103] Figure 5 For another flowchart of the information collection method, this embodiment relates to a way for the computer device to obtain a deep learning model. Based on the above embodiment, the relative positions of the first sensor and the second sensor are fixed, and S203 includes:

[0104] S401, extracting features of the second historical data to obtain second feature information corresponding to the second data.

[0105] The computer device can extract features of the second historical data to obtain second feature information corresponding to each second data in each second data frame. Specifically, the computer device can input the second historical data into a preset feature extraction model to obtain second feature information corresponding to each second data through the feature extraction model.

[0106] S402, take the first data as a training input sample, take the first feature information associated with the second data corresponding to the first data as a sample label of the training input sample, and obtain a training data set.

[0107] Further, the computer device can take the first data as a training input sample, take the second feature information associated with the second data corresponding to the first data in the above spatio-temporal correspondence as a sample label of the training input sample, and obtain a training data set.

[0108] Specifically, the computer device can obtain a training sample based on the first data and the second data in each data pair; or obtain a training sample according to the first data and the second data in part of the data pairs, which is not limited here. The computer device can set a training weight of the training sample according to the importance of the first data in the first historical data in each data pair.

[0109] S403, take the first data as an input of an initial deep learning model, take the second feature information associated with the second data as a reference output of the initial deep learning model, train the initial deep learning model, and obtain the deep learning model.

[0110] Further, the computer device can take the first data in the training data set as an input of an initial deep learning model, take the second feature information associated with the second data as a reference output of the initial deep learning model, train the initial deep learning model, and obtain the deep learning model. For example, the computer device can adjust the learning rate, sliding average coefficient and other hyperparameters of the model according to the reference output of the initial deep learning model, and obtain the deep learning model.

[0111] For example, the computer device selects historical millimeter wave radar data in a region in the first data frame as the first data, determines the first feature information of the point cloud data corresponding to the first data, and inputs the training sample set after normalization, flipping, shuffling and other preprocessing to the initial deep learning model; uses the methods of down-sampling, up-sampling and multi-size feature splicing in the initial deep learning model to extract multi-dimensional feature information of the historical millimeter wave radar data, and forms multiple feature maps of different sizes; uses target detection to calculate loss of positioning loss, classification loss, foreground and background loss, etc.; then uses gradient descent algorithm to iterate loss function, so that when the loss function is lower than a certain threshold or reaches an iteration termination condition, the deep learning model is obtained.

[0112] In one application scenario, the first sensor is a laser radar, the second sensor is an image sensor, the first sampling point data is point cloud data of a road object, and the second sampling point data is image data of the road object; the second feature information is an RGB value of the road object; when obtaining the deep learning model, the computer device can take the point cloud data as input of an initial deep learning model, take the RGB value corresponding to the image data as reference output of the initial deep learning model, train the initial deep learning model, and obtain the first deep learning model. The point cloud data input to the deep learning model by the computer device can be point cloud data of a target object or point cloud data in a preset coordinate range.

[0113] In another application scenario, the first sensor is an image sensor, the second sensor is a laser radar, the first sampling point data is image data of a road object, and the second sampling point data is point cloud data of the road object; the second feature information is depth information of the road object, and the depth information is used to represent a distance between the road object and the laser radar; when obtaining the deep learning model, the computer device can take the image data as input of an initial deep learning model, take the depth information corresponding to the point cloud data as reference output of the initial deep learning model, train the initial deep learning model, and obtain the second deep learning model.

[0114] It should be noted that the computer device can obtain the first deep learning model and the second deep learning model simultaneously according to the first historical data and the second historical data. When the image sensor fails, the computer device can obtain the RGB value of the road object according to the point cloud data obtained by the laser radar; and when the laser radar fails, the computer device can also obtain the depth information of the road object according to the image data obtained by the image sensor.

[0115] In the information collection method, the computer device performs time synchronization processing and space calibration on the first historical data and the second historical data, thereby obtaining first data and second data collected by the first sensor and the second sensor on the same object at the same time; further, through the corresponding relationship between the first data and the second data, the computer device obtains a training data set and trains the deep learning model, so that the deep learning model establishes the association between the first historical data and the second historical data, thereby the second feature information of the road object can be obtained according to the first sensor data, and the stability and reliability of the system are improved.

[0116] Figure 6 For a flowchart of the information collection method in another embodiment, the embodiment relates to one way in which the computer device performs time synchronization on the first historical data and the second historical data, and on the basis of the above embodiment, as shown in Figure 6 S301 includes:

[0117] S501, convert the first historical data and the second historical data to the same time axis.

[0118] The time axis of the obtained first historical data and second historical data may be different as the first sensor and the second sensor are two independent devices. For example, the time axis of the first sensor is a global positioning system (GPS) time axis, and the time axis of the second sensor is determined by the device itself, and there is a certain time axis difference. The computer device can convert the first historical data and the second historical data to the same time axis, so that the system obtains the first data frame and the second data frame with synchronized sampling time.

[0119] Specifically, the computer device can convert the first historical data to the time axis of the second historical data, or convert the second historical data to the time axis of the first historical data, or convert both the first historical data and the second historical data to another time axis, such as the GPS time axis, and the above conversion modes are not limited herein.

[0120] S502, obtain the first sampling time of each first data frame in the first historical data and the second sampling time of each second data frame in the second historical data under the time axis.

[0121] Further, the computer device can obtain the first sampling time of each first data frame and the second sampling time of each second data frame. The first sampling time can be a timestamp marked on the first data frame when the first sensor collects the first historical data, or the first sampling time can be obtained according to the order of each first data frame and the starting sampling time, and the obtaining method of the first sampling time is not limited herein. The obtaining method of the second sampling time is similar to that of the first sampling time, and will not be repeated herein.

[0122] S503, calculate the difference between the first sampling time and the second sampling time.

[0123] S504, if the difference is less than a preset threshold, determine that the first data frame corresponding to the first sampling time and the second data frame corresponding to the second sampling time are a data frame pair.

[0124] The computer device can consider that the difference between the first sampling time and the second sampling time is within a certain range, and can also determine the first data frame and the second data frame as a data pair. The computer device can calculate the absolute value of the difference between the first sampling time and the second sampling time; if the absolute value of the difference is less than a preset threshold, it is determined that the first data frame corresponding to the first sampling time and the second data frame corresponding to the second sampling time are a data frame pair.

[0125] The above information collection method, the computer device converts the first historical data and the second historical data to the same time axis, so that the computer device can accurately synchronize the first data frame and the second data frame; further, the computer device obtains the absolute value of the difference between the first sampling time and the second sampling time, and when the absolute value of the difference is less than the preset threshold, it is determined that the first data frame and the second data frame are a data frame pair, avoiding the time synchronization failure caused by the difference in sampling frequency and the like, so that the stability of the information collection process is improved.

[0126] Figure 7 For another embodiment of the flowchart of the information collection method, the embodiment relates to a way of spatial calibration of each data pair by the computer device, based on the above embodiment, as shown in Figure 7 The S302 includes:

[0127] S601, according to the preset conversion matrix, the second data in the data frame pair is converted into the first sensor coordinate system, and the mapping coordinates of each second data in the first sensor coordinate system are obtained.

[0128] The conversion matrix is a matrix for converting the data collected by the second sensor to the first sensor coordinate, which can be determined according to the relative pose between the first sensor and the second sensor. The relative pose can include the translation amount and the rotation angle between the first sensor and the second sensor.

[0129] The conversion matrix can be obtained by manual measurement by the staff and then input into the computer device, or it can be obtained by automatic calibration of the computer device according to the current relative pose relationship between the first sensor and the second sensor, which is not limited here. The computer device converts the second data in the data pair into the first sensor coordinate system according to the preset conversion matrix, and can obtain the mapping coordinates of each second data in the first sensor coordinate system.

[0130] The above-mentioned second data can be two-dimensional data or three-dimensional data in the second sensor coordinate system, which is not limited here. When the above-mentioned second data is two-dimensional data, the computer device can first map the second data to the second sensor coordinate system to obtain the second data corresponding to each three-dimensional space point. For example, the second data is a two-dimensional image, and the computer device can map the pixel value of the target object in the two-dimensional image to the corresponding position of the target object in the three-dimensional space according to the preset pixel conversion algorithm. Further, the computer device can convert the second data corresponding to each three-dimensional space point in the second sensor coordinate system to the first sensor coordinate system through the conversion matrix to obtain the mapping coordinates of each second data in the first sensor coordinate system.

[0131] In an embodiment, the second sampling point data is point cloud data of the laser radar, the second sensor coordinate system is a world coordinate system used by the laser radar, the first sampling point data is two-dimensional image data, and the first sensor coordinate is a pixel coordinate system of the two-dimensional image. The conversion relationship between the point cloud data and the two-dimensional image data is:

[0132]

[0133] In the conversion relationship, is an intrinsic matrix of the first sensor, is an extrinsic matrix of the first sensor; wherein r 11 ,r 12 ,…r 33 represent rotation angles between the first sensor and the second sensor, and t1-t3 represent translation amounts between the first sensor and the second sensor. is a coordinate of the second sampling point data in the world coordinate system, is a coordinate of the second sampling point data in the pixel coordinate system.

[0134] S602, in the data frame pair, the first data corresponding to the mapping coordinate is obtained.

[0135] The first data can be two-dimensional data or three-dimensional data in the first sensor coordinate system, which is not limited herein. When the first data is two-dimensional data, the computer device can first map the first data to the first sensor coordinate system to obtain the first data corresponding to each three-dimensional space point; or the computer device can correspond the coordinates to the coordinates of the first data in the two-dimensional coordinate system to determine the two-dimensional coordinates corresponding to the coordinates and the first data corresponding to the two-dimensional coordinates.

[0136] Further, the computer device can determine the first data corresponding to each mapping coordinate according to the mapping coordinates. Specifically, the computer device can find the first data corresponding to the mapping coordinate in the data pair; if the data pair does not contain the data corresponding to the mapping coordinate, the computer device can determine the first data corresponding to the mapping coordinate according to the distance between the mapping coordinate and the coordinate value of the first data. For example, when the distance between the mapping coordinate and the coordinate value of the first data is less than a preset distance threshold, the computer device can determine the first data as the first data corresponding to the mapping coordinate.

[0137] S603, the second data corresponding to each mapping coordinate and the first data corresponding to the mapping coordinate are determined as a data pair.

[0138] On the basis of the above steps, the computer device determines the second data corresponding to each mapping coordinate and the first data corresponding to each mapping coordinate, and can associate the second data with the first data through the mapping coordinates, thereby establishing the correspondence between the first data and the second data.

[0139] The above information collection method, the computer device completes the spatial calibration of the data pair through the conversion matrix, so that the computer device can obtain the second data corresponding to each mapping coordinate and the first data corresponding to each mapping coordinate, and thus the correspondence between the first data and the second data can be established according to the mapping coordinates, which is beneficial to quickly and accurately establishing the fitting mapping model.

[0140] Figure 8 For a flowchart of the information collection method in one embodiment, as Figure 8 The information collection method comprises:

[0141] S701, converting the first historical data and the second historical data to the same time axis.

[0142] S702, obtaining the first sampling time of each first data frame in the first historical data and the second sampling time of each second data frame in the second historical data under the time axis.

[0143] S703, calculating the difference between the first sampling time and the second sampling time, and if the above difference is less than a preset threshold, determining that the first data frame corresponding to the first sampling time and the second data frame corresponding to the second sampling time are a data frame pair.

[0144] S704, converting the second data in the data pair to the first sensor coordinate system according to a preset conversion matrix to obtain the mapping coordinates of each second data under the first sensor coordinate system.

[0145] S705, obtaining the first data corresponding to the mapping coordinates in the data frame pair.

[0146] S706, determining the second data corresponding to each mapping coordinate and the first data corresponding to the mapping coordinates as a data pair.

[0147] S707, performing feature extraction on the second historical data to obtain second feature information corresponding to the second data.

[0148] S708, taking the first data as a training input sample, and taking the second feature information associated with the second data corresponding to the first data as a sample label of the training input sample to obtain a training data set.

[0149] S709, training the initial deep learning model by taking the first data in the training data set as input of the initial deep learning model and taking the second feature information associated with the second data as reference output of the initial deep learning model, to obtain the deep learning model.

[0150] S710, obtaining first sensor data of the road object collected by the first sensor.

[0151] S711, determining whether the second sensor is invalid, and if so, performing S712.

[0152] S712, inputting the first sensor data into the preset deep learning model to obtain second feature information of the road object.

[0153] The information collection method has similar implementation principles and technical effects to those of the above embodiments, and will not be described here again.

[0154] In one embodiment, an information collection method is provided, as shown in the flowchart. Figure 9 The method comprises the following steps.

[0155] S801, determining whether a sensor switching condition is met according to a current sensor state.

[0156] The sensor switching condition can be sensor failure, or the computer device controlling the sensor to stop working, etc., which is not limited herein. The sensor failure can be sensor device damage, abnormal sensor collected data, etc. The computer device can determine whether the sensor is invalid according to the information returned by the sensor, or can determine whether the laser radar is invalid according to the data collected by the sensor, which is not limited herein.

[0157] S802, if the sensor switching condition is met, taking the current sensor as the second sensor, and performing the steps of the information collection method corresponding to the above embodiments to collect information.

[0158] The information collection method has similar implementation principles and technical effects to those of the above embodiments, and will not be described here again.

[0159] It should be understood that, although Figures 2-9 the steps in the flowchart of the method are displayed in sequence according to the arrows, these steps are not necessarily executed in sequence according to the arrows. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, Figures 2-9At least one of the steps in the method can include a plurality of steps or a plurality of stages, which are not necessarily performed at the same time, but can be performed at different times, and the order of the steps or the stages is not necessarily sequential, but can be performed alternately or alternately with at least one of the other steps or the steps or stages in the other steps.

[0160] In one embodiment, as shown in Figure 10 An information acquisition device is provided, comprising: an acquisition module 10 and an input module 20, wherein:

[0161] The acquisition module 10 is configured to acquire first sensor data of a road object collected by a first sensor;

[0162] The input module 20 is configured to input the first sensor data into a preset deep learning model to obtain second feature information of the road object; the deep learning model is obtained by training data collected by the first sensor and a second sensor in the same time period and the same scene; and the first feature information and the second feature information are of different types.

[0163] In one embodiment, on the basis of the above-mentioned embodiments, the device further comprises an establishment module 30, as shown in Figure 11 The establishment module 30 comprises:

[0164] The acquisition unit 301 is configured to acquire first historical data collected by the first sensor and second historical data collected by the second sensor in the same time period and the same scene;

[0165] The synchronization unit 302 is configured to perform space-time synchronization processing on the first historical data collected by the first sensor and the second historical data collected by the second sensor to obtain a space-time mapping relationship between the first historical data and the second historical data;

[0166] The establishment unit 303 is configured to establish a deep learning model based on the space-time mapping relationship between the first historical data and the second historical data.

[0167] In one embodiment, on the basis of the above-mentioned embodiments, the first historical data comprises a plurality of first data frames, and the second historical data comprises a plurality of second data frames, as shown in Figure 12 The synchronization unit 302 comprises:

[0168] The synchronization sub-unit 3021 is configured to perform time synchronization on the first historical data and the second historical data to obtain a plurality of time-synchronized data frame pairs; each data frame pair comprises a first data frame and a second data frame that are synchronized in sampling time;

[0169] The conversion subunit 3022 is configured to perform coordinate system conversion on the first data frame and the second data frame in each data frame pair to obtain a spatially synchronized data pair, and each data pair includes first data in the spatially synchronized first data frame and second data in the second data frame.

[0170] In an embodiment, on the basis of the above-mentioned embodiment, the relative positions of the first sensor and the second sensor are fixed, and the establishing unit 303 is specifically configured to: perform feature extraction on the second historical data to obtain second feature information corresponding to the second data; take the first data as a training input sample, and take the second feature information associated with the second data corresponding to the first data as a sample label of the training input sample to obtain a training data set; take the first data as an input of an initial deep learning model, take the second feature information associated with the second data as a reference output of the initial deep learning model, and train the initial deep learning model to obtain the deep learning model.

[0171] In an embodiment, on the basis of the above-mentioned embodiment, the first sensor is a laser radar, the second sensor is an image sensor, the first sampling point data is point cloud data of a road object, and the second sampling point data is image data of the road object; the second feature information is an RGB value of the road object; and the establishing unit 303 is specifically configured to: take the point cloud data as an input of an initial deep learning model, take the RGB value corresponding to the image data as a reference output of the initial deep learning model, and train the initial deep learning model to obtain the first deep learning model.

[0172] In an embodiment, on the basis of the above-mentioned embodiment, the first sensor is an image sensor, the second sensor is a laser radar, the first sampling point data is image data of a road object, and the second sampling point data is point cloud data of the road object; the second feature information is depth information of the road object, and the depth information is used to represent a distance between the road object and the laser radar; and the establishing unit 303 is specifically configured to: take the image data as an input of an initial deep learning model, take the depth information corresponding to the point cloud data as a reference output of the initial deep learning model, and train the initial deep learning model to obtain the second deep learning model.

[0173] In an embodiment, on the basis of the above-mentioned embodiment, the conversion subunit 3021 is specifically configured to: convert the first historical data and the second historical data to the same time axis; obtain a first sampling time of each first data frame in the first historical data and a second sampling time of each second data frame in the second historical data under the time axis; calculate a difference value between the first sampling time and the second sampling time; and if the difference value is less than a preset threshold, determine that the first data frame corresponding to the first sampling time and the second data frame corresponding to the second sampling time are a data frame pair.

[0174] In an embodiment, on the basis of the above-mentioned embodiment, the sampling frequencies of the first historical data and the second historical data are in a multiple relationship.

[0175] In an embodiment, on the basis of the above-mentioned embodiment, the conversion subunit 3022 is specifically configured to: convert the second data in each data pair into the first sensor coordinate system according to a preset conversion matrix to obtain a mapping coordinate of each second data in the first sensor coordinate system; in the data pair, obtain the first data corresponding to the mapping coordinate; and determine the second data corresponding to each mapping coordinate and the first data corresponding to the mapping coordinate as a data pair.

[0176] The information acquisition device provided in each of the above-mentioned embodiments can execute the above-mentioned information acquisition method embodiments, and has similar implementation principles and technical effects, which will not be described here again.

[0177] In an embodiment, an information acquisition device is provided, as shown in Figure 13 The device comprises:

[0178] The determination module 110 is configured to determine whether the sensor switching condition is met according to the current sensor state.

[0179] The acquisition module 120 is configured to, when the current sensor state meets the sensor switching condition, take the current sensor as the second sensor, execute the steps of the above-mentioned information acquisition method, and perform information acquisition.

[0180] The information acquisition device provided in each of the above-mentioned embodiments can execute the above-mentioned information acquisition method embodiments, and has similar implementation principles and technical effects, which will not be described here again.

[0181] For specific limitations of the information acquisition device, refer to the limitations of the information acquisition method in the above, which will not be described here again. Each module in the above-mentioned information acquisition device can be realized by software, hardware and a combination thereof in whole or in part. The above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to each of the above-mentioned modules.

[0182] In an embodiment, a computer device is provided, which can be a server, and an internal structure diagram of the computer device can be as shown in Figure 14As shown in the figure. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium, an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store information collection data. The network interface of the computer device is used to communicate with the external terminal through the network connection. The computer program is executed by the processor to implement an information collection method.

[0183] Those skilled in the art can understand that, Figure 14 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0184] In one embodiment, a computer device is provided, including a memory and a processor, the memory stores a computer program, and the processor executes the computer program to implement the steps of the above information collection method.

[0185] The computer device provided in the embodiment has similar implementation principles and technical effects to the above method embodiments, and will not be described here.

[0186] In one embodiment, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the steps of the above information collection method.

[0187] The computer readable storage medium provided in the embodiment has similar implementation principles and technical effects to the above method embodiments, and will not be described here.

[0188] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, storage, database or other medium used in each embodiment provided by the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0189] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of each technical feature in the above embodiments are not described, but as long as the combination of the technical features does not exist, it should be considered as the scope of the present application.

[0190] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be pointed out that for those skilled in the art, without departing from the concept of the present application, some modifications and improvements can be made, which are all within the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. An information collection method, characterized in that, The method includes: Acquire first sensor data of a road object collected by a first sensor, and extract features from the first sensor data to obtain first feature information of the road object; If the second sensor fails and the second feature information of the road object cannot be obtained, the data from the first sensor is input into a preset deep learning model to obtain the second feature information of the road object; the deep learning model is trained based on data collected by the first sensor and the second sensor in the same time period and the same scene; the first feature information and the second feature information are of different types.

2. The information collection method according to claim 1, characterized in that, Before acquiring the first sensor data of the road object collected by the first sensor, the method further includes: Acquire the first historical data collected by the first sensor and the second historical data collected by the second sensor in the same time period and the same scene; The first historical data collected by the first sensor and the second historical data collected by the second sensor are spatiotemporally synchronized to obtain the spatiotemporal mapping relationship between the first historical data and the second historical data. The deep learning model is established based on the spatiotemporal mapping relationship between the first historical data and the second historical data.

3. The information collection method according to claim 2, characterized in that, The first historical data includes multiple first data frames, and the second historical data includes multiple second data frames. The step of performing spatiotemporal synchronization processing on the first historical data collected by the first sensor and the second historical data collected by the second sensor to obtain the spatiotemporal mapping relationship between the first historical data and the second historical data includes: The first historical data and the second historical data are synchronized in time to obtain multiple pairs of time-synchronized data frames; each pair of data frames includes a first data frame and a second data frame synchronized at the sampling time. Coordinate system transformation is performed on the first and second data frames in each data frame pair to obtain spatially synchronized data pairs. Each data pair includes the first data in the spatially synchronized first data frame and the second data in the spatially synchronized second data frame.

4. The information collection method according to claim 3, characterized in that, The relative positions of the first sensor and the second sensor are fixed. The step of establishing the deep learning model based on the spatiotemporal mapping relationship between the first historical data and the second historical data includes: Feature extraction is performed on the second historical data to obtain the second feature information corresponding to the second data; The first data is used as the training input sample, and the second feature information associated with the second data corresponding to the first data is used as the sample label of the training input sample to obtain the training data set. The first data is used as the input to the initial deep learning model, and the second feature information associated with the second data is used as the reference output of the initial deep learning model. The initial deep learning model is then trained to obtain the deep learning model.

5. The information collection method according to claim 4, characterized in that, The first sensor is a lidar, the second sensor is an image sensor, the first sampling point data is point cloud data of the road object, and the second sampling point data is image data of the road object; the second feature information is the RGB value of the road object; the step of using the first data as input to an initial deep learning model, using the second feature information associated with the second data as a reference output of the initial deep learning model, and training the initial deep learning model to obtain the deep learning model includes: The point cloud data is used as the input to the initial deep learning model, and the RGB values ​​corresponding to the image data are used as the reference output of the initial deep learning model to train the initial deep learning model and obtain the first deep learning model.

6. The information collection method according to claim 4, characterized in that, The first sensor is an image sensor, the second sensor is a LiDAR, the first sampling point data is image data of the road object, and the second sampling point data is point cloud data of the road object; the second feature information is depth information of the road object, which is used to characterize the distance between the road object and the LiDAR; the step of using the first data as input to an initial deep learning model, using the second feature information associated with the second data as reference output of the initial deep learning model, and training the initial deep learning model to obtain the deep learning model includes: The image data is used as the input to the initial deep learning model, and the depth information corresponding to the point cloud data is used as the reference output of the initial deep learning model to train the initial deep learning model and obtain a second deep learning model.

7. The information collection method according to any one of claims 3-6, characterized in that, The step of synchronizing the first historical data and the second historical data to obtain multiple pairs of time-synchronized data frames includes: Convert the first historical data and the second historical data to the same timeline; Under the timeline, the first sampling time of each first data frame in the first historical data and the second sampling time of each second data frame in the second historical data are obtained; Calculate the difference between the first sampling time and the second sampling time; If the difference is less than a preset threshold, then the first data frame corresponding to the first sampling time and the second data frame corresponding to the second sampling time are determined to be a data frame pair.

8. The information collection method according to claim 7, characterized in that, The sampling frequencies of the first historical data and the second historical data are in a multiple relationship.

9. The information collection method according to any one of claims 3-6, characterized in that, The coordinate system transformation of the first and second data frames in each data frame pair to obtain spatially synchronized data pairs includes: According to the preset transformation matrix, the second data in the data frame pair is transformed into the first sensor coordinate system to obtain the mapped coordinates of each second data in the first sensor coordinate system; In the data frame pair, obtain the first data corresponding to the mapped coordinates; The second data corresponding to each of the mapping coordinates and the first data corresponding to the mapping coordinates are determined as a data pair.

10. An information collection method, characterized in that, The method includes: Based on the current sensor status, determine whether the sensor switching conditions are met; If the sensor switching condition is met, the current sensor is used as the second sensor, and the steps of the information acquisition method as described in any one of claims 1-9 are executed to acquire information.

11. An information acquisition device, characterized in that, The device includes: The acquisition module is used to acquire first sensor data of a road object collected by a first sensor, and to perform feature extraction on the first sensor data to obtain first feature information of the road object; The input module is used to input the first sensor data into a preset deep learning model to obtain the second feature information of the road object when the second sensor fails and cannot obtain the second feature information of the road object; the deep learning model is obtained by training based on the data collected by the first sensor and the second sensor in the same time period and the same scene; the first feature information and the second feature information are of different types.

12. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 10.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 10.

Images