A system and method for iterative optimization of visual algorithms

Abstract

The purpose of this application is to provide a system and method for iterative optimization of visual algorithms, in which each data acquisition module is used to collect driving data of the vehicle in which it is located; the visual map module is used to construct an abstract point cloud map based on the driving data of all vehicles in the current scene, and determine the position information of all vehicles in the current scene; the dimension conversion module is used to calculate the corresponding three-dimensional point cloud information based on the abstract point cloud map to reconstruct the three-dimensional scene; the self-annotation module is used to automatically annotate traffic targets in the reconstructed three-dimensional scene to obtain annotation information; the model training model is used to iteratively train the visual map module, the dimension conversion module and the self-annotation module according to the annotation information, and feed back the iterative training results to the data acquisition module corresponding to the current scene. Thereby, the collaborative iterative optimization of the visual algorithm is realized, and the performance of the visual algorithm is improved at low cost and high efficiency.

Description

Technical Field

[0001] The present application relates to the field of unmanned driving, and in particular to a system and method for iterative optimization of visual algorithms. Background Art

[0002] Most existing unmanned driving systems use a multi-sensor solution, that is, using multiple sensors such as lidar, high-performance navigation equipment, and cameras. This solution is relatively expensive. In recent years, with the improvement of visual algorithms and the increase in chip computing power, a visual solution that only uses camera sensors has become possible. Compared with multi-sensor solutions, visual solutions have lower costs: the cost of a single camera is only at the level of hundreds of yuan; easier to deploy: on-board cameras are simple to install, and mass production installation solutions are mature; data is easier to obtain: image data is acquired quickly, the frame rate is high, and the compression algorithm is mature. In addition, cameras can obtain the most abundant information, which is irreplaceable compared to lidar. No matter how other sensors such as lidar develop in the future, the status of vision-related algorithms will not be replaced. Facts have also proved that on new models of mainstream OEMs around the world, cameras are also developing in the direction of increasing numbers, pixels, frame rates, and dynamic responses.

[0003] Existing technical solutions cannot improve the performance of visual algorithms at low cost and high efficiency. After obtaining the point cloud map, the existing visual mapping technology requires manual marking of the topological structure and semantic information of the lanes from the map. When the actual scene changes, the map needs to be updated and maintained manually. This process has high labor costs and low efficiency. The existing 2D to 3D technology needs to rely on other high-precision sensors to obtain high-precision camera poses to complete the three-dimensional reconstruction of the scene. Due to the limited pose accuracy, its scene reconstruction performance cannot meet the automatic labeling requirements. Due to the lack of automatic labeling information, model training needs to introduce manual labeling information. However, the image only has 2D information and cannot annotate 3D information. Therefore, the existing technology still needs to introduce high-cost lidar to assist target labeling. In addition, the existing solution optimizes each visual algorithm independently, and the input information under low-cost conditions cannot be effectively improved, and its optimization performance is bound to have bottlenecks. Summary of the invention

[0004] One purpose of the present application is to provide a system and method for iterative optimization of visual algorithms to solve the problem in the prior art that it is impossible to improve the performance of visual algorithms in a low-cost and high-efficiency manner.

[0005] According to one aspect of the present application, a system for iterative optimization of a visual algorithm is provided, the system comprising:

[0006] At least one data acquisition module, visual map module, dimension conversion module, self-labeling module and model training module; wherein,

[0007] Each data collection module is used to collect driving data of the vehicle;

[0008] The visual map module is used to construct an abstract point cloud map based on the driving data of all vehicles in the current scene, and determine the position information of all vehicles in the current scene;

[0009] The dimension conversion module is used to calculate corresponding three-dimensional point cloud information according to the abstract point cloud map to reconstruct a three-dimensional scene;

[0010] The self-annotation module is used to automatically annotate traffic targets in the reconstructed three-dimensional scene, obtain annotation information, and send the annotation information to the model training module;

[0011] The model training model is used to iteratively train the visual map module, the dimension conversion module and the self-labeling module according to the annotation information, and feed back the iterative training results to the data acquisition module corresponding to the current scene to complete the iterative optimization of the visual algorithm.

[0012] Optionally, the driving data includes driving images, and the visual map module is used to locate the driving images of all vehicles in the current scene in the abstract point cloud map to obtain the position information of all vehicles in the current scene.

[0013] Optionally, the dimension conversion module is used to calculate corresponding three-dimensional point cloud information based on the abstract point cloud map, reconstruct the three-dimensional scene based on the three-dimensional point cloud information and the posture information of all vehicles in the visual map module, and send the reconstructed three-dimensional scene information to the self-annotation module.

[0014] Optionally, the system includes a cloud database, which is used to receive data transmitted after the data acquisition module compresses the collected driving data into a data stream, and stores the data according to time and vehicle identification as indexes.

[0015] Optionally, the visual map module is used to obtain a visual feature model and a semantic recognition model from the model training module, obtain deep learning feature points and feature descriptor information for the driving image through the visual feature model, and recognize semantic information from the driving image through the semantic recognition model.

[0016] Optionally, the dimensional conversion module is used to obtain posture information and semantic information from the visual map module, use a visual depth model to obtain depth information of the current scene from the driving image and the posture information, and detect the depth information based on the visual detection model and the semantic information.

[0017] Optionally, the self-annotation module is used to receive new driving data including posture information, semantic information and scene information transmitted by the dimension conversion module, automatically annotate traffic targets in the three-dimensional scene according to the new driving data, obtain annotation information, add the annotation information to the new driving data and send it to the model training module;

[0018] The self-labeling module is used to obtain the latest model training results from the model training module, and label the detection results in the current viewing angle into the target viewing angle according to the latest model training results.

[0019] Optionally, the model training module is used to iteratively train the visual feature model, the visual detection model and the semantic recognition model according to the received new driving data containing the annotation information.

[0020] According to another aspect of the present application, a method for iterative optimization of a visual algorithm is also provided, the method comprising:

[0021] Collecting driving data of a vehicle equipped with a vehicle-mounted camera, wherein the driving data includes a driving image;

[0022] Constructing an abstract point cloud map based on the driving data of all vehicles in the current scene, and inputting the driving images into a visual feature model and a semantic recognition model respectively;

[0023] Detecting depth information of the current scene according to the output result of the visual feature model, the output result of the semantic recognition model, and the visual detection model;

[0024] Calculating corresponding three-dimensional point cloud information according to the abstract point cloud map, reconstructing a three-dimensional scene according to the three-dimensional point cloud information and depth information output by the detection, and obtaining scene information;

[0025] Automatically annotating traffic targets in the reconstructed three-dimensional scene according to the semantic information and the scene information to obtain annotation information;

[0026] The visual feature model, the visual detection model and the semantic recognition model are iteratively trained according to the annotation information.

[0027] Optionally, detecting the depth information of the current scene according to the output result of the visual feature model, the result output by the semantic recognition model, and the visual detection model includes:

[0028] Determine the position information of all vehicles in the current scene according to the output result of the visual feature model and the output result of the semantic recognition model;

[0029] The depth information of the current scene is obtained from the driving image and the posture information, and the depth information is detected according to the visual detection model and the semantic information to complete the iterative optimization of the visual algorithm.

[0030] According to another aspect of the present application, a computer-readable medium is provided, on which computer-readable instructions are stored. The computer-readable instructions can be executed by a processor to implement the method described above.

[0031] Compared with the prior art, the present application provides a system for iterative optimization of visual algorithms, which includes: at least one data acquisition module, a visual map module, a dimension conversion module, a self-annotation module and a model training module; wherein each data acquisition module is used to collect driving data of the vehicle in which it is located; the visual map module is used to construct an abstract point cloud map according to the driving data of all vehicles in the current scene, and determine the position information of all vehicles in the current scene; the dimension conversion module is used to calculate the corresponding three-dimensional point cloud information according to the abstract point cloud map to reconstruct the three-dimensional scene; the self-annotation module is used to automatically annotate traffic targets in the reconstructed three-dimensional scene, obtain annotation information, and send the annotation information to the model training module; the model training model is used to iteratively train the visual map module, the dimension conversion module and the self-annotation module according to the annotation information, and feed back the iterative training results to the data acquisition module corresponding to the current scene to complete the iterative optimization of the visual algorithm. Thereby, the collaborative iterative optimization of the visual algorithm is realized, and the performance of the visual algorithm is improved at low cost and high efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0033] Figure 1 A schematic diagram of the structure of a system for iterative optimization of a visual algorithm provided according to one aspect of the present application is shown;

[0034] Figure 2 An example of a sparse point cloud map in one embodiment of the present application is shown;

[0035] Figure 3 An example of a semantic map in an embodiment of the present application is shown;

[0036] Figure 4 An example of visual depth detection in one embodiment of the present application is shown;

[0037] Figure 5 A schematic diagram showing automated collaborative iterative optimization of visual algorithms according to an embodiment of the present application;

[0038] Figure 6 A schematic flow chart of a method for iterative optimization of a visual algorithm provided according to another aspect of the present application is shown;

[0039] Figure 7 The process of automated collaborative iterative optimization of visual algorithms in one embodiment of the present application is shown.

[0040] The same or similar reference numerals in the drawings represent the same or similar components. DETAILED DESCRIPTION

[0041] The present application is described in further detail below in conjunction with the accompanying drawings.

[0042] In a typical configuration of the present application, the terminal, the device of the service network and the trusted party all include one or more processors (eg, a central processing unit (CPU)), an input / output interface, a network interface and a memory.

[0043] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0044] Computer readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, Phase-Change RAM (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of random access memory (RAM), Read-Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Flash memory or other memory technology, Compact Disc Read-Only Memory (CD-ROM), Digital Versatile Disk (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. According to the definition in this article, computer-readable media does not include non-transitory media such as modulated data signals and carrier waves.

[0045] Figure 1 A structural schematic diagram of a system for iterative optimization of a visual algorithm provided according to one aspect of the present application is shown, the system comprising: at least one data acquisition module 11, a visual map module 12, a dimension conversion module 13, a self-labeling module 14 and a model training module 15; wherein each data acquisition module 11 is used to collect driving data of the vehicle in which it is located; the visual map module 12 is used to construct an abstract point cloud map based on the driving data of all vehicles in the current scene, and determine the position information of all vehicles in the current scene; the dimension conversion module 13 is used to calculate the corresponding three-dimensional point cloud information based on the abstract point cloud map to reconstruct the three-dimensional scene; the self-labeling module 14 is used to automatically label traffic targets in the reconstructed three-dimensional scene, obtain labeling information, and send the labeling information to the model training module; the model training module 15 is used to iteratively train the visual map module, the dimension conversion module and the self-labeling module according to the labeling information, and feed back the iterative training results to the data acquisition module corresponding to the current scene to complete the iterative optimization of the visual algorithm. Therefore, by connecting the input and output of each visual algorithm module and combining the independent modules into a unified closed-loop pipeline through information complementarity, collaborative iterative optimization of the visual algorithm can be achieved, thereby improving the performance of the visual algorithm at low cost and high efficiency.

[0046] Specifically, each data acquisition module 11 is used to collect driving data of the vehicle in which it is located; here, the data acquisition module is deployed on any vehicle equipped with a vehicle-mounted camera, so that data can be collected synchronously or asynchronously in a single or multiple scenes. As the number of vehicles increases, the data collection efficiency becomes higher and higher, thereby realizing data collection in the form of crowdsourcing.

[0047] Specifically, the visual map module 12 is used to construct an abstract point cloud map based on the driving data of all vehicles in the current scene, and determine the position and posture information of all vehicles in the current scene; here, the visual map module obtains the driving data from the data acquisition module to construct the abstract point cloud map, and the visual mapping algorithm includes but is not limited to the SLAM algorithm, the SFM algorithm, etc. The abstract point cloud map is a three-dimensional map composed of sparse point clouds, and the unmanned driving positioning system can determine the position and posture information of the vehicle in the map; Figure 2 An example of a sparse point cloud map is shown in Figure 1. The mapping process can be completed through crowdsourcing: as the driving data of drivers in a scene gradually increases, the map of the scene will become more and more complete, and the accuracy of the map will become higher and higher; after the mapping process, a visual map can be obtained. Since the visual features are not dense, the visual map is an abstract point cloud map at this time, which can be used for visual positioning and navigation of unmanned driving systems. After the abstract point cloud map is constructed, the position information of all vehicles in the current scene in the abstract point cloud map is determined.

[0048] Specifically, the dimension conversion module 13 is used to calculate the corresponding three-dimensional point cloud information according to the abstract point cloud map to reconstruct the three-dimensional scene; here, the dimension conversion module is a two-dimensional to three-dimensional module (2D to 3D module), and the obtained abstract point cloud map is two-dimensional image information. The 2D to 3D module calculates the three-dimensional point cloud information from the two-dimensional image information, and uses the three-dimensional point cloud information to construct a dense three-dimensional scene, and reconstructs the three-dimensional scene, thereby obtaining the scene information of the reconstructed three-dimensional scene.

[0049] Specifically, the self-labeling module 14 is used to automatically label traffic targets in the reconstructed three-dimensional scene, obtain labeling information, and send the labeling information to the model training module; here, the self-labeling module 14 is used to comprehensively use the information transmitted by the dimension conversion module 13 to automatically label traffic targets in the three-dimensional scene, where traffic targets include vehicles, pedestrians, non-motor vehicles, cones, etc., and the road surface, lanes, signs, curbs, etc. are labeled; after all are labeled, the labeling information is obtained, the labeling information is added to the information transmitted by the dimension conversion module, and the new information containing the labeling information is transmitted to the model training module.

[0050] Specifically, the model training model 15 is used to iteratively train the visual map module, the dimension conversion module and the self-annotation module according to the annotation information, and feed back the iterative training results to the data acquisition module corresponding to the current scene to complete the iterative optimization of the visual algorithm. Here, the model training module receives information transmitted from the annotation module, which is the driving data to which the output information is gradually added through the above modules, such as the annotation information is added after the self-annotation module, so as to train the above modules using the received information, and feed back the iterative training results to the data acquisition module to optimize the data acquisition module.

[0051] In some embodiments of the present application, the driving data includes driving images, and the visual map module 12 is used to locate the driving images of all vehicles in the current scene in the abstract point cloud map to obtain the posture information of all vehicles in the current scene. Here, the driving data includes the driving images obtained by the vehicle-mounted camera, and the vehicle chassis information, and the vehicle chassis information includes the vehicle speed, steering wheel angle, etc. The data acquisition module can judge the validity of the data during the working process, such as judging whether the image is normal, whether the vehicle chassis data is normal, etc., and eliminate the driving data with problems. In the visual map module, after constructing the abstract point cloud map, the visual map module locates the driving image in the abstract point cloud map to obtain the posture information corresponding to the image information, and the image information includes the information of all vehicles in the image, so the posture information of all vehicles is obtained by positioning; the posture information accuracy is improved by improving the map accuracy; the trajectory information of the driving data is constructed according to the posture information, and the crowdsourcing trajectory information in the same scene can obtain the big data driving trajectory, which can be used to analyze the driving route topology structure of the area, the driving habits of the driver, etc. It should be noted that the above-mentioned use of the visual map module to build a map is only an example, and other map building methods can also be used, such as using a laser radar to design a laser map module.

[0052] In some embodiments of the present application, the dimension conversion module is used to calculate the corresponding three-dimensional point cloud information based on the abstract point cloud map, reconstruct the three-dimensional scene based on the three-dimensional point cloud information and the posture information of all vehicles in the visual map module, and send the reconstructed three-dimensional scene information to the self-annotation module. Here, the 2D to 3D module calculates the three-dimensional point cloud information from the abstract point cloud map, completes the conversion process from 2D to 3D, and then constructs a dense three-dimensional scene based on the posture information and three-dimensional point cloud information. The 2D to 3D module adds scene information to the data input in the previous step (the data input from the visual map module to the dimension conversion module) and sends it to the self-annotation module.

[0053] In some embodiments of the present application, the system includes a cloud database, which is used to receive the data collected by the data acquisition module after compressing the driving data into a data stream and transmitting the data, and store the data according to the time and vehicle identification as an index. Here, the data acquisition module compresses the collected data into a data stream and transmits it to the cloud storage database. The data in the cloud database is indexed according to time and vehicle identification, and the data collected by the same vehicle in the same time period is grouped together.

[0054] In some embodiments of the present application, the visual map module is used to obtain a visual feature model and a semantic recognition model from the model training module, obtain deep learning feature points and feature descriptor information from the driving image through the visual feature model, and recognize semantic information from the driving image through the semantic recognition model. Here, a visual feature model and a semantic recognition model are used in the visual map module, and these models can be obtained from the model training module; the visual feature model is an algorithm model that detects image feature points and feature descriptors, and is used in unmanned driving positioning systems and perception systems. In the visual map module, a variety of visual features including traditional features and deep learning features can be used, wherein traditional features refer to features extracted using traditional visual algorithms, such as SIFI, SURF, ORB features, etc., and deep learning features refer to features extracted using deep learning algorithms, such as SuperPoint features, etc.; the semantic recognition model is an algorithm model that detects semantic identification information in traffic scenes from images, and this model is used in the visual mapping module in the embodiments of the present application. The semantic recognition model can recognize semantic information from the current driving image, including lane lines, road boundaries, road surface markings and other information; such as Figure 3 An example of a semantic map is shown. Combining the pose information of the image, placing and splicing these semantic information according to the pose can generate a semantic map, which can be used for functions such as unmanned driving path planning. The visual map module adds pose information and semantic information to the original driving data and sends it to the 2D to 3D module.

[0055] In some embodiments of the present application, the dimension conversion module is used to obtain the posture information and semantic information from the visual map module, use the visual depth model to obtain the depth information of the current scene from the driving image and the posture information, and detect the depth information according to the visual detection model and the semantic information. Here, when the 2D to 3D module calculates the three-dimensional point cloud information, it needs the assistance of depth information. Let the image pixel coordinates be (u, v), the depth information of the pixel be d, the three-dimensional point coordinates corresponding to the pixel be (x, y, z), the three-dimensional point coordinates corresponding to the pixel be (x, y, z), the camera focal length is (fx, fy), and the camera optical center is (cx, cy). Then the coordinate conversion formula is:

[0056]

[0057] Here, we can use traditional visual reconstruction methods or deep learning methods to obtain depth information from images and pose information, and use visual detection models to detect depth information; Figure 4 The following figure shows an example of the visual depth detection result for an underground parking lot (the upper part of the figure). The lower figure is the visualization result of the pixel depth information, which is represented by color. For example, red can be used to represent the nearest (low depth) at M, and blue can be used to represent the farthest (high depth) at N. As the number of optimizations and the amount of data increase, the depth detection accuracy of the visual depth model for the scene will continue to improve. Semantic information can also be used to improve the depth detection accuracy. When the accuracy reaches a certain threshold, the visual depth model is output, which can be used in the scene detection system of unmanned vehicles. When facing some special scenes, such as low brightness or severe light reflection in the scene, the visual depth model cannot achieve the target performance, the 2D to 3D module can receive external input with higher accuracy depth information for 3D point cloud computing. This external input information can also be used as annotation information to assist in training the visual depth model. It should be noted that in the 2D to 3D module, other methods can also be used for 3D reconstruction, such as using lidar to obtain depth information and complete 3D reconstruction, or fusing millimeter wave radar and images to obtain depth information and complete 3D reconstruction.

[0058] In some embodiments of the present application, the self-annotation module is used to receive new driving data including posture information, semantic information and scene information transmitted by the dimension conversion module, automatically annotate traffic targets in the three-dimensional scene according to the new driving data, obtain annotation information, add the annotation information to the new driving data and send it to the model training module; the self-annotation module is used to obtain the latest model training results from the model training module, and annotate the detection results in the current perspective to the target perspective according to the latest model training results. Here, the 2D to 3D module adds scene information to the data input in the previous step and sends it to the self-annotation module. The information received by the self-annotation module is driving data including posture information, semantic information and scene information. It uses images, posture, semantics, scenes and other information to automatically annotate traffic targets in the three-dimensional scene, including vehicles, pedestrians, non-motor vehicles, cones, etc.; it can also annotate road surfaces, lanes, signs, curbs, etc. according to three-dimensional coordinates. After the three-dimensional coordinates of the target are annotated, they can be converted to the two-dimensional pixel coordinates corresponding to the image according to the formula. Let the three-dimensional coordinates of the target be (x, y, z), the corresponding image pixel coordinates be (u, v), and the camera focal length be (fx, fy), then the coordinate conversion formula is:

[0059]

[0060] The target boundary points or internal points of the region are projected onto the image to obtain the target area on the image, which constitutes the annotation information. The self-annotation module can obtain the latest model training results from the model training. After completing the detection of images from different perspectives, the detection results in one perspective can be annotated to another perspective. The fusion between multiple perspectives can effectively increase the number of correct annotations, reduce missed detections, and improve training performance. The self-annotation module can receive external annotation inputs to process special detection targets, or targets that cannot be annotated by self-annotation, etc. The self-annotation module adds annotation information to the previous step input (the input of the dimension conversion module to the self-annotation module) and sends it to the model training module.

[0061] In some embodiments of the present application, the model training module is used to iteratively train the visual feature model, the visual detection model, the visual depth model and the semantic recognition model according to the received new driving data containing the annotation information. Here, the model training module comprehensively uses the existing information to train the visual feature model, the visual depth model, the visual detection model and the semantic recognition model, wherein the existing information is the data transmitted from the annotation module of the previous module to the model training module, including the new driving data with added annotation information. The model training module can also train the anthropomorphic driving model based on the image and posture information, and the model can be used to predict the driver's behavior, assist in the design of anthropomorphic planning algorithms, etc. Through the system described in the present application, the previously independent multiple visual algorithm modules are formed into a closed-loop assembly line, and the automatic production of maps and models is realized through the information complementarity between modules; as the performance of the module improves, its output will drive the performance improvement of other modules on the production line, and the automatic iterative optimization of the visual algorithm is realized through this inter-module collaboration.

[0062] In some embodiments of the present application, Figure 5As shown in the schematic diagram of the automated collaborative iterative optimization of the visual algorithm of an embodiment of the present application, the data acquisition module transmits the collected data to the visual map module through the vehicle-mounted camera, constructs an abstract point cloud map and a semantic map in the visual map module, determines the big data driving trajectory, and can obtain the required model from the model training module when constructing the map; the visual map module adds the obtained posture information and semantic information to the driving data and transmits it to the 2D to 3D module, in which the three-dimensional scene reconstruction is constructed using the visual depth model and auxiliary depth information, and the reconstructed scene information is added to the output information of the visual map module and transmitted to the self-annotation module, so that the self-annotation module obtains the required model from the model training module, annotates the traffic target, and then adds the annotation information to the output information of the 2D to 3D module, and transmits it to the model training module, in which the visual feature model, visual detection model, semantic recognition model and other models are iteratively optimized, and the iterative optimization results are transmitted to the data acquisition module. Through the pipeline composed of the above modules, while iteratively optimizing the algorithm, a variety of products for unmanned driving systems are produced. They can be used for unmanned driving positioning, perception, planning and control systems, fully covering unmanned driving needs and sufficient to build a complete unmanned driving system.

[0063] Among them, the data acquisition module can start data recording by event triggering, thereby effectively reducing the total amount of data; possible triggering events may include any one or any combination of the following: when the driver takes over the unmanned vehicle, when the unmanned system exits abnormally, when a traffic accident occurs, etc. When an event occurs, the data acquisition module can choose to record the data within a period of time before the event, and classify the data according to the specific circumstances of the event. The categories may include routine, important, accident, etc. The 2D to 3D module can receive more accurate depth information from external input for three-dimensional point cloud computing; the self-annotation module can receive external annotation input for processing special detection targets, or targets that cannot be annotated by self-annotation, etc.; before starting training, the model training module can receive an external input pre-trained model or a model optimized in the last iteration. When the performance improvement of the trained model exceeds the threshold, the model training module believes that the model performance has been significantly improved, and then outputs a new model. When the performance of the model after training drops beyond a threshold, the module considers that the model performance has dropped significantly, abandons the training, and informs the staff to troubleshoot data problems through feedback signals; the threshold can be a fixed value, such as 0.05, or a variable value, such as from 0.1 to 0.01, which changes linearly or nonlinearly with the actual performance of the model.

[0064] Figure 6 A schematic flow chart of a method for iterative optimization of a visual algorithm provided according to another aspect of the present application is shown, the method comprising: steps S11 to S16, wherein:

[0065] In step S11, driving data of a vehicle equipped with a vehicle-mounted camera is collected, wherein the driving data includes driving images; here, the driving data includes driving images acquired by the vehicle-mounted camera and vehicle chassis information, and the vehicle chassis information includes vehicle speed, steering wheel angle, etc. The validity of the collected data can be judged, for example, whether the image is normal, whether the vehicle chassis data is normal, etc., and driving data with problems can be eliminated.

[0066] In step S12, an abstract point cloud map is constructed based on the driving data of all vehicles in the current scene, and the driving images are respectively input into the visual feature model and the semantic recognition model; here, the driving data is obtained to construct the abstract point cloud map, and the visual mapping algorithm includes but is not limited to the SLAM algorithm, the SFM algorithm, etc. The abstract point cloud map is a three-dimensional map composed of sparse point clouds, and the unmanned driving positioning system can determine the position and posture information of the vehicle in the map; the deep learning feature points and feature descriptor information are obtained from the driving images through the visual feature model, and the semantic recognition model is used to identify semantic information from the driving images.

[0067] In step S13, the depth information of the current scene is detected according to the output results of the visual feature model, the output results of the semantic recognition model and the visual detection model; here, the semantic recognition model can recognize semantic information from the current driving image, including lane lines, road boundaries, road surface markings and other information; obtain posture information and semantic information, use the visual depth model to obtain the depth information of the current scene from the driving image and the posture information, and detect the depth information according to the visual detection model and the semantic information.

[0068] In step S14, the corresponding three-dimensional point cloud information is calculated according to the abstract point cloud map, and the three-dimensional scene is reconstructed according to the three-dimensional point cloud information and the depth information output by the detection, and the scene information is obtained; here, when calculating the three-dimensional point cloud information, the assistance of depth information is required to obtain the posture information and semantic information, and the depth information of the current scene is obtained from the driving image and the posture information using a visual depth model, and the depth information is detected according to the visual detection model and the semantic information; thereby, the three-dimensional scene reconstruction is constructed using the depth information output by the detection and the three-dimensional point cloud information to obtain the scene information.

[0069] In step S15, traffic targets are automatically annotated in the reconstructed three-dimensional scene according to the semantic information and scene information to obtain annotation information; here, the image, posture, semantics, scene and other information are comprehensively used to automatically annotate traffic targets in the three-dimensional scene, including vehicles, pedestrians, non-motor vehicles, cones, etc.; it is also possible to annotate road surfaces, lanes, signs, curbs, etc. according to three-dimensional coordinates; thereby obtaining annotation information through automatic annotation.

[0070] In step S16, the visual feature model, the visual detection model and the semantic recognition model are iteratively trained according to the annotation information. Here, the new driving data including semantic information, posture information and annotation information is comprehensively used to train the visual algorithm, wherein the visual algorithm includes algorithms of the visual feature model, the visual depth model, the visual detection model and the semantic recognition model, so as to iteratively optimize each model.

[0071] In some embodiments of the present application, in step S13, the position information of all vehicles in the current scene is determined according to the output results of the visual feature model and the output results of the semantic recognition model; the depth information of the current scene is obtained from the driving image and the position information, and the depth information is detected according to the visual detection model and the semantic information to complete the iterative optimization of the visual algorithm. Here, the visual feature model is an algorithm model for detecting image feature points and feature descriptors, which is used in unmanned driving positioning systems and perception systems; the semantic recognition model is an algorithm model for detecting semantic identification information in traffic scenes from images, and the semantic recognition model can recognize semantic information from the current driving image, including lane lines, road boundaries, road surface markings and other information; the visual detection model is an algorithm model for detecting three-dimensional information of targets such as vehicles and pedestrians in traffic scenes from images, which is used in unmanned driving perception systems; when obtaining depth information, a visual depth model can be used, which is an algorithm model for detecting pixel distances from images, which can detect unknown scenes or unknown object information, and is used in unmanned driving perception systems and three-dimensional scene reconstruction; three-dimensional scene reconstruction is the reconstruction of local traffic scenes using dense point clouds. As the data increases, the collaborative iterative optimization of multiple algorithms can break through the cost and performance bottlenecks of a single algorithm and continuously improve the performance of visual algorithms.

[0072] Figure 7 The process of automated collaborative iterative optimization of visual algorithms in one embodiment of the present application is shown. In the startup phase, since each module has not been fully optimized, it is necessary to receive external input information: external input depth information can train the visual depth model; external input manual annotation information can assist the self-annotation module; external input pre-trained model can speed up the model training process. In this stage, it is necessary to use external input to quickly improve the performance of each module. After the performance meets the iteration requirements, it enters the intermediate stage.

[0073] In the middle stage, each module has been optimized, but there is still room for performance improvement. At this point, the iterative positive feedback in the closed-loop pipeline is stable: that is, better output is obtained by optimizing the module, and better input is used as input to optimize other modules, and so on. This stage does not require manual operation, and the iterative optimization runs automatically. When an abnormal situation occurs during the training process, a feedback signal is designed in the embodiment of the present application to prompt the operator to handle it. When the performance reaches the stopping requirement, it enters the stopping stage.

[0074] In the stop phase, each module has been fully optimized and the performance no longer increases during the iteration process. At this point, the optimization stops and waits for new data to arrive before entering the intermediate phase again.

[0075] In the middle stage of the automated collaborative iterative optimization of visual algorithms and the production line, a feedback signal is designed in the embodiment of the present application to facilitate monitoring and processing of iterative anomalies. These feedback signals and processing methods include: 2D to 3D module feedback signal and processing method, model training module feedback signal and processing method; wherein, the 2D to 3D module feedback signal and processing method are as follows: when facing certain special situations, such as low brightness, severe light reflection in the scene, etc., the visual depth model cannot achieve good training performance. At this time, the 2D to 3D module will feedback a low training performance signal and skip this set of driving data. When this signal is observed, it is necessary to take out this set of driving data, obtain more accurate depth information through other means, and enter the 2D to 3D module again with this set of data. The feedback signal and processing method of the model training module are as follows: when the performance of the trained model decreases significantly compared to before, the model training module will feedback a low training performance signal and remove this set of driving data from the training database. When this signal is observed, it is necessary to take out this set of driving data, analyze the cause, and put it back into the driving database after adjustment. If necessary, other methods can be used to annotate this set of data, which can be input into the self-annotation module externally and used with this set of data to participate in optimization iteration again.

[0076] Through the system and method described above, the goals of lower cost, higher efficiency, better performance and product diversification can be achieved. Specifically, only visual sensors are needed, and the cost of a single on-board camera is only in the hundreds of yuan level, which has obvious cost advantages compared to other sensor solutions. Under normal circumstances, the entire production process does not require manual parameters, which saves more labor costs than existing methods. Each link of the assembly line is directly driven by data, and there is no operation delay between links. Automated iterative production can work 24 hours a day, and when new driving data appears, it can be immediately used in the algorithm optimization iterative process. Collaborative iterative optimization is achieved through information complementarity between multiple algorithms. The performance improvement of a single algorithm can affect and promote the improvement of other algorithms to form positive feedback, and better optimization performance can be achieved under low-cost conditions.

[0077] The system and method described in the embodiments of the present application can produce big data driving trajectories, abstract point cloud maps, semantic maps, visual depth models, three-dimensional scene reconstruction, visual detection models, semantic recognition models, and anthropomorphic driving models, etc. Map information can be used for unmanned driving positioning and planning systems, and model information can be used for unmanned driving perception and control systems. These products are sufficient to build a complete autonomous driving system. In addition to meeting the needs of autonomous driving and the complementarity of assembly line module information, big data driving trajectories can also be used to analyze traffic flow and driver behavior habits; three-dimensional scene reconstruction can help managers perform spatial analysis and traffic scene reproduction, etc. Compared with existing methods, the output of this application is more diversified and more practical.

[0078] In addition, an embodiment of the present application also provides a computer-readable medium on which computer-readable instructions are stored. The computer-readable instructions can be executed by a processor to implement the aforementioned method for iterative optimization of a visual algorithm.

[0079] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

[0080] It should be noted that the present application can be implemented in software and / or a combination of software and hardware, for example, can be implemented using an application specific integrated circuit (ASIC), a general purpose computer or any other similar hardware device. In one embodiment, the software program of the present application can be executed by a processor to implement the steps or functions described above. Similarly, the software program of the present application (including relevant data structures) can be stored in a computer-readable recording medium, for example, a RAM memory, a magnetic or optical drive or a floppy disk and similar devices. In addition, some steps or functions of the present application can be implemented using hardware, for example, as a circuit that cooperates with a processor to perform each step or function.

[0081] In addition, a part of the present application may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present application through the operation of the computer. The program instruction for calling the method of the present application may be stored in a fixed or removable recording medium, and / or transmitted through a data stream in a broadcast or other signal-bearing medium, and / or stored in a working memory of a computer device that runs according to the program instruction. Here, according to an embodiment of the present application, a device is included, the device including a memory for storing computer program instructions and a processor for executing program instructions, wherein, when the computer program instruction is executed by the processor, the device is triggered to run the method and / or technical solution based on the aforementioned multiple embodiments according to the present application.

[0082] It is obvious to those skilled in the art that the present application is not limited to the details of the above exemplary embodiments, and that the present application can be implemented in other specific forms without departing from the spirit or basic features of the present application. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the present application is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present application. Any figure mark in the claims should not be regarded as limiting the claims involved. In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

Claims

1. A system for iterative optimization of visual algorithms, characterized in that: The system comprises: At least one data acquisition module, visual map module, dimension conversion module, self-labeling module and model training module; wherein, Each data acquisition module is used to collect driving data of the vehicle, wherein the driving data includes driving images; The visual map module is used to construct an abstract point cloud map based on the driving data of all vehicles in the current scene. The visual map module is used to locate the driving images of all vehicles in the current scene in the abstract point cloud map to obtain the position information of all vehicles in the current scene; The dimension conversion module is used to calculate corresponding three-dimensional point cloud information according to the abstract point cloud map to reconstruct a three-dimensional scene; The self-annotation module is used to automatically annotate traffic targets in the reconstructed three-dimensional scene, obtain annotation information, and send the annotation information to the model training module; The model training model is used to iteratively train the visual map module, the dimension conversion module and the self-annotation module according to the annotation information, and feed back the iterative training results to the data acquisition module corresponding to the current scene to complete the iterative optimization of the visual algorithm; The visual map module is used to obtain a visual feature model and a semantic recognition model from the model training module, obtain deep learning feature points and feature descriptor information from the driving image through the visual feature model, and recognize semantic information from the driving image through the semantic recognition model; The dimension conversion module is used to obtain the posture information and semantic information from the visual map module, obtain the depth information of the current scene from the driving image and the posture information using the visual depth model, and detect the depth information according to the visual detection model and the semantic information; The self-annotation module is used to receive the new driving data including posture information, semantic information and scene information transmitted by the dimension conversion module, automatically annotate the traffic targets in the three-dimensional scene according to the new driving data, obtain the annotation information, add the annotation information to the new driving data and send it to the model training module; The self-labeling module is used to obtain the latest model training results from the model training module, and label the detection results in the current viewing angle into the target viewing angle according to the latest model training results.

2. The system according to claim 1, characterized in that The dimension conversion module is used to calculate the corresponding three-dimensional point cloud information based on the abstract point cloud map, reconstruct the three-dimensional scene based on the three-dimensional point cloud information and the posture information of all vehicles in the visual map module, and send the reconstructed three-dimensional scene information to the self-annotation module.

3. The system according to claim 1, characterized in that The system includes a cloud database, which is used to receive data transmitted after the driving data collected by the data collection module is compressed into a data stream, and store the data according to time and vehicle identification as indexes.

4. The system according to claim 1, characterized in that The model training module is used to iteratively train the visual feature model, the visual detection model, the visual depth model and the semantic recognition model according to the received new driving data containing the annotation information.

5. A method for iterative optimization of a visual algorithm using the system according to any one of claims 1 to 4, characterized in that: The method comprises: Collecting driving data of a vehicle equipped with a vehicle-mounted camera, wherein the driving data includes a driving image; Constructing an abstract point cloud map based on the driving data of all vehicles in the current scene, and inputting the driving images into a visual feature model and a semantic recognition model respectively; Detecting depth information of the current scene according to the output result of the visual feature model, the output result of the semantic recognition model, and the visual detection model; Calculating corresponding three-dimensional point cloud information according to the abstract point cloud map, reconstructing a three-dimensional scene according to the three-dimensional point cloud information and depth information output by the detection, and obtaining scene information; Automatically annotating traffic targets in the reconstructed three-dimensional scene according to the semantic information and the scene information to obtain annotation information; The visual feature model, the visual detection model and the semantic recognition model are iteratively trained according to the annotation information to complete iterative optimization of the visual algorithm.

6. The method according to claim 5, characterized in that Detecting the depth information of the current scene according to the output result of the visual feature model, the output result of the semantic recognition model, and the visual detection model, including: Determine the position information of all vehicles in the current scene according to the output result of the visual feature model and the output result of the semantic recognition model; Depth information of the current scene is acquired from the driving image and the posture information, and the depth information is detected according to a visual detection model and the semantic information.

7. A computer-readable medium having computer-readable instructions stored thereon, wherein the computer-readable instructions can be executed by a processor to implement the method according to claim 5 or 6.

Images