Semantic environment processing method and device for intelligent driving, and electronic device

Abstract

The application provides a semantic environment processing method and device for intelligent driving, electronic equipment, computer readable storage medium and computer program product. The method comprises: acquiring a plurality of images collected by a vehicle camera during driving of a host vehicle; performing map element extraction processing on each image to obtain map elements in each image; constructing a visual point cloud corresponding to each image according to the map elements in each image; performing local map breaking processing on the driving process of the host vehicle according to a local map breaking condition, and performing visual point cloud fusion processing on a plurality of visual point clouds between adjacent two times of local map breaking to obtain a local semantic environment; performing road breaking processing on the driving process of the host vehicle according to a road breaking condition, and performing semantic environment fusion processing on a plurality of local semantic environments between adjacent two times of road breaking to obtain a global semantic environment. Through the application, a semantic environment can be constructed online and at low cost.

Description

Technical Field

[0001] This application relates to vehicle technology, and more particularly to a semantic environment processing method, device, electronic device, computer-readable storage medium, and computer program product for intelligent driving. Background Technology

[0002] In the field of intelligent driving, especially the recently popular Navigate On Autopilot (NOA) and Urban NOA, semantic reconstruction of the road environment is indispensable. The reconstructed semantic environment can assist the perception module in improving perception accuracy and perception distance through prior knowledge; assist the positioning module in improving positioning accuracy in the absence of satellite signals or in the case of poor satellite signals; and assist the planning and control module in completing long-distance path planning, etc.

[0003] In solutions provided by related technologies, vehicle-mounted LiDAR is typically used to construct semantic environments. However, the laser point clouds obtained by vehicle-mounted LiDAR scanning lack semantic meaning and require offline segmentation processing on a map production platform, which is costly and time-consuming. Furthermore, laser point clouds require specialized data collection vehicles, which are also expensive. In summary, constructing semantic environments using solutions provided by related technologies is costly. Summary of the Invention

[0004] This application provides a semantic environment processing method, device, electronic device, computer-readable storage medium, and computer program product for intelligent driving, which can construct a semantic environment online and at low cost.

[0005] The technical solution of this application is implemented as follows:

[0006] This application provides a semantic environment processing method for intelligent driving, including:

[0007] Acquire multiple frames of images captured by the vehicle's onboard camera during the vehicle's driving process;

[0008] Map feature extraction is performed on each frame of the image to obtain the map features in each frame of the image;

[0009] Construct a visual point cloud corresponding to each frame of the image based on the map features in each frame.

[0010] The vehicle's driving process is subjected to local map interruption processing based on local map interruption conditions, and the multi-frame visual point cloud between two adjacent local map interruptions is subjected to visual point cloud fusion processing to obtain the local semantic environment.

[0011] The vehicle's driving process is interrupted according to the road interruption conditions, and multiple local semantic environments between two adjacent road interruptions are fused to obtain the global semantic environment.

[0012] This application provides a semantic environment processing device for intelligent driving, comprising:

[0013] The acquisition module is used to acquire multiple frames of images captured by the vehicle-mounted camera during the vehicle's driving process;

[0014] The extraction module is used to extract map features from each frame of the image to obtain the map features in each frame of the image.

[0015] The building module is used to construct the visual point cloud corresponding to each frame of the image based on the map features in each frame.

[0016] The first interruption module is used to perform local map interruption processing on the vehicle's driving process according to the local map interruption conditions, and to perform visual point cloud fusion processing on the multi-frame visual point cloud between two adjacent local map interruptions to obtain the local semantic environment.

[0017] The second interruption module is used to perform road interruption processing on the vehicle's driving process according to the road interruption conditions, and to perform semantic environment fusion processing on multiple local semantic environments between two adjacent road interruptions to obtain a global semantic environment.

[0018] This application provides an electronic device, including:

[0019] Memory, used to store executable instructions;

[0020] The processor, when executing executable instructions stored in the memory, implements the semantic environment processing method for intelligent driving provided in this application.

[0021] This application provides a computer-readable storage medium storing executable instructions for inducing a processor to execute and implement the semantic environment processing method for intelligent driving provided in this application.

[0022] This application provides a computer program product including executable instructions for implementing the semantic environment processing method for intelligent driving provided in this application when executed by a processor.

[0023] This application has the following beneficial effects:

[0024] This application acquires multiple frames of images captured by an onboard camera during vehicle movement; extracts map features from each frame to obtain map features; constructs a visual point cloud corresponding to each frame based on the map features; performs local map segmentation processing on the vehicle's movement based on local map segmentation conditions, and fuses the visual point clouds of multiple frames between two adjacent local map segmentations to obtain a local semantic environment; performs road segmentation processing on the vehicle's movement based on road segmentation conditions, and fuses the semantic environments of multiple local semantic environments between two adjacent road segmentations to obtain a global semantic environment. In this way, a semantic environment can be constructed online and at low cost, without the need for additional sensors or a large number of professional personnel; at the same time, segmentation based on local map segmentation and road segmentation conditions reduces data stacking, reasonably represents structured road data, and improves the accuracy of the constructed semantic environment. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a schematic diagram of the architecture of the semantic environment processing system for intelligent driving provided in an embodiment of this application;

[0027] Figure 2 This is a schematic diagram of the structure of the vehicle-mounted device provided in an embodiment of this application;

[0028] Figure 3A This is a first flowchart illustrating the semantic environment processing method for intelligent driving provided in this application embodiment;

[0029] Figure 3B This is a second flowchart illustrating the semantic environment processing method for intelligent driving provided in this application embodiment;

[0030] Figure 3C This is a schematic diagram of the third process of the semantic environment processing method for intelligent driving provided in the embodiments of this application;

[0031] Figure 4 This is a schematic diagram of the fourth process of the semantic environment processing method for intelligent driving provided in the embodiments of this application. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0033] In the following description, references to "some embodiments" describe a subset of all possible embodiments; however, it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict. In the following description, the term "a plurality of" means at least two.

[0034] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0035] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0036] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0037] 1) Semantic Environment: Also known as the intelligent driving semantic environment, it describes the environment surrounding the vehicle. The semantic environment includes various map elements, such as stop lines, lane lines, and curbs. In the embodiments of this application, the semantic environment can be represented by vectorization. For example, various map elements can be classified into two vector categories: lines (such as lane lines) and surfaces (such as zebra crossings). In this way, the stored semantic environment has high interpretability and maintainability; at the same time, it takes into account the advantages of low storage cost and rich information.

[0038] 2) Coordinate system: used to describe the position and trajectory of an object in space. Through the coordinate system, abstract geometric concepts can be transformed into specific numerical calculations, thus facilitating the solution of various mathematical and physical problems.

[0039] The coordinate systems involved in this application's embodiments include a vehicle coordinate system and a global coordinate system. The vehicle coordinate system is centered on the vehicle, with the x-axis describing the vehicle's left and right positions and the y-axis describing its front and rear positions. Of course, the x and y axes can be interchanged in some cases. The global coordinate system refers to the Earth coordinate system; compared to the vehicle coordinate system, each coordinate in the global coordinate system is unique in the real world. In this application's embodiments, since the vehicle coordinate system is centered on the vehicle, projection between the vehicle coordinate system and the global coordinate system can be achieved based on the vehicle's pose.

[0040] This application provides a semantic environment processing method, apparatus, electronic device, computer-readable storage medium, and computer program product for intelligent driving, which can construct a semantic environment online and at low cost. The following describes exemplary applications of the electronic device provided in this application. The electronic device provided in this application can be implemented as an in-vehicle device or as a server.

[0041] See Figure 1 , Figure 1 This is an architectural diagram of a semantic environment processing system 100 for intelligent driving provided in an embodiment of this application. The vehicle-mounted device 400 is connected to the server 200 via the network 300. The vehicle-mounted device 400 is a general term for electronic devices installed in the vehicle, which are used to provide functions such as data calculation, braking control, communication control, and driving assistance.

[0042] In some embodiments, taking an in-vehicle device as an example, the semantic environment processing method for intelligent driving provided in this application can be implemented by an in-vehicle device. For example, the in-vehicle device 400 acquires multiple frames of images captured by an in-vehicle camera during the vehicle's driving process; performs map element extraction processing on each frame of image to obtain map elements in each frame of image; constructs a visual point cloud corresponding to each frame of image based on the map elements in each frame of image; performs local map interruption processing on the vehicle's driving process according to local map interruption conditions, and performs visual point cloud fusion processing on the multiple frames of visual point clouds between two adjacent local map interruptions to obtain a local semantic environment; performs road interruption processing on the vehicle's driving process according to road interruption conditions, and performs semantic environment fusion processing on the multiple local semantic environments between two adjacent road interruptions to obtain a global semantic environment.

[0043] In some embodiments, taking the electronic device as a server as an example, the semantic environment processing method for intelligent driving provided in this application embodiment can also be implemented by a server. For example, the vehicle-mounted device 400 can send multiple frames of images collected by the vehicle-mounted camera during the vehicle's driving process to the server 200. The server 200 performs map feature extraction processing on each frame of image to obtain the map features in each frame of image; constructs a visual point cloud corresponding to each frame of image based on the map features in each frame of image; performs local map interruption processing on the vehicle's driving process according to local map interruption conditions, and performs visual point cloud fusion processing on the multiple frames of visual point clouds between two adjacent local map interruptions to obtain a local semantic environment; performs road interruption processing on the vehicle's driving process according to road interruption conditions, and performs semantic environment fusion processing on the multiple local semantic environments between two adjacent road interruptions to obtain a global semantic environment. The above method delegates some operations to the server 200, which can reduce the computational and storage pressure on the vehicle-mounted device 400.

[0044] In some embodiments, the global semantic environment can assist the vehicle's perception module in improving perception accuracy and perception distance; and / or, the global semantic environment can assist the vehicle's positioning module in improving positioning accuracy in the absence of satellite signals or in the presence of poor satellite signals; and / or, the global semantic environment can assist the vehicle's planning and control module in completing long-distance path planning, etc.

[0045] In some embodiments, the in-vehicle device 400 or server 200 can implement the semantic environment processing method for intelligent driving provided in this application embodiment by running a computer program. For example, the computer program can be a native program or software module in an operating system; it can be a native application (APP), i.e., a program that needs to be installed in the operating system to run; it can also be a mini-program, i.e., a program that only needs to be downloaded to a browser environment to run; or it can be a mini-program that can be embedded in any APP, and the mini-program can be controlled by the user to run or close. In short, the above-mentioned computer program can be any form of application, module or plugin.

[0046] In some embodiments, server 200 may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0047] Taking the electronic device provided in this application as an example, which is an in-vehicle device, it can be understood that in the case where the electronic device is a server, Figure 2 Some parts of the structure shown (such as the user interface, presentation module, and input processing module) can be omitted. See also Figure 2 , Figure 2 This is a structural schematic diagram of the vehicle-mounted device 400 provided in an embodiment of this application. Figure 2 The illustrated vehicle-mounted device 400 includes at least one processor 410, a memory 450, at least one network interface 420, and a user interface 430. The various components in the vehicle-mounted device 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to implement communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 440.

[0048] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0049] User interface 430 includes one or more output devices 431 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0050] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices physically located away from the processor 410.

[0051] The memory 450 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 450 described in this application embodiment is intended to include any suitable type of memory.

[0052] In some embodiments, memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0053] Operating system 451 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0054] The network communication module 452 is used to reach other computing devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0055] Presentation module 453 is configured to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 431 (e.g., a display screen, a speaker, etc.) associated with user interface 430;

[0056] The input processing module 454 is used to detect and translate one or more user inputs or interactions from one or more input devices 432.

[0057] In some embodiments, the semantic environment processing device for intelligent driving provided in this application can be implemented in software. Figure 2 A semantic environment processing device 455 for intelligent driving, stored in memory 450, is shown. This device can be software in the form of programs and plug-ins, and includes the following software modules: an acquisition module 4551, an extraction module 4552, a construction module 4553, a first interruption module 4554, and a second interruption module 4555. These modules are logically connected and can therefore be arbitrarily combined or further split according to their implemented functions. The functions of each module will be described below.

[0058] The semantic environment processing method for intelligent driving provided in this application will be described by referring to exemplary applications and implementations of the electronic devices provided in the embodiments of this application.

[0059] See Figure 3A , Figure 3A This is a flowchart illustrating a semantic environment processing method for intelligent driving provided in an embodiment of this application, which will be combined with... Figure 3A The steps shown are explained.

[0060] In step 101, multiple frames of images are acquired by the vehicle-mounted camera during the vehicle's driving process.

[0061] Here, the vehicle-mounted camera periodically acquires and processes images while the vehicle is in motion, thus obtaining multiple frames of images.

[0062] In some embodiments, the vehicle-mounted camera can be a forward-looking camera. Considering that when using a surround-view camera, it is necessary to stitch together a surround-view image, which will result in the loss of height information during the process, making it impossible to collect the elevation coordinates of map features, therefore, in this embodiment, the image acquisition and processing can be performed using a forward-looking camera to ensure the accuracy of the height information in the final constructed semantic environment.

[0063] In step 102, map feature extraction processing is performed on each frame of the image to obtain the map features in each frame of the image.

[0064] Here, map feature extraction processing is performed on each frame of the image to obtain the map features in each frame of the image. Each frame of the image may include one or more map features, and each map feature is described by a series of points.

[0065] It is worth noting that the embodiments of this application do not limit the method of map feature extraction. For example, a pre-trained deep learning model can be used to extract map features from each frame of the image to obtain the map features in each frame. The embodiments of this application also do not limit the type of map features, such as lane lines, road arrows, stop lines, etc.

[0066] In step 103, a visual point cloud corresponding to each frame image is constructed based on the map features in each frame image.

[0067] Here, a visual point cloud corresponding to each frame of the image is constructed based on the map features in each frame. That is, the map features in each frame of the image are described in the global coordinate system. Compared with the laser point cloud obtained by scanning with vehicle-mounted LiDAR, the visual point cloud here can be regarded as a pseudo point cloud. That is, the visual point cloud is not directly acquired, but is obtained by a series of processing on the image acquired by the vehicle-mounted camera.

[0068] It is worth noting that a point cloud is a three-dimensional data set composed of a large number of points, each of which contains spatial location information and possible other attributes.

[0069] In some embodiments, the above-mentioned construction of the visual point cloud corresponding to each frame image based on the map features in each frame image can be achieved in the following manner: obtain the camera parameters of the vehicle camera and the vehicle pose corresponding to each frame image; based on the camera parameters of the vehicle camera and the vehicle pose corresponding to each frame image, project the map features in each frame image from the vehicle coordinate system to the global coordinate system to obtain the visual point cloud corresponding to each frame image.

[0070] Since the images captured by the vehicle-mounted camera are described in the vehicle's own coordinate system, centered on the vehicle, the camera parameters (such as intrinsic and extrinsic parameters) and the vehicle's pose corresponding to each frame can be obtained. Based on these parameters and the vehicle's pose, the map features in each frame are projected from the vehicle's coordinate system to the global coordinate system, resulting in a visual point cloud. For example, based on the camera parameters and the vehicle's pose corresponding to the first frame, the map features in the first frame are projected from the vehicle's coordinate system to the global coordinate system, yielding the visual point cloud for that frame. This method, based on camera parameters and the vehicle's pose, achieves projection between coordinate systems, improving the accuracy of the constructed visual point cloud.

[0071] In some embodiments, the vehicle-mounted camera and the pose sensor (a sensor used to acquire the vehicle's pose) can be time-aligned to obtain the vehicle's pose corresponding to each frame of the image. Here, considering the potential frame rate inconsistency between the vehicle-mounted camera and the pose sensor, time alignment can be performed. This ensures that the corresponding vehicle pose can be determined for each frame acquired by the vehicle-mounted camera. For example, for an image with timestamp t acquired by the vehicle-mounted camera, interpolation is performed between the vehicle pose acquired by the pose sensor with timestamp less than t and the latest acquisition time, and the vehicle pose with timestamp greater than t and the earliest acquisition time, to obtain the vehicle pose corresponding to the image with timestamp t. In other words, time alignment can be achieved through interpolation.

[0072] In step 104, the vehicle's driving process is subjected to local map interruption processing according to the local map interruption conditions, and the multi-frame visual point cloud between two adjacent local map interruptions is subjected to visual point cloud fusion processing to obtain the local semantic environment.

[0073] Here, the local map interruption process is processed according to the local map interruption conditions. It can be understood that during the driving process, the local map will be interrupted multiple times according to the local map interruption conditions. For two adjacent local map interruptions, the visual point cloud of multiple frames between the two adjacent local map interruptions is fused to obtain the local semantic environment.

[0074] It's worth noting that the purpose of breaking roads based on local map break conditions is to structure the roads, facilitating the generation of different lanes and relationships within the local semantic environment, while also reducing data stacking. Local map break conditions can be changes in specific map features, or other conditions.

[0075] In some embodiments, visual point cloud fusion processing of multiple frames of visual point clouds may refer to overlaying the same map element in multiple frames of visual point clouds to obtain the map element in the local semantic environment.

[0076] In some embodiments, when performing visual point cloud fusion processing on multiple frames of visual point clouds between two adjacent local map interruptions, redundant data (i.e., duplicate data) between the multiple frames of visual point clouds is removed to avoid the redundant data from adversely affecting the accuracy of the local semantic environment.

[0077] In some embodiments, the local map interruption conditions include at least one of the following: the number of lanes on the current road where the vehicle is located changes; or the vehicle travels a distance greater than a first length threshold on the current road.

[0078] Here, the local map interruption condition considers changes in the lane situation of the current road where the vehicle is located. For example, interruption is performed when the number of lanes on the current road where the vehicle is located changes, which can more accurately generate different lanes and related relationships in the local semantic environment. When the travel length of the vehicle on the current road exceeds the first length threshold, interruption is performed considering factors such as possible lane changes and changes in traffic conditions. At the same time, the travel length of the vehicle on the current road is cleared to facilitate subsequent judgment on whether to interrupt.

[0079] In step 105, the driving process of the vehicle is interrupted according to the road interruption conditions, and the semantic environment of multiple local semantic environments between two adjacent road interruptions is fused to obtain the global semantic environment.

[0080] Based on the existing local semantic environment, the vehicle's driving process is interrupted according to the road interruption conditions. Then, the semantic environment of multiple local semantic environments between two adjacent road interruptions is fused to obtain the global semantic environment.

[0081] It's worth noting that the purpose of breaking roads based on break conditions is to structure the roads, facilitating the generation of different roads and their relationships within the global semantic environment, while also reducing data stacking. Road break conditions can include changes in the road itself, or other conditions.

[0082] In some embodiments, when performing semantic environment fusion processing on multiple local semantic environments between two adjacent road interruptions, redundant data (i.e., duplicate data) between multiple local semantic environments is removed to avoid the redundant data from adversely affecting the accuracy of the global semantic environment.

[0083] In some embodiments, the road interruption conditions include at least one of the following: the stop line of the current road where the vehicle is located disappears; the current road where the vehicle is located is a ramp; the travel length of the vehicle on the current road is greater than a second length threshold; the vehicle makes a U-turn.

[0084] Here, the road interruption condition considers changes in the current road where the vehicle is located. For example, this might involve the following situations: the stop line on the current road disappears; the current road is a ramp, i.e., a merging / converging zone appears; the vehicle's travel length on the current road exceeds a second length threshold, where the second length threshold can be greater than the first length threshold; or the vehicle makes a U-turn. Therefore, at least one of the above situations can be used as the road interruption condition. When interrupting based on the road interruption condition, the vehicle's travel length on the current road is reset to zero to facilitate subsequent determination of whether to interrupt.

[0085] It is worth noting that the vehicle will turn around when it meets the conditions for doing so. These conditions include at least one of the following: the turn signal is on; the steering wheel angle meets the preset angle change conditions, such as turning from 0 degrees to 180 degrees; or the vehicle's posture meets the preset posture change conditions, such as changing from due north to due south.

[0086] In some embodiments, if the local map interruption condition or the road interruption condition involves the vehicle's chassis information (such as steering wheel angle), the vehicle camera and chassis sensor (the sensor used to collect chassis information) are time-aligned to obtain chassis information corresponding to each frame of image, thereby enabling a more accurate determination of whether the local map interruption condition or the road interruption condition is met.

[0087] In some embodiments, when performing semantic environment fusion processing on multiple local semantic environments between two adjacent road interruptions, the semantic environment processing method for intelligent driving further includes: determining lane topology relationships based on map features in multiple local semantic environments between two adjacent road interruptions; and determining road topology relationships based on vehicle trajectories in multiple local semantic environments between two adjacent road interruptions.

[0088] Here, the lane topology (the relationship between lanes) is determined based on map features in multiple local semantic environments between two adjacent road breaks, and this lane topology is reflected in the global semantic environment. The map features used to determine the lane topology may include at least one of the following: road guidance signs (such as straight signs, left turn signs, U-turn signs, etc.) and traffic lights.

[0089] Simultaneously, based on the vehicle trajectory in multiple local semantic environments between two adjacent road breaks, the road topology relationships (associations between roads) between different roads are determined, and these relationships are reflected in the global semantic environment. For example, if the vehicle trajectory is found to be moving from road A to road B through multiple local semantic environments between two adjacent road breaks, then a topological relationship (connection relationship) exists between road A and road B. Through this method, lane topology relationships and road topology relationships can be accurately determined, improving the accuracy of the constructed global semantic environment.

[0090] like Figure 3A As shown, the embodiments of this application can construct semantic environments online and at low cost without the need for additional sensors or a large number of professional personnel. At the same time, by breaking down data according to local map breaking conditions and road breaking conditions, data stacking can be reduced, structured road data can be reasonably expressed, and the accuracy of the constructed semantic environment can be improved.

[0091] In some embodiments, see Figure 3B , Figure 3B This is a flowchart illustrating a semantic environment processing method for intelligent driving provided in an embodiment of this application. Figure 3A Step 102 shown can be implemented through steps 201 to 202, which will be explained in conjunction with each step.

[0092] In step 201, target detection processing is performed on each frame of the image to obtain candidate map elements in each frame of the image and the prediction probability corresponding to the candidate map elements.

[0093] Here, target detection processing is performed on each frame of the image to obtain candidate map elements (or candidate targets) and the prediction probabilities corresponding to the candidate map elements in each frame of the image. Each frame of the image may include one or more candidate map elements. The prediction probability corresponding to the candidate map element represents the probability that the candidate map element is a real map element. The prediction probability corresponding to the candidate map element should be greater than a preset probability lower limit (such as 0.5).

[0094] It is worth noting that candidate map features can be described by a series of points; candidate map features have image attributes, including but not limited to location and orientation.

[0095] In some embodiments, each frame of an image can be processed by a deep learning model that supports the detection of multiple types of map features; alternatively, each frame of an image can be processed by a deep learning model corresponding to each type of map feature (such as a lane line detection model, a road arrow detection model, etc.), meaning that a deep learning model only supports the detection of the corresponding type of map feature.

[0096] In step 202, candidate map elements in multiple frames of images are tracked according to the predicted probabilities corresponding to the candidate map elements, and the successfully tracked candidate map elements are determined as the extracted map elements.

[0097] Since multiple frames of images are acquired periodically by the vehicle-mounted camera, the true map features exhibit continuity across these frames, meaning they don't only appear in a single frame. Based on this characteristic, candidate map features in the multiple frames are tracked according to their predicted probabilities. Candidate map features that fail to track are considered false detections (i.e., not true map features), while those that are successfully tracked are considered true map features. This improves the accuracy of the extracted map features, thereby enhancing the accuracy of the subsequently constructed semantic environment. This application does not limit the target tracking algorithm used, such as Simple Online and Realtime Tracking (SORT) algorithms.

[0098] It is worth noting that, considering the potential existence of multiple types of map features, tracking processing is performed separately for each type of map feature to improve the accuracy of the tracking process. For example, candidate map features that might be lane lines in multiple frames are tracked separately, as are candidate map features that might be road arrows in multiple frames. For ease of understanding, the tracking processing discussed below refers to tracking processing of a single type of map feature.

[0099] It is worth noting that, considering that changes in map features may occur when the local map interruption condition is met, the multi-frame images targeted by the tracking processing can refer to the multi-frame images after a certain interruption based on the local map interruption condition and before the next interruption. In this way, the interruption operation can be avoided from having an adverse effect on the tracking processing process, thereby improving the accuracy of the tracking processing.

[0100] In some embodiments, the above-described tracking process of candidate map elements in multiple frames of images based on the predicted probabilities corresponding to the candidate map elements can be implemented as follows: An active tracking object is generated based on the first candidate map element in the initial frame image, and the confidence level of the active tracking object is initialized; wherein, the initial frame image represents at least one frame image with the earliest acquisition time among the multiple frames; each frame image acquired after the initial frame image is traversed, and the following processing is performed on the traversed images: the first candidate map element in the traversed images is matched with the active tracking object, and the confidence level of the corresponding active tracking object is incremented when a match is successful; the second candidate map element in the traversed images is... The algorithm performs matching processing on active tracking objects that fail to match, and increments the confidence level of the corresponding active tracking object when a match is successful. It then matches the first candidate map feature that fails to match with the lost tracking object in the traversed image, and identifies the corresponding lost tracking object as an active tracking object when a match is successful. Active tracking objects that fail to match are identified as lost tracking objects. The confidence level of the lost tracking object is decremented, and tracking of candidate map features that successfully match the lost tracking object with a confidence level reaching a confidence threshold is considered failed. Specifically, the predicted probability of the first candidate map feature is greater than a first probability threshold, and the predicted probability of the second candidate map feature is less than a second probability threshold.

[0101] This provides an example of a tracking process. First, candidate map features obtained through object detection can be divided into first candidate map features and second candidate map features. The predicted probability of the first candidate map feature is greater than a first probability threshold, and the predicted probability of the second candidate map feature is less than a second probability threshold. The first probability threshold is greater than or equal to the second probability threshold; for example, the first probability threshold is 0.8, and the second probability threshold is 0.7. Since the first candidate map feature is more likely to be a real map feature, an active tracking object is generated based on the first candidate map feature in the initial frame image, and the confidence level of the active tracking object is initialized (e.g., initialized to 1). This ensures that the starting point of the tracking has high accuracy. The initial frame image represents at least one frame image with the earliest acquisition time among multiple frames. There may be one or more active tracking objects. For example, if the initial frame image is the first frame image, a matching tracking object is generated based on the first candidate map feature in the first frame image to serve as the active tracking object. It is worth noting that the tracking object involved in this embodiment is a virtual object constructed for implementing the tracking process and has no practical significance.

[0102] Then, based on the acquisition time from early to late, each frame acquired after the initial frame is sequentially traversed. The subsequent process will be explained using any frame acquired during the traversal as an example. For example, the first candidate map feature in the traversed image is matched with the active tracking object, and the confidence of the corresponding active tracking object is incremented (e.g., incremented by 1) when a match is successful. Then, the second candidate map feature in the traversed image is matched with the active tracking object that failed to match (here, it means that it failed to match the first candidate map feature in the traversed image), and the confidence of the corresponding active tracking object is incremented when a match is successful. Then, the first candidate map feature that failed to match (meaning that it failed to match the active tracking object) in the traversed image is matched with the lost tracking object, and the lost tracking object is identified as the active tracking object when a match is successful. Then, the active tracking object that failed to match (meaning that it failed to match both the first and second candidate map features in the traversed image) is identified as the lost tracking object. Then, the confidence of the lost tracking object is decremented (e.g., decremented by 1), and the tracking of candidate map features that successfully matched the lost tracking object with a confidence threshold (such as 0) is determined to be untracked.

[0103] Upon completion of the traversal, it is determined that the candidate map features that successfully match the active tracking object have been successfully tracked.

[0104] The above method combines active and lost tracking objects for tracking processing. Tracking objects that are successfully matched become active tracking objects and their credibility is increased, while tracking objects that are not successfully matched become lost tracking objects and their credibility is reduced. This dynamic update mechanism helps to improve the accuracy of tracking and avoid invalid or erroneous tracking.

[0105] In some embodiments, the initial frame image includes the first frame image in a multi-frame image and a verification frame image acquired after the first frame image. The above-mentioned generation of active tracking objects based on the first candidate map features in the initial frame image can be achieved in the following manner: generating matching initial tracking objects based on the first candidate map features in the first frame image; matching the first candidate map features in the verification frame image with the initial tracking objects; and determining the corresponding initial tracking object as an active tracking object when the match is successful.

[0106] Here, the initial frame image includes the first frame image in a multi-frame image set, and a verification frame image acquired after the first frame image. The verification frame image can include one or more frames; for example, it could include only the second frame image, or it could include both the second and third frames. In this case, a matching initial tracking object is generated based on the first candidate map features in the first frame image. Since the initial tracking object is generated solely based on the first frame image and is not sufficiently reliable, the first candidate map features in the verification frame image are matched against the initial tracking object. If a match is successful, the corresponding initial tracking object is designated as an active tracking object. If the initial tracking object fails to match any of the first candidate map features in the verification frame image, the initial tracking object is deleted. This method further ensures a high degree of accuracy in the starting point of the tracking.

[0107] In some embodiments, the above-mentioned matching process between the first candidate map features in the traversed images and the active tracking object can be achieved by performing image attribute fusion processing on the image attributes of all candidate map features in the historical frame images that successfully match the active tracking object, to obtain the image attributes of the active tracking object; wherein, the historical frame images represent images acquired before the traversed images; and the first candidate map features in the traversed images are matched with the active tracking object on the image attributes.

[0108] Here, object detection processing can obtain the image attributes (such as position, orientation, etc.) of candidate map features in the image. Therefore, the matching process mentioned above can refer to matching image attributes. For example, image attribute fusion processing is performed on the image attributes of all candidate map features that successfully match the active tracking object in historical frame images to obtain the image attributes of the active tracking object. Here, historical frame images refer to images acquired before the images traversed. The method of image attribute fusion processing is not limited; for example, the Kalman filter algorithm can be used. At the same time, the image attributes of the first candidate map feature in the traversed images are known. Therefore, the first candidate map feature in the traversed images is matched with the active tracking object in terms of image attributes. Here, the conditions for successful matching are not limited; for example, the similarity between image attributes can be greater than a similarity threshold.

[0109] Since image attributes are the representation of candidate map features in an image, matching image attributes can improve the interpretability and effectiveness of the matching process.

[0110] In some embodiments, after determining the corresponding lost tracking object as an active tracking object when a match is successful, the semantic environment processing method for intelligent driving further includes: performing matching processing on the first candidate map features that failed to match in the traversed image with the unconfirmed tracking object, and determining the corresponding unconfirmed tracking object as an active tracking object when a match is successful; generating a matching unconfirmed tracking object based on the first candidate map features that failed to match in the traversed image.

[0111] Here, the first candidate map features in the traversed images have high reliability. Therefore, the first candidate map features that failed to match in the traversed images (meaning they failed to match both active and lost tracking objects) are matched with unconfirmed tracking objects, and when a match is successful, the corresponding unconfirmed tracking object is identified as an active tracking object. Then, matching unconfirmed tracking objects are generated based on the first candidate map features that failed to match in the traversed images (meaning they failed to match both active and lost tracking objects and unconfirmed tracking objects).

[0112] By reconfirming the first candidate map features that failed to match using the above method, we can avoid missed detections and effectively deal with the situation where new map features appear in the intermediate frame image.

[0113] like Figure 3B As shown, in this embodiment of the application, target detection processing is performed on each frame of image to obtain candidate map elements in each frame of image and the prediction probability corresponding to the candidate map elements. Based on the prediction probability corresponding to the candidate map elements, the candidate map elements in multiple frames of images are tracked, and the successfully tracked candidate map elements are determined as the extracted map elements. This can improve the accuracy of the extracted map elements, thereby improving the accuracy of the semantic environment subsequently constructed.

[0114] In some embodiments, see Figure 3C , Figure 3C This is a flowchart illustrating a semantic environment processing method for intelligent driving provided in an embodiment of this application. Figure 3A Step 104 shown can be implemented through steps 301 to 305, which will be explained in conjunction with each step.

[0115] In step 301, the vehicle's driving process is subjected to local map interruption processing according to the local map interruption conditions.

[0116] In step 302, for the same map element in the visual point cloud of multiple frames between two adjacent local map interruptions, the multiple points corresponding to the same map element are sorted.

[0117] Here, we provide an example of visual point cloud fusion processing. For ease of understanding, we will use the same map element (named map element A) in multiple frames of visual point cloud between two adjacent local map interruptions as an example.

[0118] First, multiple points representing map element A are identified in the multi-frame visual point cloud, and these points are sorted to avoid the inability to accurately construct map element A in the local semantic environment due to inaccurate order.

[0119] In some embodiments, the above-mentioned sorting process for multiple points corresponding to the same map feature can be achieved as follows: sort the vehicle coordinates corresponding to multiple frames of visual point clouds according to the acquisition time to obtain the vehicle trajectory, and project the vehicle trajectory onto the coordinate axis corresponding to the vehicle's heading in the vehicle coordinate system; project multiple points corresponding to the same map feature onto the coordinate axis corresponding to the vehicle's heading in the vehicle coordinate system; and sort the multiple points corresponding to the same map feature in the vehicle coordinate system according to the vehicle trajectory.

[0120] Here, for each frame of image captured by the vehicle-mounted camera, the corresponding vehicle coordinates can be determined (the vehicle coordinates are included in the vehicle pose), and these vehicle coordinates can be described in a global coordinate system. Then, the vehicle coordinates corresponding to multiple frames of visual point clouds are sorted according to the acquisition time to obtain the vehicle trajectory. This trajectory is then projected onto the coordinate axis corresponding to the vehicle's heading in the vehicle coordinate system. This projection is necessary because most of the vehicle's movement is described by the direction corresponding to its heading. Simultaneously, multiple points corresponding to map element A are projected onto the coordinate axis corresponding to the vehicle's heading in the vehicle coordinate system. Since the projected vehicle trajectory indicates direction, these points are sorted in the vehicle coordinate system according to the direction indicated by the trajectory. This ensures that the sorted points are consistent with the vehicle trajectory in direction, improving the accuracy of the sorting.

[0121] In step 303, the original coordinates of the sorted points are subjected to coordinate fitting processing to obtain the target fitting parameters.

[0122] Here, the original coordinates of multiple sorted points are subjected to coordinate fitting processing to obtain target fitting parameters. Coordinate fitting processing refers to fitting the relationship between different sub-coordinates in the original coordinates (such as the relationship between the x-coordinate and the y-coordinate). That is, the target fitting parameters describe the relationship between different sub-coordinates, such as how to calculate the y-coordinate from the x-coordinate, or how to calculate the x-coordinate from the y-coordinate. The original coordinates can refer to the coordinates in the vehicle coordinate system or the coordinates in the global coordinate system.

[0123] In some embodiments, before performing coordinate fitting processing on the original coordinates of the sorted multiple points to obtain the target fitting parameters, the semantic environment processing method for intelligent driving further includes: performing coordinate fitting processing on the original coordinates of multiple points corresponding to the same map feature to obtain intermediate fitting parameters; calculating the coordinate error based on the original coordinates of the multiple points corresponding to the same map feature and the intermediate fitting parameters to obtain the coordinate error of the multiple points corresponding to the same map feature; and removing points whose coordinate error is greater than the error threshold from the multiple points corresponding to the same map feature.

[0124] Here, coordinate fitting can be performed on the original coordinates of multiple points corresponding to map element A to obtain intermediate fitting parameters. Since the intermediate fitting parameters describe the relationship between different sub-coordinates in the original coordinates, the coordinate error of the multiple points corresponding to map element A can be calculated based on the original coordinates and the intermediate fitting parameters. For example, if the x-coordinate of a point in the original coordinates is x and the y-coordinate is y, and the y-coordinate calculated based on the x-coordinate x and the intermediate fitting parameters is y', then the coordinate error of that point is the absolute value of the difference between y and y'.

[0125] The larger the coordinate error of a point, the more the original coordinates of that point deviate from the general pattern of the original coordinates. Therefore, among the multiple points corresponding to map element A, points with coordinate errors greater than the error threshold are removed. Points with coordinate errors greater than the error threshold can be regarded as outliers and are therefore removed to avoid outliers affecting the construction of map element A in the local semantic environment.

[0126] In step 304, coordinate update processing is performed based on the original coordinates of the sorted points and the target fitting parameters to obtain the target coordinates of the sorted points.

[0127] Here, coordinate updates are performed based on the original coordinates of the sorted points and the target fitting parameters to obtain the target coordinates of the sorted points. For example, if the original coordinates of a point have x as the x-coordinate and y as the y-coordinate, and the target fitting parameters describe how to calculate the y-coordinate from the x-coordinate, then the y-coordinate is calculated from the x-coordinate x and the target fitting parameters, resulting in the final target coordinates of that point having x as the x-coordinate and y as the y-coordinate.

[0128] In step 305, the same map element in the local semantic environment is constructed based on the target coordinates of the sorted points.

[0129] Compared to the original coordinates, the target coordinates obtained through coordinate update processing are more regular and consistent. Therefore, constructing map element A in the local semantic environment based on the target coordinates of multiple sorted points makes the constructed map element A more accurate. The construction method of map element A depends on its actual shape. For example, if map element A is a lane line, the target coordinates of multiple points can be connected sequentially according to the sorted order to obtain map element A in the local semantic environment; if map element A is a road arrow, the target coordinates of multiple points can be connected sequentially according to the sorted order, and then an outer bounding ring can be added to the shape formed by the connection to obtain map element A in the local semantic environment.

[0130] In some embodiments, the same map feature in the local semantic environment can be constructed based on the target coordinates of multiple sorted and uniformly sampled points, thus avoiding poor construction results caused by excessive point density. The sampling length is not limited.

[0131] In some embodiments, the sorted points can be evenly divided into multiple segments, and steps 303 to 305 described above can be performed on each segment.

[0132] Here, for map element A, such as a roundabout, direct coordinate fitting may not be possible or the results may be poor. Therefore, the sorted points can be evenly divided into multiple segments, and steps 303 to 305 above can be performed on each segment. For example, if the sorted points are point 1, point 2, ..., point 9, they can be evenly divided into three segments: the first segment includes points 1, 2, and 3; the second segment includes points 4, 5, and 6; and the third segment includes points 7, 8, and 9. The length of the segments is not limited.

[0133] like Figure 3C As shown, the embodiments of this application obtain the target coordinates by updating the original coordinates of the points, making the target coordinates more regular and uniform, thereby making the map elements constructed in the local semantic environment smoother; at the same time, sorting multiple points enables the map elements in the local semantic environment to be constructed in an orderly manner, which can improve its accuracy.

[0134] The following will describe an exemplary application of the embodiments of this application in a practical application scenario. The embodiments of this application can be implemented based on the existing sensors on the vehicle side, such as the forward-looking camera, pose sensor, and chassis sensor in the intelligent driving vehicle. There is no need to add additional professional data acquisition equipment. When the driver activates the semantic environment construction function during the driving process (or it can be automatically activated by the vehicle), the vehicle side can automatically construct the semantic environment in real time without storing or processing the collected data offline.

[0135] This application provides embodiments such as Figure 4 The diagram shown is a flowchart of a semantic environment processing method for intelligent driving, which will combine... Figure 4 The explanation is presented step by step.

[0136] Step 1: Acquire video data (including multiple frames) from the forward-facing camera, vehicle pose data from the pose sensor, and chassis information from the chassis sensor. Time-align the data from these sensors to obtain the vehicle pose and chassis information corresponding to each frame. The vehicle pose includes, but is not limited to, vehicle coordinates (x, y, z) and vehicle attitude (roll, pitch, yaw). The chassis information includes, but is not limited to, steering wheel angle, steering wheel angular velocity, vehicle speed, and gear information.

[0137] In some embodiments, the image timestamp of the video data (which may be the image timestamp of the first frame) can be used as a benchmark to search the sensor data queues of other sensors. Sensor data with timestamps earlier than the image timestamps in the sensor data queues are deleted, thereby avoiding invalid sensor data occupying storage space. Then, for a certain frame of the video data, the last frame of sensor data (sensor_data1) with a timestamp less than the image timestamp of that frame and the first frame of sensor data (sensor_data2) with a timestamp greater than the image timestamp of that frame are searched in the sensor data queue. Interpolation processing is then performed on sensor_data1 and sensor_data2 to obtain the sensor data corresponding to that frame. In this way, a one-to-one correspondence between the image and the sensor data can be established, overcoming the problem of inconsistent frame rates between different sensors.

[0138] Step 2: Use a deep learning model to perform object detection processing on each frame of the video data to obtain the location of candidate map features in each frame (which can be described by 3D coordinates in the vehicle coordinate system) and the predicted probability of the candidate map features. The types of map features involved in this application embodiment include, but are not limited to, lane lines, road arrows, roadside signs, and stop lines.

[0139] In some embodiments, object detection processing can be performed on each frame of the image using deep learning models (such as lane detection models, road arrow detection models, etc.) corresponding to various types of map features. For example, object detection processing can be performed on each frame of the image using the lane detection model PVALane to obtain the position of candidate lane lines in each frame of the image and the prediction probability corresponding to the candidate lane lines.

[0140] Step 3: Model each type of map feature, construct a tracking model for each type of map feature, and track the detected candidate map features of that type according to the tracking model. Output and record the tracked ID, location, feature type, etc., and determine the successfully tracked candidate map features as the extracted map features. The tracking process should robustly output the tracking results, maximizing the use of the detection results provided by the deep learning model.

[0141] In some embodiments, step 3 can be implemented by steps 3.1 to 3.10 below. An example will be given by tracking a single type of candidate map feature.

[0142] Step 3.1: For each frame of image, distinguish between high-scoring candidate map features (corresponding to the first candidate map features mentioned above) and low-scoring candidate map features (corresponding to the second candidate map features mentioned above) based on the detection results output by the model.

[0143] Step 3.2: Generate a matching initial tracker (corresponding to the initial tracking object above) based on the high-scoring candidate map features in the first frame image.

[0144] Step 3.3: For the verification frame image following the first frame image (such as the second frame image), match the high-scoring candidate map features in the verification frame image with the initial tracker. If the match is successful, the initialization is successful, and the initial tracker is set as the active tracker (corresponding to the active tracking object mentioned above). At the same time, the confidence of the active tracker is initialized (e.g., initialized to 1). If all high-scoring candidate map features in the verification frame image fail to match the initial tracker, the initial tracker is deleted. Then, traverse each frame image following the verification frame image and execute steps 3.4 to 3.10 for each traversed image.

[0145] Step 3.4: Match the high-scoring candidate map features in the traversed images with the active tracker. If a match is successful, increase the confidence of the successful active tracker by 1.

[0146] Step 3.5: Match the low-scoring candidate map features in the traversed images with the active trackers that failed to match in Step 3.4. If a match is successful, increase the confidence of the successful active tracker by 1.

[0147] Step 3.6: Match the high-scoring candidate map features that failed to match in Step 3.4 with the lost tracker (corresponding to the lost tracking object mentioned above). If the match is successful, put the successfully matched lost tracker into the active tracker queue, that is, as the active tracker.

[0148] Step 3.7: Match the high-scoring candidate map features that failed to match in Step 3.6 with the unconfirm trackers (corresponding to the unconfirmed tracking objects mentioned above). If the match is successful, the successfully matched unconfirm trackers are added to the active tracker queue, i.e., they are used as active trackers.

[0149] Step 3.8: Add the active trackers that failed to match in Step 3.5 to the lost tracker queue, i.e., make them lost trackers.

[0150] Step 3.9: Generate matching trackers based on the high-scoring candidate map features that failed to match in Step 3.7, and add them to the unconfirm tracker queue.

[0151] Step 3.10: Decrement the credibility of all lost trackers in the lost tracker queue by 1. When the credibility of a lost tracker reaches 0, delete that lost tracker.

[0152] Step 4: Based on the camera parameters of the vehicle-mounted camera and the vehicle pose corresponding to each frame, project the map elements in the multiple frames from the vehicle coordinate system to the global coordinate system to obtain the visual point cloud corresponding to each frame.

[0153] Step 5: Based on the local map interruption conditions, perform local map interruption processing on the vehicle's driving process, and perform visual point cloud fusion processing on the multi-frame visual point clouds between two adjacent local map interruptions to obtain the local semantic environment (or local map vector data). Visual point cloud fusion processing can be understood as first overlaying the same map element in multiple frames of visual point clouds, and then generating the corresponding vector data. The local map interruption conditions include at least one of the following: the number of lanes on the current road where the vehicle is located changes; the vehicle's travel length on the current road exceeds a first length threshold. By interrupting the collected data according to the local map interruption conditions, it is not only beneficial to build the correlation between various map elements, but also to reduce the consumption of computing power and memory by each module during vehicle-side mapping, thereby improving the system's execution efficiency.

[0154] In some embodiments, when the vehicle's driving process is interrupted according to the interruption conditions of the local map, the tracking process is also interrupted, that is, the tracking process is restarted after the interruption.

[0155] In some embodiments, step 5 can be implemented by the following steps 5.1 to 5.4. Taking a certain map feature, such as lane lines, as an example, the process of constructing lane lines in a local semantic environment is illustrated.

[0156] Step 5.1: Sort multiple points corresponding to the same lane line in the multi-frame visual point cloud to avoid the final constructed lane line from folding back. For example, the vehicle coordinates corresponding to the multi-frame visual point cloud can be sorted according to the acquisition time to obtain the vehicle trajectory, and the vehicle trajectory can be projected onto the coordinate axis corresponding to the vehicle's heading in the vehicle coordinate system; multiple points corresponding to the same lane line can be projected onto the coordinate axis corresponding to the vehicle's heading in the vehicle coordinate system; and multiple points corresponding to the same lane line can be sorted in the vehicle coordinate system according to the vehicle trajectory.

[0157] Step 5.2: Divide the sorted points into segments of a certain length. For each segment, iteratively calculate the parameters of the fitted point cloud using the least squares method until the loss function is minimized (i.e., coordinate fitting is performed). For ease of distinction, the fitted parameters obtained here are called intermediate fitted parameters. The purpose of segmentation is to ensure that each segment can be fitted using the least squares method, improving fitting accuracy. This is suitable for situations such as roundabouts where direct fitting is not possible.

[0158] This application does not limit the loss function used in its embodiments. For example:

[0159]

[0160] Where n represents the number of points, x i The x-coordinate of the point in its original coordinates, y i The ordinate represents the original coordinates of the point, and a, b, and c are the fitting parameters.

[0161] Once the coordinate fitting process is complete, the coordinate fitting equation can be output:

[0162] y = ax 2 +bx+c

[0163] Step 5.3: Using the intermediate fitting parameters obtained in Step 5.2, calculate the coordinate error of all points and remove points whose coordinate error is greater than the error threshold.

[0164] Step 5.4: For the remaining points after step 5.3, recalculate the parameters of the fitted point cloud using the least squares method until the loss function is minimized, and obtain the target fitted parameters.

[0165] Step 5.5: Update the original coordinates of the remaining points after step 5.3 according to the target fitting parameters to obtain the target coordinates, and connect the target coordinates of each point according to the order of the points to obtain the lane line point string result, that is, construct the lane lines in the local semantic environment. Among them, multiple points can be sampled evenly to avoid the point density being too dense, which would result in poor lane line effect.

[0166] It is worth noting that the above method is also applicable to other map elements and can satisfy the fitting of lines of various shapes. For example, the construction method of stop lines and lane lines in the local semantic environment is similar, while road arrows need to be expressed by an outer bounding circle.

[0167] Step 6: Perform road interruption processing on the vehicle's driving process according to the road interruption conditions, and perform semantic environment fusion processing on multiple local semantic environments between two adjacent road interruptions to obtain the global semantic environment. Among them, the road interruption conditions include at least one of the following: the stop line of the current road where the vehicle is located disappears; the current road where the vehicle is located is a ramp; the driving length of the vehicle on the current road is greater than a second length threshold; the vehicle makes a U-turn.

[0168] In some embodiments, during the semantic environment fusion process, redundant data (or overlapping data, i.e., identical data) between different local semantic environments are deleted, thereby reducing the redundancy of the global semantic environment and improving its accuracy.

[0169] In some embodiments, during the semantic environment fusion process, lane topology relationships are determined based on map features in multiple local semantic environments between two adjacent road breaks. For example, lane topology relationships are determined based on road guidance signs (such as straight signs, left turn signs, U-turn signs, etc.) and traffic lights. At the same time, road topology relationships between different roads are determined based on vehicle trajectories in multiple local semantic environments between two adjacent road breaks. For example, if a vehicle trajectory indicates that it has entered road B from road A, then a topological relationship (connection relationship) is determined between road A and road B.

[0170] By breaking roads according to the road breaking conditions, the topological relationships between lanes and roads can be generated online in real time, meeting the requirements for online real-time mapping.

[0171] The embodiments of this application can achieve at least the following technical effects:

[0172] 1) Semantic environments can be built online in real time without the need for additional sensors or a large number of professional personnel, resulting in lower costs.

[0173] 2) By tracking each map element throughout its entire lifecycle from appearance to disappearance, the accuracy and precision of tracking are improved, providing a more accurate basis for point cloud overlay in subsequent local mapping.

[0174] 3) It can robustly fit lane lines of various shapes (straight lines, curves, spirals, etc.) to meet the needs of vectorization.

[0175] 4) The construction of local semantic environment can perform structured processing of roads, which is convenient for downstream modules (planning and control, perception, localization, etc.) of actual intelligent driving after the vehicle-side map is completed.

[0176] 5) The construction of the global semantic environment connects the various local semantic environments and determines the lane topology and road topology, which can meet the usage requirements of downstream modules of intelligent driving (planning and control, perception, positioning, etc.).

[0177] The following continues to describe an exemplary structure of the semantic environment processing device 455 for intelligent driving provided in the embodiments of this application, implemented as a software module. In some embodiments, such as... Figure 2 As shown, the software modules in the intelligent driving semantic environment processing device 455 stored in the memory 450 may include: an acquisition module 4551, used to acquire multiple frames of images captured by the vehicle camera during the vehicle's driving process; an extraction module 4552, used to perform map element extraction processing on each frame of image to obtain map elements in each frame of image; a construction module 4553, used to construct a visual point cloud corresponding to each frame of image based on the map elements in each frame of image; a first interruption module 4554, used to perform local map interruption processing on the vehicle's driving process according to local map interruption conditions, and perform visual point cloud fusion processing on multiple frames of visual point clouds between two adjacent local map interruptions to obtain a local semantic environment; and a second interruption module 4555, used to perform road interruption processing on the vehicle's driving process according to road interruption conditions, and perform semantic environment fusion processing on multiple local semantic environments between two adjacent road interruptions to obtain a global semantic environment.

[0178] In some embodiments, the extraction module 4552 is further configured to: perform target detection processing on each frame image to obtain candidate map elements in each frame image and the prediction probability corresponding to the candidate map elements; perform tracking processing on candidate map elements in multiple frames image according to the prediction probability corresponding to the candidate map elements, and determine the successfully tracked candidate map elements as the extracted map elements.

[0179] In some embodiments, the extraction module 4552 is further configured to: generate active tracking objects based on the first candidate map features in the initial frame image, and initialize the confidence of the active tracking objects; wherein the initial frame image represents at least one frame image with the earliest acquisition time among multiple frames; traverse each frame image acquired after the initial frame image, and perform the following processing on the traversed images: match the first candidate map features in the traversed images with the active tracking objects, and increment the confidence of the corresponding active tracking object when a match is successful; match the second candidate map features in the traversed images with the active tracking objects that failed to match. The algorithm performs the following steps: First, it increments the confidence level of the corresponding active tracking object upon successful matching. Second, it matches the first candidate map element that failed to match with the lost tracking object in the traversed image, identifying the lost tracking object as an active tracking object upon successful matching. Third, it identifies active tracking objects that failed to match as lost tracking objects. Fourth, it decrements the confidence level of the lost tracking object and determines that tracking of candidate map elements that successfully match the lost tracking object with a confidence level reaching a confidence threshold has failed. The predicted probability of the first candidate map element is greater than a first probability threshold, and the predicted probability of the second candidate map element is less than a second probability threshold.

[0180] In some embodiments, the initial frame image includes the first frame image in a multi-frame image and a verification frame image acquired after the first frame image; the extraction module 4552 is further configured to: generate a matching initial tracking object based on the first candidate map feature in the first frame image; perform matching processing between the first candidate map feature in the verification frame image and the initial tracking object, and determine the corresponding initial tracking object as an active tracking object when the matching is successful.

[0181] In some embodiments, the extraction module 4552 is further configured to: perform image attribute fusion processing on the image attributes of all candidate map features that successfully match the active tracking object in the historical frame image to obtain the image attributes of the active tracking object; wherein, the historical frame image represents an image whose acquisition time is before the traversed image; and perform matching processing on the image attributes of the first candidate map feature in the traversed image and the active tracking object.

[0182] In some embodiments, the extraction module 4552 is further configured to: perform matching processing on the first candidate map features that failed to match in the traversed image and the unconfirmed tracking objects, and determine the corresponding unconfirmed tracking objects as active tracking objects when the match is successful; and generate matching unconfirmed tracking objects based on the first candidate map features that failed to match in the traversed image.

[0183] In some embodiments, the first interruption module 4554 is further configured to: perform the following processing on the same map feature in the multi-frame visual point cloud between two adjacent local map interruptions: sorting multiple points corresponding to the same map feature; performing coordinate fitting processing on the original coordinates of the sorted multiple points to obtain target fitting parameters; performing coordinate update processing based on the original coordinates of the sorted multiple points and the target fitting parameters to obtain target coordinates of the sorted multiple points; and constructing the same map feature in the local semantic environment based on the target coordinates of the sorted multiple points.

[0184] In some embodiments, the first interruption module 4554 is further configured to: sort the vehicle coordinates corresponding to the multiple frames of visual point clouds according to the acquisition time to obtain the vehicle trajectory, and project the vehicle trajectory onto the coordinate axis corresponding to the vehicle's heading in the vehicle coordinate system; project multiple points corresponding to the same map feature onto the coordinate axis corresponding to the vehicle's heading in the vehicle coordinate system; and sort the multiple points corresponding to the same map feature in the vehicle coordinate system according to the vehicle trajectory.

[0185] In some embodiments, the first interruption module 4554 is further configured to: perform coordinate fitting processing on the original coordinates of multiple points corresponding to the same map feature to obtain intermediate fitting parameters; calculate the coordinate error based on the original coordinates and intermediate fitting parameters of multiple points corresponding to the same map feature to obtain the coordinate error of multiple points corresponding to the same map feature; and remove points whose coordinate error is greater than the error threshold from the multiple points corresponding to the same map feature.

[0186] In some embodiments, the construction module 4553 is further configured to: obtain the camera parameters of the vehicle-mounted camera and the vehicle pose corresponding to each frame image; and, based on the camera parameters of the vehicle-mounted camera and the vehicle pose corresponding to each frame image, project the map features in each frame image from the vehicle coordinate system to the global coordinate system to obtain the visual point cloud corresponding to each frame image.

[0187] In some embodiments, the local map interruption conditions include at least one of the following: the number of lanes on the current road where the vehicle is located changes; the driving length of the vehicle on the current road is greater than a first length threshold; the road interruption conditions include at least one of the following: the stop line on the current road where the vehicle is located disappears; the current road where the vehicle is located is a ramp; the driving length of the vehicle on the current road is greater than a second length threshold; the vehicle makes a U-turn.

[0188] In some embodiments, the second interruption module 4555 is further configured to: determine lane topology relationships based on map features in multiple local semantic environments between two adjacent road interruptions; and determine road topology relationships based on vehicle trajectories in multiple local semantic environments between two adjacent road interruptions.

[0189] This application provides a computer program product or computer program that includes executable instructions stored in a computer-readable storage medium. An electronic device's processor reads the executable instructions from the computer-readable storage medium and executes the executable instructions, causing the electronic device to perform the semantic environment processing method for intelligent driving described in this application.

[0190] This application provides a computer-readable storage medium storing executable instructions. When the executable instructions are executed by a processor, the processor will execute the semantic environment processing method for intelligent driving provided in this application.

[0191] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0192] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0193] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0194] As an example, executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0195] The above are merely embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A semantic environment processing method for intelligent driving, characterized in that, include: Acquire multiple frames of images captured by the vehicle's onboard camera during the vehicle's driving process; Map feature extraction is performed on each frame of the image to obtain the map features in each frame of the image; Construct a visual point cloud corresponding to each frame of the image based on the map features in each frame. The vehicle's driving process is subjected to local map interruption processing based on local map interruption conditions, and the multi-frame visual point cloud between two adjacent local map interruptions is subjected to visual point cloud fusion processing to obtain the local semantic environment. The vehicle's driving process is interrupted according to the road interruption conditions, and multiple local semantic environments between two adjacent road interruptions are fused to obtain a global semantic environment. The local map interruption conditions include at least one of the following: the number of lanes on the current road where the vehicle is located changes; the driving length of the vehicle on the current road is greater than a first length threshold. The road interruption conditions include at least one of the following: the stop line of the current road where the vehicle is located disappears; the current road where the vehicle is located is a ramp; the driving length of the vehicle on the current road is greater than the second length threshold; the vehicle makes a U-turn.

2. The method according to claim 1, characterized in that, The process of extracting map features from each frame of the image to obtain the map features in each frame includes: Perform target detection processing on each frame of the image to obtain candidate map elements in each frame and the prediction probability corresponding to the candidate map elements; The candidate map elements in the multi-frame images are tracked based on the predicted probabilities corresponding to the candidate map elements, and the successfully tracked candidate map elements are determined as the extracted map elements.

3. The method according to claim 2, characterized in that, The step of tracking candidate map features in the multi-frame images based on the predicted probabilities corresponding to the candidate map features includes: Active tracking objects are generated based on the first candidate map features in the initial frame image, and the credibility of the active tracking objects is initialized; wherein, the initial frame image represents at least one frame image with the earliest acquisition time among the multiple frames; Iterate through each frame acquired after the initial frame and perform the following processing on each frame: The first candidate map features in the traversed images are matched with active tracking objects, and the confidence of the corresponding active tracking object is incremented when a match is successful. The second candidate map features in the traversed image are matched with the active tracking objects that failed to match, and the confidence of the corresponding active tracking object is incremented when the match is successful. The first candidate map feature that failed to match in the traversed image is matched with the lost tracking object, and when the match is successful, the corresponding lost tracking object is determined as the active tracking object. Active tracking objects that fail to match are identified as lost tracking objects; The credibility of lost tracking objects is decremented, and the tracking of candidate map features that successfully match the lost tracking objects with credibility reaching the credibility threshold is determined to be unsuccessful. Among them, the predicted probability of the first candidate map element is greater than the first probability threshold, and the predicted probability of the second candidate map element is less than the second probability threshold.

4. The method according to claim 3, characterized in that, The initial frame image includes the first frame image in the multi-frame image, and the verification frame image whose acquisition time is after the first frame image; The step of generating active tracking objects based on the first candidate map features in the initial frame image includes: Generate a matching initial tracking object based on the first candidate map feature in the first frame image; The first candidate map feature in the verification frame image is matched with the initial tracking object, and the corresponding initial tracking object is determined as the active tracking object when the match is successful.

5. The method according to claim 3, characterized in that, The process of matching the first candidate map features in the traversed images with the active tracking objects includes: Image attribute fusion processing is performed on the image attributes of all candidate map features that successfully match the active tracking object in the historical frame images to obtain the image attributes of the active tracking object; wherein, the historical frame images represent images whose acquisition time is before the images traversed; The first candidate map features in the traversed images are matched with the active tracking objects in terms of image attributes.

6. The method according to claim 3, characterized in that, After determining the corresponding lost tracking object as an active tracking object upon successful matching, the method further includes: The first candidate map element that fails to match in the traversed image is matched with the unconfirmed tracking object, and when the match is successful, the corresponding unconfirmed tracking object is identified as the active tracking object. Based on the first candidate map feature that failed to match in the traversed image, generate a matching unconfirmed tracking object.

7. The method according to claim 1, characterized in that, The step of performing visual point cloud fusion processing on multiple frames of visual point clouds between two adjacent local map interruptions to obtain a local semantic environment includes: For the same map feature in multiple frames of visual point cloud between two adjacent local map interruptions, perform the following processing: Sort the multiple points corresponding to the same map feature; The original coordinates of the sorted points are subjected to coordinate fitting to obtain the target fitting parameters. Based on the original coordinates of the sorted points and the target fitting parameters, coordinate update processing is performed to obtain the target coordinates of the sorted points. The same map element in the local semantic environment is constructed based on the target coordinates of multiple sorted points.

8. The method according to claim 7, characterized in that, The sorting process for multiple points corresponding to the same map feature includes: The vehicle coordinates corresponding to the multi-frame visual point cloud are sorted according to the acquisition time to obtain the vehicle trajectory, and the vehicle trajectory is projected onto the coordinate axis corresponding to the vehicle's heading in the vehicle coordinate system. Project multiple points corresponding to the same map element onto the coordinate axis in the vehicle coordinate system corresponding to the vehicle's heading. In the vehicle coordinate system, multiple points corresponding to the same map element are sorted according to the vehicle trajectory.

9. The method according to claim 7, characterized in that, Before performing coordinate fitting processing on the original coordinates of the sorted points to obtain the target fitting parameters, the method further includes: The original coordinates of multiple points corresponding to the same map feature are subjected to coordinate fitting to obtain intermediate fitting parameters. The coordinate error is calculated based on the original coordinates of multiple points corresponding to the same map feature and the intermediate fitting parameters to obtain the coordinate error of multiple points corresponding to the same map feature. Points with coordinate errors greater than the error threshold are removed from multiple points corresponding to the same map feature.

10. The method according to claim 1, characterized in that, The step of constructing a visual point cloud corresponding to each frame of image based on map features in each frame includes: Obtain the camera parameters of the vehicle-mounted camera and the vehicle's position and pose corresponding to each frame of the image; Based on the camera parameters of the vehicle-mounted camera and the vehicle pose corresponding to each frame of the image, the map elements in each frame of the image are projected from the vehicle coordinate system to the global coordinate system to obtain the visual point cloud corresponding to each frame of the image.

11. The method according to any one of claims 1 to 10, characterized in that, When performing semantic environment fusion processing on multiple local semantic environments between two adjacent road interruptions, the method further includes: The lane topology is determined based on map features in multiple local semantic environments between two adjacent road interruptions; The road topology is determined based on the vehicle trajectories in multiple local semantic environments between two adjacent road breaks.

12. A semantic environment processing device for intelligent driving, characterized in that, include: The acquisition module is used to acquire multiple frames of images captured by the vehicle-mounted camera during the vehicle's driving process; The extraction module is used to extract map features from each frame of the image to obtain the map features in each frame of the image. The building module is used to construct the visual point cloud corresponding to each frame of the image based on the map features in each frame. The first interruption module is used to perform local map interruption processing on the vehicle's driving process according to the local map interruption conditions, and to perform visual point cloud fusion processing on the multi-frame visual point cloud between two adjacent local map interruptions to obtain the local semantic environment. The second interruption module is used to perform road interruption processing on the vehicle's driving process according to the road interruption conditions, and to perform semantic environment fusion processing on multiple local semantic environments between two adjacent road interruptions to obtain a global semantic environment. The local map interruption conditions include at least one of the following: the number of lanes on the current road where the vehicle is located changes; the driving length of the vehicle on the current road is greater than a first length threshold. The road interruption conditions include at least one of the following: the stop line of the current road where the vehicle is located disappears; the current road where the vehicle is located is a ramp; the driving length of the vehicle on the current road is greater than the second length threshold; the vehicle makes a U-turn.

13. An electronic device, characterized in that, include: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, It stores executable instructions for implementing the method of any one of claims 1 to 11 when executed by a processor.

Images