Passable area determination method

Abstract

The embodiment of the invention provides a passable area determination method. The method comprises the following steps: acquiring environment data, collected by a sensor of a vehicle, at the current moment; obtaining passable area reference information corresponding to the current position of the vehicle according to the high-precision map; and inputting the environment data and the passable area reference information into a passable area determination model, and determining passable area initial information corresponding to the current moment.

Description

Technical Field

[0001] This specification relates to the technical field of mobile control, and particularly relates to a method for determining a passable area. Background Art

[0002] In the field of mobile control, for example, in the fields of autonomous driving and intelligent robots, it is a way to realize real-time unmanned operation of a terminal through an automated system based on a processor. The passable area is the area where a vehicle can pass without obstacles. In the field of mobile control, real-time perception of the passable area of a vehicle is extremely important. However, the information of the passable area (such as lane lines and passable area boundary lines) constructed based on a single model and / or rule often has unstable shapes and / or positions, which causes difficulties for downstream mobile control based on the passable area.

[0003] Therefore, there is a need to provide a method for determining a passable area that can provide stable passable area information. Summary of the Invention

[0004] One or more embodiments of this specification provide a method for determining a passable area, which is executed by a vehicle. The method for determining a passable area includes: obtaining environmental data at the current moment collected by sensors of the vehicle; obtaining passable area reference information corresponding to the current position of the vehicle according to a high-precision map; inputting the environmental data and the passable area reference information into a passable area determination model to determine the initial passable area information corresponding to the current moment.

[0005] One or more embodiments of this specification provide a method for determining a passable area, which is executed by a vehicle. The method for determining a passable area includes: obtaining multiple initial passable area information output by multiple models; aligning the multiple initial passable area information in time and space to obtain multiple aligned passable area information; determining the target passable area information corresponding to the current moment according to the multiple aligned passable area information. Brief Description of the Drawings

[0006] This specification will be further described in the form of exemplary embodiments, and these exemplary embodiments will be described in detail through the drawings. These embodiments are not restrictive. In these embodiments, the same numbers represent the same structures, where:

[0007] Figure 1 is an exemplary schematic diagram of an application scenario of a passable area determination system executed by a vehicle according to some embodiments of this specification;

[0008] Figure 2 is an exemplary module diagram of a passable area determination system according to some embodiments of this specification;

[0009] Figure 3is an exemplary flowchart of a passable area determination method shown in some embodiments of this specification;

[0010] Figure 4 is an exemplary flowchart of obtaining environmental data at the current moment shown in some embodiments of this specification;

[0011] Figure 5 is an exemplary flowchart of determining initial information of a passable area corresponding to the current moment shown in some embodiments of this specification;

[0012] Figure 6 is an exemplary flowchart of determining an updated query embedding shown in some embodiments of this specification;

[0013] Figure 7 is another exemplary flowchart of determining initial information of a passable area corresponding to the current moment shown in some embodiments of this specification;

[0014] Figure 8 is another exemplary flowchart of a passable area determination method executed by a vehicle shown in some embodiments of this specification;

[0015] Figure 9 is a flowchart of determining alignment information of multiple passable areas shown in some embodiments of this specification;

[0016] Figure 10 is a flowchart of determining target information of a passable area corresponding to the current moment shown in some embodiments of this specification;

[0017] Figure 11 is an exemplary schematic diagram of an obstacle area shown in some embodiments of this specification. Detailed implementation manners

[0018] To more clearly illustrate the technical solutions of the embodiments of this specification, the following will briefly introduce the accompanying drawings required for the description of the embodiments. Obviously, the accompanying drawings in the following description are only some examples or embodiments of this specification. For those of ordinary skill in the art, without creative efforts, this specification can also be applied to other similar scenarios based on these drawings. Unless obvious from the language context or otherwise stated, the same reference numerals in the figures represent the same structure or operation.

[0019] It should be understood that the "system", "device", "unit" and / or "module" used herein is a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the said words can be replaced by other expressions.

[0020] As shown in this specification and the claims, unless the context clearly indicates otherwise, words such as "a", "an", "one", and / or "the" are not specifically singular and may also include the plural. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0021] Flowcharts are used in this specification to illustrate the operations performed by the system according to the embodiments of this specification. It should be understood that the previous or subsequent operations do not necessarily need to be executed precisely in sequence. On the contrary, the steps can be processed in reverse order or simultaneously. At the same time, other operations can also be added to these processes.

[0022] Figure 1 It is an exemplary schematic diagram of the application scenario of the passable area determination system according to some embodiments of this specification. The systems and methods in this application can be applied to the scenarios of autonomous vehicles, for example, the pure driverless scenario without a driver, the assisted driving module in the vehicle, etc. The systems and methods of this application can also be applied to the scenarios of autonomous mobile robots, for example, food delivery robots, inspection robots, express delivery robots, etc. The vehicles in this application can include various vehicles traveling on the road, such as taxis, private cars, carpooling vehicles, shared vehicles, buses, trucks, passenger cars, work vehicles (such as sanitation vehicles, etc.), robots, or any combination thereof. In some embodiments, the passable area determination system 100 can provide stable passable area information by implementing the methods and / or processes disclosed in this specification.

[0023] In some embodiments, as Figure 1 shown, the passable area determination system 100 can include a vehicle 110, a user terminal 120, a storage device 130, and a network 140.

[0024] Vehicle 110 can be a vehicle for the passable area determination system 100. For example, vehicle 110 can be a vehicle, a robot, etc. In some embodiments, vehicle 110 can receive instructions issued by a processing device and complete corresponding tasks according to the instructions. For example, vehicle 110 can receive instructions such as starting to drive and stopping driving, and automatically control vehicle 110 to complete the corresponding operations. In some embodiments, vehicle 110 can include a positioning component (e.g., a GPS module), a timing component (e.g., a timer), sensors (e.g., a motion sensor, a camera, a lidar, etc.), a communication component (e.g., a GPS communication module, a short-range wireless communication module, etc.), etc. In some embodiments, vehicle 110 can communicate with at least one component of the passable area determination system 100 via network 140. For example, vehicle 110 can send environmental data collected by sensors to the processing device via network 140. In some embodiments, vehicle 110 can include a processing device. The processing device can be used to process data and / or information from at least one component of the passable area determination system 100 or an external data source (e.g., a cloud data center). For example, the processing device can use a passable area determination model to determine the initial passable area information corresponding to the current moment based on environmental data and passable area reference information. For another example, the processing device can obtain the target passable area information based on multiple initial passable area information output by multiple models. In some embodiments, the processing device can include a central processing unit (CPU), a digital signal processor (DSP), a system on chip (SoC), a microcontroller unit (MCU), a computer, a user console, etc. or any combination thereof. In some embodiments, the processing device can include a single server or a server group. The server group can be centralized or distributed. In some embodiments, the processing device can be local or remote. In some embodiments, the processing device can be implemented on a cloud platform. By way of example only, the cloud platform can include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an internal cloud, a multi-layer cloud, etc. or any combination thereof.

[0025] The client 120 can implement the interaction between the user and the passable area determination system 100. In some embodiments, the client 120 can be a terminal device used by the user when using the vehicle. In some embodiments, the client 120 can display the boundary line, road elements, obstacles, etc. of the passable area to the user. In some embodiments, the client 120 can include a mobile device, a tablet computer, a laptop computer, other devices with input and / or output functions, etc. or any combination thereof.

[0026] The storage device 130 can be used to store data, instructions, and / or any other information. For example, the storage device can store high-precision maps, initial information of passable areas corresponding to historical times, etc. In some embodiments, the storage device may include a random access memory (RAM), a read-only memory (ROM), a mass storage device, a removable storage device, a volatile read-write memory, etc., or any combination thereof. In some embodiments, the storage device can be integrated or included in one or more other components (such as a processing device, a vehicle 110, a client 120) of the passable area determination system 100.

[0027] The network 140 can facilitate the exchange of information and / or data. In some embodiments, one or more components (such as a processing device, a vehicle 110, a client 120, a storage device 130) of the passable area determination system 100 can send information and / or data to other components of the passable area determination system 100 through the network 140. For example, the processing device can obtain environmental data from the vehicle 110 via the network 140. For another example, the processing device can obtain a high-precision map from the storage device 130 via the network 140. In some embodiments, the network 140 can include any one or more of a wired network or a wireless network. In some embodiments, the network 140 can include a cable network, an optical fiber network, a telecommunications network, the Internet, a local area network (LAN), a wide area network (WAN), a wireless local area network (WLAN), a metropolitan area network (MAN), a public switched telephone network (PSTN), a Bluetooth network, a ZigBee network, a near field communication (NFC), an in-device bus, an in-device line, a cable connection, etc., or any combination thereof. In some embodiments, the network connection between the components of the passable area determination system 100 can adopt one of the above methods or multiple methods. In some embodiments, the network can be various topological structures such as point-to-point, shared, centralized, etc., or a combination of multiple topological structures.

[0028] It should be noted that the passable area determination system 100 is provided for illustrative purposes only and is not intended to limit the scope of this specification. For those of ordinary skill in the art, various changes and modifications can be made according to the description of this specification. For example, the passable area determination system 100 can also include a database, an information source, etc. For another example, the passable area determination system 100 can be implemented on other devices to achieve similar or different functions. However, these changes and modifications will not deviate from the scope of this specification.

[0029] Figure 2 is an exemplary module diagram of a passable area determination system shown in some embodiments of this specification. In some embodiments, the passable area determination system 200 can be implemented through a processing device. In some embodiments, as Figure 2As shown, the passable area determination system 200 may include a first acquisition module 210, a second acquisition module 220, a first determination module 230, a third acquisition module 240, an information alignment module 250, and / or a second determination module 260.

[0030] The first acquisition module 210 may be configured to acquire environmental data at the current moment collected by sensors of the vehicle. In some embodiments, the first acquisition module 210 may perform one or more of the following operations: acquire image data at the current moment collected by a camera of the vehicle; convert the image data into bird's-eye view image data; acquire point cloud data at the current moment collected by a lidar of the vehicle; convert the point cloud data into bird's-eye view point cloud data; determine environmental data based on the bird's-eye view image data and the bird's-eye view point cloud data. For a detailed description of the first acquisition module 210, reference may be made to step 310.

[0031] The second acquisition module 220 may be configured to acquire passable area reference information corresponding to the current position of the vehicle according to a high-precision map. For a detailed description of the second acquisition module 220, reference may be made to step 320.

[0032] The first determination module 230 can be used to input environmental data and passable area reference information into the passable area determination model to determine the initial passable area information corresponding to the current moment. In some embodiments, the passable area determination model can include an attention mechanism model. In some embodiments, the first determination module 230 can perform one or more of the following operations: based on the passable area determination model, convert the passable area reference information into a prior query embedding; input the initial query embedding into the passable area determination model; use the passable area determination model to determine the updated query embedding corresponding to the current moment according to the initial query embedding, the prior query embedding, and the environmental data; based on the passable area determination model, decode the updated query embedding corresponding to the current moment to obtain the initial passable area information corresponding to the current moment. In some embodiments, the first determination module 230 can perform one or more of the following operations: obtain the initial passable area information corresponding to the historical time; convert the initial passable area information corresponding to the historical time into the vehicle coordinate system corresponding to the current moment to obtain the passable area historical information; input the passable area historical information into the passable area determination model to determine the initial passable area information corresponding to the current moment. In some embodiments, the first determination module 230 can perform one or more of the following operations: based on the passable area determination model, convert the initial passable area information corresponding to the historical time into the vehicle coordinate system corresponding to the current moment to obtain a position embedding; based on the passable area determination model, generate a tracking query embedding according to the position embedding and the updated query embedding corresponding to the historical time; based on the passable area determination model, determine the updated query embedding corresponding to the current moment according to the tracking query embedding. For a detailed description of the first determination module 230, refer to step 330.

[0033] The third acquisition module 240 can be used to acquire multiple initial passable area information output by multiple models. For a detailed description of the third acquisition module 240, refer to step 810.

[0034] The information alignment module 250 can be used to align multiple initial passable area information in time and space to obtain multiple aligned passable area information. In some embodiments, the information alignment module 250 can perform one or more of the following operations: according to the position relationship corresponding to the vehicle at the current moment, convert the multiple initial passable area information corresponding to multiple models to the current moment to obtain multiple time-aligned information corresponding to multiple models; cluster the multiple time-aligned information to obtain one or more obstacle areas; determine a bird's-eye view area according to the current position of the vehicle, and the bird's-eye view area includes multiple grids; project one or more obstacle areas onto the bird's-eye view area to obtain multiple aligned passable area information, and each obstacle area corresponds to one or more grids. For a detailed description of the information alignment module 250, refer to step 820.

[0035] The second determination module 260 can be used to determine the target information of the passable area corresponding to the current moment according to the alignment information of multiple passable areas. In some embodiments, the second determination module 260 can perform one or more of the following operations: for each grid corresponding to an obstacle area, determine a jitter weight according to at least one of the confidence level, height information, occupancy type, and model source of the obstacle area corresponding to the grid; optimize one or more obstacle areas based on the jitter weight; and determine the target information of the passable area corresponding to the current moment based on the optimized obstacle area. In some embodiments, the second determination module 260 can determine the confidence level of the initial information of the passable area corresponding to the current moment based on the initial information of the passable area corresponding to the current moment and the initial information of the passable area corresponding to the historical moment for each model. For a detailed description of the second determination module 260, reference can be made to step 830.

[0036] It should be noted that the above description of the passable area determination system 200 and its modules is only for convenience of description and does not limit this specification to the scope of the examples given. It can be understood that for those skilled in the art, after understanding the principle of the device, they may, without departing from this principle, make any combination of the various modules, or form a sub-device and connect it to other modules. In some embodiments, Figure 2 the modules disclosed in may be different modules in a system, or a module may implement the functions of two or more of the above modules. For example, each module can share a storage module, or each module can have its own storage module respectively. Such deformations are all within the protection scope of this specification.

[0037] Figure 3 is an exemplary flowchart of a passable area determination method according to some embodiments of this specification. In some embodiments, process 300 can be executed by the passable area determination system 100 (for example, a processing device) or the passable area determination system 200. For example, process 300 can be stored in a storage device in the form of a program or an instruction. When the passable area determination system 100 (for example, a processing device) or the passable area determination system 200 executes the instruction, process 300 can be implemented. The operation schematic diagram of process 300 presented below is illustrative. In some embodiments, the process can be completed by using one or more additional operations not described and / or one or more operations not discussed. Additionally, Figure 3 the order of the operations of process 300 shown and described below is not restrictive.

[0038] Step 310, obtain the environmental data at the current moment collected by the sensors of the vehicle. Specifically, step 310 can be executed by the first acquisition module 210.

[0039] The environmental data at the current moment can be the environmental information corresponding to the current position of the vehicle. The current position of the vehicle can include the positioning position of the vehicle at the current moment. In some embodiments, the first acquisition module 210 can acquire the current position of the vehicle through the positioning component of the vehicle. The environmental information corresponding to the current position of the vehicle can be the relevant information of the target object within a preset range (for example, 10m, 20m, 50m, etc.) of the current position of the vehicle.

[0040] In some embodiments, the target object can include, but is not limited to, road elements (such as lanes, intersections, road signs), obstacles (obstacles on the road, landmark buildings on both sides of the road), other moving objects (such as pedestrians, other vehicles), etc. In some embodiments, the environmental information can include road characteristics, obstacle characteristics, relevant characteristics of other moving objects, etc.

[0041] The road characteristics can be the characteristics reflecting the road information. In some embodiments, the road characteristics can include at least one of the road structure characteristics, traffic control sign information, and traffic flow characteristics. The road structure characteristics are the characteristics that can reflect the road structure and the relevance between roads. For example, the road structure characteristics can include road types (such as motor vehicle lanes, non-motor vehicle lanes, sidewalks, etc.), the number of roads, the number of confluence lanes, intersection types (such as crossroads, T-junctions, etc.), the number of intersections, the road connection structure, the road topology structure, etc. The traffic control sign information can be the information of traffic indication signs and / or traffic control signs. For example, the traffic control sign information can include signs or pavement markings of traffic-sensitive areas (such as schools, hospitals, etc.), driving rules (such as speed limits, one-way streets, lane lines, turning signs, sidewalks, stop lines, etc.), and traffic signal logic rules. For example, the current indication color of the traffic signal (such as red, green, and yellow lights), the remaining time of the current indication color, and the color change rule (such as 20s red - 5s yellow - 20s green - 5s yellow). In some embodiments, the road characteristics can include the position, size, shape, category, etc. of the road elements.

[0042] The obstacle characteristics can be the characteristics reflecting the obstacle information. In some embodiments, the obstacle characteristics can include the position, size, shape, category, etc. of obstacles on the road (such as utility poles, street lights, road dividers, etc.), buildings on both sides of the road and their entrances and exits.

[0043] The relevant characteristics of other moving objects can be the characteristics reflecting the information of other moving objects. In some embodiments, the relevant characteristics of other moving objects can include the moving trajectory, moving speed, position, moving direction, size, category, etc. of other moving objects.

[0044] In some embodiments, the environmental data may include image features from an aerial view perspective. In some embodiments, the first acquisition module 210 may acquire environmental data using sensors (e.g., cameras, lidar, etc.). For a detailed description of acquiring the environmental data at the current moment collected by the vehicle's sensors, reference may be made to Figure 4 and its related descriptions, which will not be elaborated herein.

[0045] In some embodiments, the environmental data may further include the traffic period during which the vehicle is currently traveling (e.g., morning rush hour, evening rush hour, idle period), weather information (e.g., heavy fog, rainy day), etc. In some embodiments, the first acquisition module 210 may acquire environmental data from big data (e.g., including real-time traffic trajectories uploaded by vehicles and pedestrians).

[0046] Step 320, according to the high-precision map, acquire the passable area reference information corresponding to the current position of the vehicle. Specifically, step 320 may be executed by the second acquisition module 220.

[0047] The passable area is the area where the vehicle can travel without obstacles. The passable area reference information may be the relevant information of the passable area marked on the map. In some embodiments, the passable area reference information corresponding to the current position of the vehicle may include lane lines, boundary lines of the map passable area, etc. within a preset range (e.g., 10 meters, 20 meters, 50 meters, etc.) of the vehicle's current position.

[0048] Specifically, the second acquisition module 220 may determine the passable area marked on the map according to the markings of the high-precision map. Further, the second acquisition module 220 may use an extraction model to extract the passable area reference information from the high-precision map stored in the storage device 130. In some embodiments, the extraction model may include but is not limited to the Visual Geometry Group Network model, Inception NET model, Fully Convolutional Network model, segmentation network model, and Mask-Region Convolutional Neural Network model, etc.

[0049] Step 330, input the environmental data and the passable area reference information into the passable area determination model to determine the initial passable area information corresponding to the current moment. Specifically, step 330 may be executed by the first determination module 230.

[0050] The initial information of the passable area can be information related to the initial passable area determined based on environmental data and a high-precision map. In some embodiments, the initial information of the passable area corresponding to the current moment may include lane lines within a preset range of the current position of the vehicle (for example, 10 meters, 20 meters, 50 meters, etc.), boundary lines of the initial passable area, and the like.

[0051] The passable area determination model can be a model for perceiving passable area information. In some embodiments, the passable area determination model may include an attention mechanism model. For example, the passable area determination model can be an Encoder-Decoder model, a Self-Attention model, a Transformer model, a BERT model, etc. The attention mechanism model can include an encoding part and a decoding part, which can perform interactive encoding and decoding on multiple input encodings respectively, so that the decoded features can fuse the information of multiple encodings.

[0052] For a detailed description of determining the initial information of the passable area corresponding to the current moment based on the passable area determination model, reference can be made to Figure 5 and its related description, which will not be elaborated here.

[0053] In some embodiments of this specification, including an attention mechanism model in the passable area determination model can better perform interactive encoding on various data such as environmental data and reference information of the high-precision map, so that the initial information of the passable area can fully fuse the offline high-precision map information and real-time environmental data information.

[0054] In some embodiments, the first determination module 230 can also obtain the initial information of the passable area corresponding to the current moment based on the initial information of the passable area corresponding to the historical time. For a detailed description of obtaining the initial information of the passable area corresponding to the current moment based on the initial information of the passable area corresponding to the historical time, reference can be made to Figure 7 and its related description, which will not be elaborated here.

[0055] In some embodiments of this specification, determining the initial information of the passable area corresponding to the current moment based on environmental data and passable area reference information can, by fusing the reference information of the high-precision map, enable the initial information of the passable area to provide more stable real-time detection results of lane lines and drivable area boundaries.

[0056] Figure 4 is an exemplary flowchart of obtaining environmental data at the current moment shown in some embodiments of this specification. In some embodiments, Figure 4It can be executed by the passable area determination system 100 (e.g., a processing device) or the passable area determination system 200 (e.g., the first acquisition module 210). For example, Figure 4 It can be stored in a storage device in the form of a program or instructions. When the passable area determination system 100 (e.g., a processing device) or the passable area determination system 200 (e.g., the first acquisition module 210) executes the instructions, the process 400 can be implemented. The operation schematic diagram of the process 400 presented below is illustrative. In some embodiments, the process can be completed by using one or more additional operations not described and / or one or more operations not discussed. Additionally, Figure 4 the order of the operations of the process 400 shown and described below is not restrictive.

[0057] Step 410, acquire the image data at the current moment collected by the camera of the vehicle.

[0058] The image data at the current moment can be an image describing the environmental information corresponding to the current position of the vehicle. In some embodiments, the format of the image data can include, but is not limited to, the Joint Photographic Experts Group (JPEG) image format, the Tagged Image File Format (TIFF) image format, the Graphics Interchange Format (GIF) image format, the Kodak Flash PiX (FPX) image format, the Digital Imaging and Communications in Medicine (DICOM) image format, etc. In some embodiments, the image data can be a two-dimensional (2D) image or a three-dimensional (3D) image.

[0059] The camera of the vehicle can be a camera mounted on the vehicle. For example, an around-view multi-camera, an infrared camera, etc. mounted on the vehicle. In some embodiments, the first acquisition module 210 can acquire the image data at the current moment collected by the camera of the vehicle in real time through a network.

[0060] Step 420, convert the image data into bird's-eye view image data.

[0061] The bird's-eye view image data can be image data from a bird's-eye view perspective. In some embodiments, the image data can include multiple image features. The image features can be features characterizing the position, size, shape, and state of the target object.

[0062] Specifically, the first acquisition module 210 can use an image backbone network to extract first image features from the image data. The first image features can be image features from the perspective of the camera's perspective view, including the two-dimensional plane information of the target object. The image backbone network is a network for extracting features from images. In some embodiments, the image backbone network may include, but is not limited to, one or a combination of multiple types such as a Convolutional Neural Network (CNN), a Transformer model, and lightweight network models (e.g., DenseNet, MobileNet, etc.).

[0063] Furthermore, the first acquisition module 210 can convert the first image features into second image features through perspective transformation to obtain bird's-eye view image data. The second image features can be image features from the bird's-eye view perspective, including the two-dimensional plane information and height information of the target object. In some embodiments, the first acquisition module 210 can determine the corresponding height estimate based on the first image features from multiple camera perspective views, and then convert the first image features into the corresponding second image features based on the height estimate corresponding to the first image features.

[0064] Step 430: Obtain the point cloud data at the current moment collected by the lidar of the vehicle.

[0065] The point cloud can be a set of points used to describe the three-dimensional space where the vehicle is located. The point cloud data at the current moment can be a set of points in the three-dimensional space representing the size and pose information of the target object at the current moment. The points in the point cloud of the target object can correspond to the voxels of the target object, including the position information and features (e.g., color features, etc.) of the corresponding voxels in the three-dimensional space.

[0066] Specifically, the first acquisition module 210 can obtain the coordinate position corresponding to the target object based on the image data. The coordinate position corresponding to the target object can be the coordinate position of the outer contour of the target object in the three-dimensional space, which can be represented by the coordinate positions of the vertices of the outer contour of the target object. For example, if the target object is a cuboid, the coordinate position corresponding to the target object can be represented by the coordinate positions of the 8 vertices of the cuboid. Only as an example, the position coordinates corresponding to the target object can be determined based on the center point coordinates of the target object, the size of the target object, the rotation angle of the center point of the target object relative to the vehicle (e.g., the center point of the vehicle), and the two-dimensional bounding box of the target object.

[0067] Further, the first acquisition module 210 may acquire the point cloud of the target object from the lidar point cloud according to the coordinate position corresponding to the target object. The lidar point cloud may be the point cloud of the three-dimensional space where the vehicle is located acquired by the lidar of the vehicle. In some embodiments, the first acquisition module 210 may screen out the point cloud related to the target object from the lidar point cloud corresponding to the image data based on the coordinate position of the target object. Only as an example, the first acquisition module 210 may input the coordinate position of the target object and the lidar point cloud into a point cloud detector, and the point cloud detector may acquire the point corresponding to the coordinate position from the lidar point cloud as the point cloud of the target object.

[0068] Step 440, convert the point cloud data into bird's-eye view point cloud data.

[0069] The bird's-eye view point cloud data may be the point cloud data from the bird's-eye view perspective. As can be seen from the foregoing, the point cloud data may include the point clouds of multiple target objects.

[0070] Specifically, the first acquisition module 210 may utilize a point cloud backbone network to extract first point cloud features from the point cloud data. The first point cloud features may be the point cloud features from the lidar perspective, including the three-dimensional information of the target object. The point cloud backbone network is a network for extracting features from the point cloud. In some embodiments, the point cloud backbone network may include, but is not limited to, one or a combination of multiple of the Point-to-Box network, Point Pillars network, Point Net network, etc.

[0071] Further, the first acquisition module 210 may convert the first point cloud features into second point cloud features through perspective transformation to obtain the bird's-eye view point cloud data. The second point cloud features may be the point cloud features from the bird's-eye view perspective, including the two-dimensional plane information and height information of the target object. In some embodiments, the first acquisition module 210 may determine the corresponding two-dimensional plane information and height estimation information based on the three-dimensional information of the target object in the first point cloud features, and then convert the first point cloud features into the corresponding second point cloud features based on the two-dimensional plane information and height estimation information corresponding to the first point cloud features.

[0072] Step 450, determine the environmental data according to the bird's-eye view image data and the bird's-eye view point cloud data.

[0073] In some embodiments, the first acquisition module 210 may fuse the bird's-eye view image data and the bird's-eye view point cloud data to obtain the environmental data. Only as an example, the first acquisition module 210 may splice multiple second image features and multiple second point cloud features on the same feature channel respectively to obtain multiple spliced features, thereby determining the environmental data.

[0074] In some embodiments of the present specification, the image data and the point cloud data are converted to a bird's-eye view perspective and then stitched together to obtain environmental data, so that more accurate environmental data in the bird's-eye view perspective can be obtained based on data from multiple perspectives collected by different sensors.

[0075] Figure 5 is an exemplary flowchart for determining the initial information of the passable area corresponding to the current moment shown in some embodiments of the present specification. In some embodiments, Figure 5 it can be executed by the passable area determination system 100 (e.g., a processing device) or the passable area determination system 200 (e.g., the first determination module 230). For example, Figure 5 it can be stored in a storage device in the form of a program or instructions. When the passable area determination system 100 (e.g., a processing device) or the passable area determination system 200 (e.g., the first determination module 230) executes the instructions, the process 500 can be implemented. The operation schematic diagram of the process 500 presented below is illustrative. In some embodiments, the process can be completed by using one or more additional operations not described and / or one or more operations not discussed. Additionally, Figure 5 the order of the operations of the process 500 shown and described below is not restrictive.

[0076] Step 510, based on the passable area determination model, convert the passable area reference information into a prior query embedding.

[0077] The prior query embedding can be an embedding encoding of the passable area reference information. The prior query embedding can characterize the boundary information of the map drivable area determined based on the high-precision map.

[0078] In some embodiments, the first determination module 230 can input multiple polyline segments of the boundary line in the passable area reference information into the polyline segment encoder, and the polyline segment encoder outputs the encodings corresponding to the multiple polyline segments (i.e., the prior query embedding).

[0079] Step 520, input the initial query embedding into the passable area determination model.

[0080] The initial query embedding can be an embedding encoding of the random initial information of the passable area. In some embodiments, the first determination module 230 can randomly generate a corresponding initial query embedding based on the prior query embedding. For example, randomly modify the elements of the prior query embedding to 0 or 1.

[0081] Step 530, using the passable area determination model, determine the updated query embedding corresponding to the current moment according to the initial query embedding, the prior query embedding, and the environmental data.

[0082] The updated query embedding can be an encoding that fuses the initial query embedding, the prior query embedding, and the environmental data.

[0083] In some embodiments, the first determination module 230 can interactively encode the initial query embedding, the prior query embedding, and the environmental data using the encoding part of the traversable area determination model to obtain the updated query embedding corresponding to the current moment. Specifically, the encoding part of the traversable area determination model can first determine the first attention corresponding to the concatenation features of the initial query embedding, the prior query embedding, and the environmental data respectively; then, based on the first attention corresponding to each initial query embedding, each prior query embedding, and each concatenation feature, perform a first encoding on them to determine the second attention and the first interaction encoding between each pair of the initial query embeddings, the prior query embeddings, and the concatenation features; then further perform a second encoding based on the second attention corresponding to each initial query embedding, each prior query embedding, and each concatenation feature to determine the third attention and the second interaction encoding between each pair of the initial query embeddings, the prior query embeddings, and the concatenation features;...; until obtaining the Nth interaction encoding corresponding to the last encoding (the Nth encoding); finally, the fully connected layer of the traversable area determination model obtains the updated query embedding based on the interaction encoding corresponding to at least one encoding.

[0084] In some embodiments, the first determination module 230 can further determine the updated query embedding corresponding to the current moment based on the updated query embedding corresponding to the historical time. For a detailed description of determining the updated query embedding corresponding to the current moment based on the updated query embedding corresponding to the historical time, reference can be made to Figure 6 and its related descriptions, which will not be elaborated here.

[0085] Step 540, based on the traversable area determination model, decode the updated query embedding corresponding to the current moment to obtain the initial information of the traversable area corresponding to the current moment.

[0086] Specifically, the decoding part (e.g., multi-layer perceptron) of the traversable area determination model can decode the updated query embedding corresponding to the current moment to obtain the initial information of the traversable area corresponding to the current moment.

[0087] In some embodiments of this specification, the traversable area determination model performs interactive encoding on the reference information, the initial query embedding, the prior query embedding, and the environmental data to obtain an updated query embedding that fully integrates high-precision map information, so that the obtained initial information of the traversable area has a target detection result with a higher recall rate.

[0088] Figure 6 is an exemplary flowchart for determining the updated query embedding shown in some embodiments of this specification. In some embodiments, Figure 6It can be executed by the passable area determination system 100 (e.g., a processing device) or the passable area determination system 200 (e.g., the first determination module 230). For example, Figure 6 It can be stored in a storage device in the form of a program or instructions. When the passable area determination system 100 (e.g., a processing device) or the passable area determination system 200 (e.g., the first determination module 230) executes the instructions, the process 600 can be implemented. The operation schematic diagram of the process 600 presented below is illustrative. In some embodiments, the process can be completed by using one or more additional operations not described and / or one or more operations not discussed. Additionally, Figure 6 the order of the operations of the process 600 shown and described below is not restrictive.

[0089] Step 610: Based on the passable area determination model, convert the initial information of the passable area corresponding to the historical time to the vehicle coordinate system corresponding to the current moment to obtain a position embedding.

[0090] During the passing process, the vehicle will collect multiple frames of environmental data in real time. For each frame of environmental data collected, the process 300 will be executed to determine the initial information of the passable area corresponding to the current moment (or the current frame of environmental data). The historical time can be the previous one or more moments within a period of time before the current moment.

[0091] The initial information of the passable area corresponding to the historical time can be the initial information of the passable area determined based on the previous frame or several previous frames of environmental data of the current frame, including lane lines within a preset range of the position where the vehicle is at the historical time, boundary lines of the initial passable area, etc. The position embedding can be an embedding code representing the initial information of the passable area corresponding to the historical time.

[0092] Specifically, the first determination module 230 can convert the coordinate positions of the lane lines and the broken line segments of the boundary lines of the initial passable area corresponding to the historical time to the vehicle coordinate system corresponding to the current moment based on the displacement relationship of the vehicle from the historical time to the current moment to obtain the historical information of the passable area. Further, the first determination module 230 can encode the historical information of the passable area by using the encoding part of the passable area determination model to obtain the corresponding position embedding.

[0093] Step 620: Based on the passable area determination model, generate a tracking query embedding according to the position embedding and the updated query embedding corresponding to the historical time.

[0094] The tracking query embedding can be an embedding code that combines the initial information of the passable area corresponding to the historical time and the updated query embedding corresponding to the historical time. In some embodiments, the first determination module 230 can use the passable area determination model to splice the position embedding and the updated query embedding corresponding to the historical time on the feature channel to generate the tracking query embedding.

[0095] Step 630: Based on the passable area determination model, determine the updated query embedding corresponding to the current moment according to the tracking query embedding.

[0096] In some embodiments, the first determination module 230 can use the encoding part of the passable area determination model to perform interactive encoding on the tracking query embedding, the initial query embedding corresponding to the current moment, the prior query embedding corresponding to the current moment, and the environmental data corresponding to the current moment to obtain the updated query embedding corresponding to the current moment. For a detailed description of performing exchange encoding using the encoding part of the passable area determination model, refer to step 530, which will not be elaborated here.

[0097] In some embodiments of this specification, the recursive use of the initial information of the passable area corresponding to the historical time and the updated query embedding corresponding to the historical time can make the updated query embedding have better inter-frame consistency and stability, thereby improving the robustness of the initial information of the passable area determined based on the updated query embedding in occlusion scenarios.

[0098] Figure 7 It is an exemplary flowchart for determining the updated query embedding shown according to some embodiments of this specification. In some embodiments, Figure 7 It can be executed by the passable area determination system 100 (for example, a processing device) or the passable area determination system 200 (for example, the first determination module 230). For example, Figure 7 It can be stored in a storage device in the form of a program or instruction. When the passable area determination system 100 (for example, a processing device) or the passable area determination system 200 (for example, the first determination module 230) executes the instruction, the process 700 can be implemented. The operation schematic diagram of the process 700 presented below is illustrative. In some embodiments, one or more additional operations not described and / or one or more operations not discussed can be used to complete the process. Additionally, Figure 7 The order of the operations of the process 700 shown and described below is not restrictive.

[0099] Step 710: Obtain the initial information of the passable area corresponding to the historical time.

[0100] As described above, the initial information of the passable area corresponding to the historical time can be the initial information of the passable area determined based on the environmental data of the previous frame or several previous frames of the current frame, including lane lines within a preset range of the position where the vehicle is located at the historical time, boundary lines of the initial passable area, etc.

[0101] In some embodiments, the first determination module 230 may store the initial information of the passable area corresponding to the current moment in the storage device 130, and obtain the initial information of the passable area corresponding to the historical time from the storage device 130.

[0102] Step 720, convert the initial information of the passable area corresponding to the historical time into the vehicle coordinate system corresponding to the current moment to obtain the historical information of the passable area.

[0103] Specifically, the first determination module 230 may, based on the displacement relationship of the vehicle from the historical time to the current moment, convert the coordinate positions of the lane lines corresponding to the historical time and the broken line segments of the boundary lines of the initial passable area into the vehicle coordinate system corresponding to the current moment to obtain the historical information of the passable area.

[0104] Step 730, input the historical information of the passable area into the passable area determination model to determine the initial information of the passable area corresponding to the current moment.

[0105] In some embodiments, the first determination module 230 may encode the historical information of the passable area using the encoding part of the passable area determination model to obtain a historical information embedding, and then decode the historical information embedding using the decoding part of the passable area determination model to obtain the initial information of the passable area corresponding to the current moment.

[0106] In some embodiments, the first determination module 230 may also input other information (for example, reference information) into the passable area determination model, and the passable area determination model may perform interactive encoding on the historical information of the passable area and other information to obtain a historical information embedding.

[0107] Figure 8 It is an exemplary flowchart of the passable area determination method shown in some embodiments of this specification. In some embodiments, process 800 may be executed by the passable area determination system 100 (for example, a processing device) or the passable area determination system 200. For example, process 800 may be stored in the storage device in the form of a program or instructions, and when the passable area determination system 100 (for example, a processing device) or the passable area determination system 200 executes the instructions, process 800 may be implemented. The operation schematic diagram of process 800 presented below is illustrative. In some embodiments, the process may be completed using one or more additional operations not described and / or one or more operations not discussed. Additionally, Figure 8The order of operations of the process 800 shown and described below is not restrictive.

[0108] Step 810, obtain multiple initial information of passable regions output by multiple models. Specifically, step 810 can be executed by the third acquisition module 240.

[0109] The multiple models can be multiple different models. For example, the multiple models can be models trained based on different model structures and / or different training samples.

[0110] In some embodiments, the multiple models can at least include a passable region determination model. For example, the multiple perception models can include M1, M2, and M3, where M1, M2, and M3 can be a self-attention model, a Transformer model, and a BERT model respectively. For a detailed description of using the passable region determination model to output the initial information of the passable region, reference can be made to Figures 3 to 7 and its related description, which will not be elaborated here.

[0111] In some embodiments, the multiple models can also include a trained 3D detection model, a 3D segmentation model, a BEV occupancy space model, a BEV online map model, etc. Correspondingly, in some embodiments, the initial information of the passable region can also include a 3D target detection box of the target object, the semantics and / or segmentation results of the point cloud, the occupancy grid and obstacles from the BEV perspective, the vectorized elements from the BEV perspective, etc.

[0112] Merely as an example, the third acquisition module 240 can use the target detection model to obtain the 3D detection box of the target object from the image data. As another example, the third acquisition module 240 can use the classification model to obtain the corresponding semantic segmentation result of the point cloud based on the point cloud of the target object. As another example, the third acquisition module 240 can use automated algorithms and / or manual annotation and other methods to extract annotations from the online map to obtain road elements.

[0113] Step 820, align the multiple initial information of passable regions in time and space to obtain multiple aligned information of passable regions. Specifically, step 820 can be executed by the information alignment module 250.

[0114] The multiple aligned information of passable regions can be information of multiple passable regions aligned in time and space. It can be understood that the frame rates and time points of the multiple models may be different, resulting in different times of the output initial information of the passable regions by the multiple models, so that the multiple initial information of the passable regions can correspond to different moments and / or different positions. Therefore, it is necessary to align the multiple initial information of the passable regions in time and space. For a detailed description of obtaining the multiple aligned information of passable regions, reference can be made to Figure 9And its related descriptions will not be elaborated here.

[0115] Step 830: Determine the target information of the passable area corresponding to the current moment according to the alignment information of multiple passable areas. Specifically, step 830 can be executed by the second determination module 260.

[0116] The target information of the passable area can be the relevant information of the target passable area determined based on the initial information of multiple passable areas. In some embodiments, the target information of the passable area corresponding to the current moment may include lane lines within a preset range of the current position of the vehicle (for example, 10 meters, 20 meters, 50 meters, etc.), boundary lines of the target passable area, etc.

[0117] For a detailed description of determining the target information of the passable area corresponding to the current moment according to the alignment information of multiple passable areas, reference can be made to Figure 10 And its related descriptions will not be elaborated here.

[0118] In some embodiments of this specification, the initial information of multiple passable areas output by multiple models is fully utilized and fused into a unified and stable alignment information, which improves the credibility of the target information of the passable area obtained based on the alignment information.

[0119] Figure 9 is a flowchart of determining the alignment information of multiple passable areas according to some embodiments of this specification. In some embodiments, Figure 9 it can be executed by the passable area determination system 100 (for example, a processing device) or the passable area determination system 200 (for example, the information alignment module 250). For example, Figure 9 it can be stored in a storage device in the form of a program or instruction. When the passable area determination system 100 (for example, a processing device) or the passable area determination system 200 (for example, the information alignment module 250) executes the instruction, the process 900 can be implemented. The operation schematic diagram of the process 900 presented below is illustrative. In some embodiments, one or more additional operations not described and / or one or more operations not discussed can be used to complete this process. Additionally, Figure 9 the order of the operations of the process 900 shown and described below is not restrictive.

[0120] Step 910: Convert the initial information of multiple passable areas corresponding to multiple models to the current moment according to the position relationship of the vehicle corresponding to the current moment, and obtain the time alignment information of multiple models corresponding to multiple models.

[0121] The positional relationship corresponding to the vehicle at the current moment can be the relationship between the current position corresponding to the vehicle at the current moment and the historical position corresponding to the vehicle at the historical moment corresponding to the initial information of multiple passable regions. Merely as an example, the positional relationship corresponding to the vehicle at the current moment can be represented by the displacement from the current position corresponding to the vehicle at the current moment to the historical position corresponding to the vehicle at the historical moment. For example, if the coordinates of the current position corresponding to the vehicle at the current moment are (x0, y0, z0), and the historical position corresponding to the vehicle at the historical moment corresponding to model M1 is (x1, y1, z1), then the positional relationship corresponding to the vehicle at the current moment can include (δx1, δy1, δz1) = (x0 - x1, y0 - y1, z0 - z1).

[0122] The multiple time alignment information corresponding to multiple models can be multiple passable region information aligned in time. In some embodiments, the multiple time alignment information corresponding to multiple models can include the alignment positions of boundary lines, target objects, road elements, etc. after being aligned in time. In some embodiments, the multiple time alignment information corresponding to multiple models can be aligned based on the current moment.

[0123] Specifically, the information alignment module 250 can map the initial information of multiple passable regions corresponding to multiple models into the vehicle coordinate system corresponding to the current moment based on the positional relationship corresponding to the vehicle at the current moment, to obtain the multiple time alignment information corresponding to multiple models. For example, the initial information of the passable region output by model M1 at the historical moment includes the position coordinates (P1x, P1y, P1z) of the turning point P1 of the polyline of the boundary line of the initial passable region. The information alignment module 250 can obtain the time alignment information corresponding to M1 based on the positional relationship (δx1, δy1, δz1) corresponding to the vehicle at the current moment: the alignment position (P1x’, P1y’, P1z’) of the turning point P1 = (P1x, P1y, P1z) + (δx1, δy1, δz1). Another example, for the position coordinates (P2x, P2y, P2z) of the 3D detection box P2 of the target object "cone sign" output by model M4, the information alignment module 250 can obtain the time alignment information corresponding to M4 based on the positional relationship (δx4, δy4, δz4) corresponding to the vehicle at the current moment: the alignment position (P2x’, P2y’, P2z’) of the 3D detection box P2 of the "cone sign" = (P2x, P2y, P2z) + (δx4, δy4, δz4).

[0124] Step 920: Cluster the multiple time alignment information to obtain one or more obstacle regions.

[0125] An obstacle region can be a region where the vehicle cannot pass freely. In some embodiments, the information alignment module 250 can cluster the multiple time alignment information based on spatial positions to obtain one or more obstacle regions.

[0126] Specifically, the information alignment module 250 may cluster multiple time alignment information with adjacent alignment positions based on the alignment positions corresponding to the multiple time alignment information, obtain at least one set of time alignment information, determine the central position of the obstacle area as the central position of the multiple alignment positions corresponding to the multiple time alignment information in each set of time alignment information, and determine the corresponding obstacle area based on the central position of the obstacle area and the distribution of the multiple alignment position coordinates. For example, Figure 11 is an exemplary schematic diagram of the obstacle area shown according to some embodiments of the present specification, such as Figure 11 shown, the set of time alignment information may include the first set of time alignment information. The central position O1 of the obstacle area I is determined based on the central position of the multiple alignment positions (represented by black squares in the figure) in the first set of time alignment information, and the obstacle area I is determined based on O1 and the multiple alignment positions in the first set of time alignment information. Similarly, the obstacle area II can be determined based on the second set of time alignment information.

[0127] In some embodiments, the clustering may include, but is not limited to, a combination of one or more of the following algorithms: K-MEANS clustering algorithm, mean shift clustering algorithm, DBSCAN (Density-Based Spatial Clustering of Applications with Noise) clustering algorithm, etc.

[0128] Step 930, determine the bird's-eye view area according to the current position of the vehicle.

[0129] The bird's-eye view area may be an area for determining the target information of the passable area to be determined. In some embodiments, the information alignment module 250 may determine the area within a preset range of the current position of the vehicle as the bird's-eye view area. For example, the space 30m to the left and right and 60m to the front and rear of the current position of the vehicle is determined as the bird's-eye view area. In some embodiments, the bird's-eye view area may include multiple grids. For example, grids with a side length of 10 cm.

[0130] Step 940, project one or more obstacle areas onto the bird's-eye view area to obtain multiple passable area alignment information.

[0131] Specifically, the information alignment module 250 may project one or more obstacle areas onto the bird's-eye view area based on the corresponding relationship between the obstacle area position and the bird's-eye view area position, and obtain the space occupied by the obstacle area in the bird's-eye view area, that is, multiple passable area alignment information.

[0132] In some embodiments, each obstacle area may correspond to one or more grids. For example, as Figure 11As shown, the obstacle area I and the obstacle area II can respectively correspond to multiple grids.

[0133] In some embodiments of the present specification, converting the initial information of the passable areas of multiple models to the same expression space with unified time can eliminate the fusion difficulty caused by different accuracies, different processing times, and different output data formats of multiple models, thereby improving the unity and stability of the aligned information.

[0134] Figure 10 is an exemplary flowchart of the passable area determination method shown in some embodiments of the present specification. In some embodiments, the process 1000 can be executed by the passable area determination system 100 (e.g., a processing device) or the passable area determination system 200 (e.g., the second determination module 260). For example, the process 1000 can be stored in a storage device in the form of a program or an instruction. When the passable area determination system 100 (e.g., a processing device) or the passable area determination system 200 (e.g., the second determination module 260) executes the instruction, the process 1000 can be implemented. The operation schematic diagram of the process 1000 presented below is illustrative. In some embodiments, the process can be completed by using one or more additional operations not described and / or one or more operations not discussed. Additionally, Figure 10 The order of the operations of the process 1000 shown and described below is not restrictive.

[0135] Step 1010, for each grid corresponding to an obstacle area, determine a jitter weight according to at least one of the confidence level, height information, occupancy type, and model source of the obstacle area corresponding to the grid.

[0136] The confidence level of the obstacle area can characterize the reliability of the prediction of the obstacle area by multiple models. Among them, the greater the confidence level of the obstacle area, the higher the reliability. In some embodiments, the second determination module 260 can determine the confidence level of the obstacle area based on the confidence levels of the initial information of the passable areas corresponding to the current moment output by multiple models.

[0137] The confidence level of the initial information of the passable area corresponding to the current moment output by each model reflects the stability of the output result of each model over time. Among them, the greater the confidence level of the initial information of the passable area corresponding to the current moment, the higher the stability of the corresponding model output result. In some embodiments, for each model, the confidence level of the initial information of the passable area corresponding to the current moment can be determined based on the initial information of the passable area corresponding to the current moment and the initial information of the passable area corresponding to the historical moment. Specifically, the confidence level of the initial information of the passable area corresponding to the current moment can be determined based on the difference between the initial information of the passable area corresponding to the current moment and the initial information of the passable area corresponding to the historical moment. Among them, the smaller the difference, the greater the corresponding confidence level. In some embodiments, the second determination module 260 can first determine the differences between the initial information at the current moment and the initial information corresponding to each historical moment, and then calculate the average value of the multiple differences to obtain the difference between the initial information of the passable area corresponding to the current moment and the initial information of the passable area corresponding to the historical moment. In some embodiments, the second determination module 260 can calculate the sub-differences between the initial passable areas in terms of position, shape, contour, area, etc. in the initial information of the passable area corresponding to the current moment and the initial information of the passable area corresponding to the historical moment respectively, and then perform operations such as weighted summation and averaging on the multiple sub-differences to obtain the difference between the initial information of the passable area corresponding to the current moment and the initial information of the passable area corresponding to the historical moment.

[0138] In some embodiments, the second determination module 260 can determine multiple time alignment information in each obstacle area, and then determine the corresponding multiple models according to the multiple time alignment information, and determine the confidence level of the corresponding obstacle area based on the confidence levels of the initial information of the passable area corresponding to the current moment corresponding to the multiple models. In some embodiments, the second determination module 260 can perform operations such as summing and averaging on the confidence levels of the initial information of the passable area corresponding to the current moment corresponding to the multiple models to determine the confidence level of the corresponding obstacle area. For example, as Figure 11 shown, the second determination module 260 can determine 6 time alignment information included in the obstacle area I, and then determine the corresponding models (for example, M1, M1, M4, M2, M4, M1) according to the 6 time alignment information, and determine the confidence level BI of the obstacle area I based on the confidence levels B1, B2, and B4 corresponding to M1, M2, and M4, where BI = (3B1 + B2 + B4) / 6.

[0139] The height information can characterize the height features of the obstacle area. In some embodiments, the height information may include the minimum value, maximum value, average value, and / or variance of one or more heights corresponding to one or more grids of each obstacle area. For example, the height information of obstacle area I may include the minimum value of 1 m and the maximum value of 3 m among the 6 heights corresponding to 6 grids.

[0140] The occupancy type can characterize the obstacle category within the obstacle area. For example, the occupancy type may include "cone barrel" (which can be represented by 1), "pedestrian" (2), "other vehicle" (which can be represented by 3), etc. For example, the occupancy type of obstacle area I may include the types 1, 1, 2, 1, 2, 3 corresponding to 6 obstacles.

[0141] The model source can characterize the source of multiple models corresponding to each obstacle area. In some embodiments, the model source may include a passable area determination model, a perception model, an online map model, etc. For example, the model source of obstacle area I may include the passable area determination models corresponding to M1 and M2 (which can be represented by 1), and the perception model corresponding to M4 (which can be represented by 2).

[0142] The status information of the obstacle area can reflect the static features of the obstacle area. In some embodiments, the second determination module 260 may determine the status information of the obstacle area according to at least one of the confidence level, height information, occupancy type, and model source of the obstacle area corresponding to the grid. For example, the status information of obstacle area I may be SI = [BI; 1, 3; 1, 2; 1, 1, 0, 1, 0.5, 0].

[0143] The jitter weight can represent the stability of the occupied grid. The occupied grid can be a grid occupied by an obstacle. In some embodiments, the second determination module 260 may determine the occupied grid in the obstacle area based on the alignment information. The smaller the jitter weight of the occupied grid, the higher the stability.

[0144] In some embodiments, the second determination module 260 may determine the jitter weight of the occupied grid based on the status information of the obstacle area corresponding to the occupied grid.

[0145] In some embodiments, the second determination module 260 may determine the jitter weight of the occupied grid based on the average value of the confidence levels of multiple models corresponding to the occupied grid (for example, arithmetic mean, weighted mean, etc.). Among them, the larger the average value, the smaller the jitter weight of the occupied grid.

[0146] In some embodiments, the second determination module 260 may determine the jitter weight of the occupied grid based on the height information of the obstacle area corresponding to the occupied grid. Among them, the closer the height information is to the preset range, the more credible the detection result is, and the smaller the corresponding jitter weight is; on the contrary, it indicates that the detection result is not credible, and the corresponding jitter weight is larger.

[0147] In some embodiments, the second determination module 260 may determine the jitter weight of the occupied grid based on the occupancy models output by multiple models corresponding to the obstacle area corresponding to the occupied grid. Among them, the more unified the occupancy models output by the multiple models are, the smaller the jitter weight of the occupied grid is; on the contrary, the jitter weight of the occupied grid is larger.

[0148] In some embodiments, the second determination module 260 may determine the jitter weight of the occupied grid based on the model source of the obstacle area corresponding to the occupied grid. Among them, the more model sources there are, the higher the credibility that the models from different sources all believe that there is an obstacle in the occupied grid, and the smaller the corresponding jitter weight of the occupied grid is; on the contrary, the corresponding jitter weight of the occupied grid is larger.

[0149] Step 1020, optimize one or more obstacle areas based on the jitter weight.

[0150] In some embodiments, the second determination module 260 may determine the grid corresponding to the jitter weight greater than the weight threshold as a candidate grid. The candidate grid may be a grid whose possibility of being occupied by an obstacle is considered unstable. In some embodiments, the second determination module 260 may reconstruct the candidate grid.

[0151] Specifically, the second determination module 260 may use a Kalman filter to smooth the state information of the candidate grid to obtain the corresponding smoothed state information. For example, the smoothed state information of candidate grid a may be a'. The smoothed state information includes at least one of the occupancy probability (the probability of being occupied by an obstacle), occupancy category, minimum height, maximum height, etc. of the candidate grid. Kalman filtering is a known technology and will not be elaborated here. Further, the second determination module 260 may optimize the multiple candidate grids after the obstacle area reconstruction based on the smoothed state information to obtain the optimized obstacle area. For example, in the smoothed state information of the candidate grid, the occupancy probability is 1%, indicating that the probability of this candidate grid being occupied by an obstacle is small, and this candidate grid may be deleted from the obstacle area.

[0152] Step 1030, determine the passable area target information corresponding to the current moment based on the optimized obstacle area.

[0153] In some embodiments, the second determination module 260 may determine the area outside the optimized obstacle area in the bird's-eye view area as the passable area corresponding to the current moment, and obtain the target information of the passable area corresponding to the current moment.

[0154] In some embodiments of the present specification, based on the confidence level and jitter weight of the obstacle area, the obstacle area is optimized, thereby reducing the inter-frame jump of the model output result and improving the robustness of the target information of the passable area.

[0155] It should be noted that the above description is provided for illustrative purposes only and is not intended to limit the scope of the present specification. Those of ordinary skill in the art can make various changes and modifications according to the description of the present specification. However, these changes and modifications do not depart from the scope of the present specification. In some embodiments, the above process may include one or more additional operations, or one or more of the above operations may be omitted.

[0156] The beneficial effects that the embodiments of the present specification may bring include, but are not limited to: (1) determining the initial information of the passable area corresponding to the current moment based on the environmental data and the reference information of the passable area. By fusing the reference information of the high-precision map, the initial information of the passable area can provide more stable real-time detection results of lane lines and drivable area boundaries; (2) stitching the image data and the point cloud data after converting them to the bird's-eye view perspective to obtain environmental data, so that more accurate environmental data in the bird's-eye view perspective can be obtained based on the data from multiple perspectives collected by different sensors; (3) performing interactive coding on the reference information, the initial query embedding, the prior query embedding, and the environmental data based on the passable area determination model to obtain an updated query embedding that fully integrates the high-precision map information, so that the obtained initial information of the passable area can fully integrate the offline high-precision map information and the real-time environmental data information, thereby having a target detection result with a higher recall rate; (4) the recursive use of the initial information of the passable area corresponding to the historical time and the updated query embedding corresponding to the historical time can make the updated query embedding have better inter-frame consistency and stability, thereby improving the robustness of the initial information of the passable area determined based on the updated query embedding in the occlusion scenario; (5) based on the confidence level and jitter weight of the obstacle area, the obstacle area is optimized, thereby reducing the inter-frame jump of the model output result and improving the robustness of the target information of the passable area; (6) converting the initial information of the passable areas of multiple models to the same expression space with unified time can eliminate the fusion difficulty caused by different accuracies, different processing times, and different output data formats of multiple models, thereby improving the unity and stability of the aligned information; (7) based on the confidence level and jitter weight of the obstacle area, the obstacle area is optimized, thereby reducing the inter-frame jump of the model output result and improving the robustness of the target information of the passable area.

[0157] It should be noted that different embodiments may produce different beneficial effects. In different embodiments, the possible beneficial effects may be any one or several combinations of the above, or any other possible beneficial effects that can be obtained.

[0158] The basic concepts have been described above. Obviously, for those skilled in the art, the above detailed disclosure is only an example and does not constitute a limitation to this specification. Although not explicitly stated here, those skilled in the art may make various modifications, improvements, and corrections to this specification. Such modifications, improvements, and corrections are proposed in this specification, so such modifications, improvements, and corrections still fall within the spirit and scope of the exemplary embodiments of this specification.

[0159] At the same time, this specification uses specific terms to describe the embodiments of this specification. Such as "one embodiment", "an embodiment", and / or "some embodiments" mean a certain feature, structure, or characteristic related to at least one embodiment of this specification. Therefore, it should be emphasized and noted that the "one embodiment" or "an embodiment" or "an alternative embodiment" mentioned twice or more at different positions in this specification does not necessarily refer to the same embodiment. In addition, certain features, structures, or characteristics in one or more embodiments of this specification can be appropriately combined.

[0160] In addition, unless clearly stated in the claims, the order of the processing elements and sequences, the use of numerical letters, or the use of other names in this specification are not used to limit the order of the processes and methods in this specification. Although some currently considered useful embodiments of the invention are discussed through various examples in the above disclosure, it should be understood that such details only serve the purpose of illustration. The appended claims are not limited to the disclosed embodiments. On the contrary, the claims are intended to cover all modifications and equivalent combinations that conform to the essence and scope of the embodiments of this specification. For example, although the system components described above can be implemented by hardware devices, they can also be implemented only through software solutions, such as installing the described system on existing servers or mobile devices.

[0161] Similarly, it should be noted that in order to simplify the expression of the disclosure of this specification and thus help the understanding of one or more embodiments of the invention, in the description of the embodiments of this specification above, sometimes multiple features are merged into one embodiment, drawing, or description thereof. However, this disclosure method does not mean that the features required by the object of this specification are more than those mentioned in the claims. In fact, the features of the embodiments are less than all the features of the individual embodiments disclosed above.

[0162] In some embodiments, numbers are used to describe components and the quantity of attributes. It should be understood that such numbers used in the description of embodiments are, in some examples, modified by the modifiers "about", "approximate" or "substantially". Unless otherwise specified, "about", "approximate" or "substantially" indicate that the said numbers allow a variation of ±20%. Accordingly, in some embodiments, the numerical parameters used in the specification and claims are approximate values, which may vary according to the characteristics required by individual embodiments. In some embodiments, the numerical parameters should consider the specified significant digits and adopt the method of retaining the general number of digits. Although the numerical ranges and parameters used in some embodiments of this specification to confirm the breadth of their scope are approximate values, in specific embodiments, such numerical settings are as precise as possible within the feasible range.

[0163] For each patent, patent application, patent application publication and other materials cited in this specification, such as articles, books, specifications, publications, documents, etc., their entire contents are hereby incorporated into this specification by reference. Except for the application history documents that are inconsistent with or conflict with the content of this specification, and also except for the documents that limit the broadest scope of the claims of this specification (currently or subsequently appended to this specification). It should be noted that if there are inconsistencies or conflicts between the descriptions, definitions, and / or uses of terms in the supplementary materials of this specification and the content described in this specification, the descriptions, definitions, and / or uses of terms in this specification shall prevail.

[0164] Finally, it should be understood that the embodiments described in this specification are only used to illustrate the principles of the embodiments of this specification. Other variations may also fall within the scope of this specification. Therefore, by way of example and not limitation, alternative configurations of the embodiments of this specification may be considered to be consistent with the teachings of this specification. Accordingly, the embodiments of this specification are not limited to the embodiments explicitly introduced and described in this specification.

Claims

1. A passable area determination method, executed by a vehicle, includes: Obtaining environmental data at the current moment collected by sensors of the vehicle; Obtaining passable area reference information corresponding to the current position of the vehicle according to a high-precision map; Inputting the environmental data and the passable area reference information into a passable area determination model to determine initial passable area information corresponding to the current moment.

2. The method according to claim 1, characterized in that, The obtaining of the environmental data at the current moment collected by sensors of the vehicle includes: Obtaining image data at the current moment collected by a camera of the vehicle; Converting the image data into bird's-eye view image data; Obtaining point cloud data at the current moment collected by a lidar of the vehicle; Converting the point cloud data into bird's-eye view point cloud data; Determining the environmental data according to the bird's-eye view image data and the bird's-eye view point cloud data.

3. The method according to claim 1, wherein The method further includes: Obtaining initial passable area information corresponding to a historical time; Converting the initial passable area information corresponding to the historical time into the vehicle coordinate system corresponding to the current moment to obtain historical passable area information; Inputting the historical passable area information into the passable area determination model to determine the initial passable area information corresponding to the current moment.

4. The method according to claim 1, wherein The passable area determination model includes an attention mechanism model.

5. The method according to claim 4, wherein The inputting of the environmental data and the passable area reference information into the passable area determination model to determine the initial passable area information corresponding to the current moment includes: Based on the passable area determination model, converting the passable area reference information into a prior query embedding; Inputting an initial query embedding into the passable area determination model; Using the passable area determination model to determine an updated query embedding corresponding to the current moment according to the initial query embedding, the prior query embedding, and the environmental data; Decoding the updated query embedding corresponding to the current moment based on the passable area determination model to obtain the initial passable area information corresponding to the current moment.

6. The method according to claim 5, characterized in that The determining of the updated query embedding corresponding to the current moment further includes: Based on the passable area determination model, converting the initial passable area information corresponding to a historical time into the vehicle coordinate system corresponding to the current moment to obtain a position embedding; Based on the passable area determination model, generating a tracking query embedding according to the position embedding and the updated query embedding corresponding to the historical time; Based on the passable area determination model, determining the updated query embedding corresponding to the current moment according to the tracking query embedding.

7. A passable area determination method, executed by a vehicle, includes: Obtaining multiple initial passable area information output by multiple models; Aligning the multiple initial passable area information in time and space to obtain multiple passable area alignment information; Determining target passable area information corresponding to the current moment according to the multiple passable area alignment information.

8. The method according to claim 7, wherein The method further includes: For each of the models, based on the initial information of the passable area corresponding to the current moment and the initial information of the passable area corresponding to the historical moment, determine the confidence level of the initial information of the passable area corresponding to the current moment.

9. The method according to claim 7, wherein The aligning the multiple initial passable area information corresponding to the multiple models in time and space to obtain multiple aligned passable area information includes: According to the positional relationship of the vehicle corresponding to the current moment, convert the multiple initial passable area information corresponding to the multiple models to the current moment to obtain multiple time-aligned information corresponding to the multiple models; Cluster the multiple time-aligned information to obtain one or more obstacle areas; Determine an aerial view area according to the current position of the vehicle, and the aerial view area includes multiple grids; Project the one or more obstacle areas onto the aerial view area to obtain the multiple aligned passable area information, and each obstacle area corresponds to one or more of the grids.

10. The method according to claim 9, characterized in that, The determining the target information of the passable area corresponding to the current moment according to the multiple aligned passable area information includes: For each grid corresponding to the obstacle area, determine a jitter weight according to at least one of the confidence level, height information, occupancy type, and model source of the obstacle area corresponding to the grid; Optimize the one or more obstacle areas based on the jitter weight; Based on the optimized obstacle area, determine the target information of the passable area corresponding to the current moment.

Images