Obstacle response method and device, equipment, storage medium and computer program product

Abstract

The invention discloses an obstacle response method, device and equipment, a storage medium and a computer program product, and relates to the technical field of computers, and the method comprises the steps: carrying out the obstacle detection of a vehicle driving road of a current driving vehicle, and obtaining the obstacle information of a road obstacle in the vehicle driving road; judging whether the road obstacle meets a preset semantic reasoning triggering condition or not based on the obstacle information; if yes, semantic reasoning is carried out on the multi-modal environment data of the current driving vehicle, and an obstacle semantic tag corresponding to the road obstacle is determined; and responding to the road obstacle based on the obstacle semantic label. According to the invention, when the obstacle meets the semantic reasoning condition, the semantic label of the obstacle is determined through semantic reasoning, and the obstacle is responded according to the semantic label, so that the problem that the obstacle response method in the prior art can only realize geometric obstacle detection and cannot respond to the obstacle in the prior art is solved. And thus, the technical problem that accurate response to the obstacle cannot be realized is solved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to obstacle response methods, apparatus, devices, storage media, and computer program products. Background Technology

[0002] With the rapid development of intelligent driving technology, the autonomous decision-making and response capabilities of vehicles in complex road environments have become a key research focus. Among these, the detection and response to road obstacles is one of the key functions of intelligent driving systems, directly impacting driving safety and the driving experience.

[0003] Currently, traditional obstacle response methods mainly rely on sensors such as visual cameras, LiDAR, and millimeter-wave radar to collect multimodal data of the road environment, and combine convolutional networks or clustering algorithms to locate and coarsely classify obstacles, and then formulate obstacle responses based on the classification results. However, this method can only achieve obstacle detection at the geometric level, and the detection accuracy is not high. Therefore, it cannot generate optimal response strategies for obstacle response when facing complex and ever-changing real-world driving scenarios. Summary of the Invention

[0004] The main objective of this application is to provide an obstacle response method, apparatus, device, storage medium, and computer program product, aiming to solve the technical problem that existing obstacle response methods can only achieve obstacle detection at the geometric level, resulting in the inability to achieve accurate response to obstacles.

[0005] To achieve the above objectives, this application proposes an obstacle response method, the method comprising: Obstacle detection is performed on the road where the vehicle is currently traveling to obtain obstacle information of road obstacles in the road where the vehicle is traveling; Based on the obstacle information, determine whether the road obstacle meets the preset semantic reasoning trigger condition; If the conditions are met, semantic reasoning is performed on the multimodal environment data of the currently driving vehicle to determine the obstacle semantic label corresponding to the road obstacle. The obstacle semantic label is used to characterize the type and semantic information of the road obstacle. The system responds to the road obstacles based on the obstacle semantic tags.

[0006] In one embodiment, the step of detecting obstacles on the road where the currently traveling vehicle is traveling and obtaining obstacle information of road obstacles in the road includes: Collect multimodal environmental data of the currently driving vehicle, including vehicle environment images, lidar point cloud data, and radar target data; The multimodal environment data is input into a preset target detection model, which includes a spatiotemporal calibration submodule and a target detection network. The spatiotemporal calibration submodule performs spatiotemporal alignment processing on the vehicle environment image, the lidar point cloud data, and the radar target data to output multimodal fusion data. The target detection network performs obstacle detection on the road where the currently driving vehicle is traveling based on the multimodal fusion data, thereby obtaining obstacle information of road obstacles in the road where the vehicle is traveling.

[0007] In one embodiment, before the step of determining whether the road obstacle meets the preset semantic reasoning triggering condition based on the obstacle information, the method further includes: Based on the obstacle information, determine the location information corresponding to all bounding boxes of the road obstacles; Redundant bounding boxes are determined from each bounding box based on the location information; The redundant bounding boxes are filtered, and the obstacle information is updated after the filtering is completed.

[0008] In one embodiment, the step of determining whether the road obstacle meets the preset semantic reasoning triggering condition based on the obstacle information includes: Determine the bounding box confidence level of all bounding boxes corresponding to the road obstacles based on the updated obstacle information; The target bounding box is determined from each bounding box based on the bounding box confidence score, and the target bounding box is the bounding box with the highest confidence score among the bounding boxes. Determine whether the confidence level of the target bounding box exceeds a preset confidence threshold; If the condition is exceeded, the road obstacle is determined to meet the preset semantic reasoning trigger condition.

[0009] In one embodiment, the step of performing semantic reasoning on the multimodal environmental data of the currently driving vehicle to determine the obstacle semantic label corresponding to the road obstacle includes: The bounding box coordinates of the target bounding box are determined based on the obstacle information; The bounding box coordinates are mapped to the original high-resolution image coordinate system to obtain the high-resolution bounding box coordinates of the target bounding box; The region of interest in the multimodal environment data is determined based on the high-resolution bounding box coordinates. The region of interest is a local region in the multimodal environment data that is related to the road obstacle. Semantic reasoning is performed on the region of interest to determine the obstacle semantic labels corresponding to the road obstacles.

[0010] In one embodiment, the step of performing semantic reasoning on the region of interest to determine the obstacle semantic label corresponding to the road obstacle includes: The region of interest is input into a preset multimodal large model, which is a lightweight multimodal large model for vehicle end. Structured prompts are constructed based on the high-resolution RGB image, high-resolution thermal image, LiDAR point cloud data, and vehicle context information of the region of interest. The preset multimodal large model generates obstacle description text corresponding to the road obstacles based on the structured prompt words; The obstacle semantic label corresponding to the road obstacle is determined based on the obstacle description text.

[0011] In one embodiment, the step of responding to the road obstacle based on the obstacle semantic label includes: The semantic tags of the obstacles are parsed to generate obstacle response commands; The road obstacle is responded to based on the obstacle response command.

[0012] In one embodiment, the step of parsing the obstacle semantic tags and generating obstacle response instructions includes: The semantic tags of the obstacles are parsed to determine the risk level, obstacle type, and handling recommendations for the road obstacles; Generate a response strategy for the road obstacle based on the risk level, the obstacle type, and the proposed handling; Obstacle response commands are generated according to the response strategy.

[0013] In one embodiment, the obstacle response command includes a vehicle obstacle avoidance command; the step of responding to the road obstacle based on the obstacle response command includes: If the obstacle response command is a vehicle obstacle avoidance command, then the current vehicle is replanned to obtain the replanned driving path. The vehicle is controlled to travel along the replanned path in order to avoid road obstacles.

[0014] In one embodiment, the obstacle response command further includes an obstacle warning command; the step of responding to the road obstacle based on the obstacle response command includes: If the obstacle response command is an obstacle warning command, then the obstacle warning method corresponding to the road obstacle is determined; The obstacle warning method is used to warn of road obstacles.

[0015] Furthermore, to achieve the above objectives, this application also proposes an obstacle response device, the device comprising: The obstacle detection module is used to detect obstacles on the road where the vehicle is currently traveling and to obtain obstacle information of road obstacles in the road where the vehicle is traveling. The semantic reasoning triggering module is used to determine whether the road obstacle meets the preset semantic reasoning triggering conditions based on the obstacle information. The semantic reasoning module is used to perform semantic reasoning on the multimodal environment data of the currently driving vehicle if the conditions are met, and determine the obstacle semantic label corresponding to the road obstacle. The obstacle semantic label is used to characterize the type and semantic information of the road obstacle. An obstacle response module is used to respond to the road obstacles based on the obstacle semantic tags.

[0016] In one embodiment, the obstacle detection module is further configured to collect multimodal environmental data of the currently driving vehicle, the multimodal environmental data including vehicle environment images, lidar point cloud data and radar target data; The obstacle detection module is also used to input the multimodal environment data into a preset target detection model, wherein the preset target detection model is provided with a spatiotemporal calibration submodule and a target detection network; The obstacle detection module is also used to perform spatiotemporal alignment processing on the vehicle environment image, the lidar point cloud data and the radar target data through the spatiotemporal calibration submodule, and output multimodal fusion data; The obstacle detection module is further configured to perform obstacle detection on the vehicle's driving road based on the multimodal fusion data through the target detection network, and obtain obstacle information of road obstacles in the vehicle's driving road.

[0017] In one embodiment, the semantic reasoning triggering module is further configured to determine the bounding box confidence level corresponding to all bounding boxes of the road obstacle based on the updated obstacle information; The semantic reasoning triggering module is also used to determine the target bounding box from each bounding box based on the bounding box confidence, wherein the target bounding box is the bounding box with the highest confidence. The semantic reasoning triggering module is also used to determine whether the confidence level of the target bounding box exceeds a preset confidence threshold. The semantic reasoning triggering module is also used to determine that the road obstacle meets the preset semantic reasoning triggering conditions if the condition is exceeded.

[0018] In one embodiment, the semantic reasoning module is further configured to determine the bounding box coordinates of the road obstacle based on the obstacle information; The semantic reasoning module is also used to map the bounding box coordinates to the original high-definition image coordinate system to obtain high-resolution bounding box coordinates; The semantic reasoning module is further configured to determine the region of interest in the multimodal environment data based on the high-resolution bounding box coordinates, wherein the region of interest is a local region in the multimodal environment data that is related to the road obstacle; The semantic reasoning module is also used to perform semantic reasoning on the region of interest to determine the obstacle semantic label corresponding to the road obstacle.

[0019] In one embodiment, the semantic reasoning module is further configured to input the region of interest into a preset multimodal large model, wherein the preset multimodal large model is a vehicle-side lightweight multimodal large model; The semantic reasoning module is also used to construct structured prompt words based on the high-resolution RGB image, high-resolution thermal imaging image, lidar point cloud data and vehicle context information of the region of interest; The semantic reasoning module is also used to generate obstacle description text corresponding to the road obstacle based on the structured prompt words using the preset multimodal large model; The semantic reasoning module is also used to determine the obstacle semantic label corresponding to the road obstacle based on the obstacle description text.

[0020] In one embodiment, the obstacle response module is further configured to parse the obstacle semantic tags and generate obstacle response instructions; The obstacle response module is also used to respond to the road obstacle based on the obstacle response command.

[0021] In one embodiment, the obstacle response module is further configured to parse the semantic tags of the obstacles to determine the risk level, obstacle type, and handling recommendations of the road obstacles; The obstacle response module is also used to generate a response strategy corresponding to the road obstacle based on the risk level, the obstacle type, and the handling suggestion; The obstacle response module is also used to generate obstacle response instructions according to the response strategy.

[0022] In addition, to achieve the above objectives, this application also proposes an obstacle response device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the obstacle response method as described above.

[0023] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and which, when executed by a processor, implements the steps of the obstacle response method as described above.

[0024] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the obstacle response method as described above.

[0025] This application provides an obstacle response method. The method involves detecting obstacles on the road where the vehicle is currently traveling, obtaining obstacle information of road obstacles; determining whether the road obstacle meets a preset semantic reasoning trigger condition based on the obstacle information; if so, performing semantic reasoning on the multimodal environmental data of the vehicle to determine the obstacle semantic label corresponding to the road obstacle, whereby the obstacle semantic label represents the type of road obstacle; and responding to the road obstacle based on the obstacle semantic label. Compared to existing obstacle response methods, which have low obstacle detection accuracy and cannot generate optimal response strategies in complex scenarios, this invention can determine whether semantic reasoning is triggered based on obstacle information, and when semantic reasoning is triggered, determine the obstacle's semantic label through semantic reasoning, and finally respond to the obstacle based on the obstacle semantic label. This solves the technical problem that existing obstacle response methods can only achieve geometric-level obstacle detection, resulting in an inability to achieve accurate obstacle response, thereby improving driving safety. Attached Figure Description

[0026] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0027] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a flowchart illustrating an embodiment of the obstacle response method of this application. Figure 2 This is a flowchart illustrating Embodiment 2 of the obstacle response method of this application; Figure 3 This is a structural diagram of the preset target detection model in the obstacle response method of this application; Figure 4 This is a flowchart illustrating Embodiment 3 of the obstacle response method of this application; Figure 5 This is a schematic diagram of the overall process of the obstacle response method in this application; Figure 6 This is a schematic diagram of the module structure of the obstacle response device according to an embodiment of this application; Figure 7 This is a schematic diagram of the device structure of the hardware operating environment involved in the obstacle response method in the embodiments of this application.

[0029] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0030] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0031] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0032] The main solution of this application embodiment is: to detect obstacles on the road where the current vehicle is traveling and obtain obstacle information of road obstacles in the road; to determine whether the road obstacles meet the preset semantic reasoning trigger conditions based on the obstacle information; if they meet the conditions, to perform semantic reasoning on the multimodal environment data of the current vehicle to determine the obstacle semantic label corresponding to the road obstacle, the obstacle semantic label being used to characterize the type of road obstacle; and to respond to the road obstacle based on the obstacle semantic label.

[0033] Because existing obstacle response methods can only detect obstacles at the geometric level, their detection accuracy is not high. Therefore, they cannot generate optimal response strategies for obstacle response when faced with complex and ever-changing real-world driving scenarios.

[0034] This application provides a solution that can determine whether semantic reasoning is triggered based on obstacle information, and determine the semantic label of the obstacle through semantic reasoning when semantic reasoning is triggered. Finally, it responds to the obstacle based on the obstacle semantic label, thereby solving the technical problem that the obstacle response method in the prior art can only achieve obstacle detection at the geometric level, resulting in the inability to achieve accurate response to obstacles, and thus improving driving safety.

[0035] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device, obstacle response device, or vehicle intelligent driving system including an obstacle response device capable of realizing the above functions. The following description uses a vehicle intelligent driving system (hereinafter referred to as the system) as an example to illustrate this embodiment and the following embodiments.

[0036] Based on this, embodiments of this application provide an obstacle response method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the obstacle response method of this application.

[0037] In this embodiment, the obstacle response method includes steps S10 to S40: Step S10: Perform obstacle detection on the road where the current vehicle is traveling to obtain obstacle information of road obstacles in the road where the vehicle is traveling.

[0038] It should be understood that the aforementioned "currently traveling vehicle" refers to any vehicle currently traveling on the road; correspondingly, the aforementioned "road" refers to the segment of road the vehicle is currently traveling on, including lanes, sidewalks, roadside facilities, etc. In practical applications, the road environment may contain various obstacles, such as other vehicles, pedestrians, animals, fallen objects, etc. These obstacles can affect the normal driving of vehicles. Therefore, obstacle detection is required on the road the vehicle is currently traveling on to ensure driving safety.

[0039] It is understood that the aforementioned road obstacles can be objects on the road that may affect driving safety or require special attention. In this embodiment, road obstacles can include static obstacles (such as fallen objects or roadblocks) or dynamic obstacles (such as pedestrians, animals, or other vehicles). Correspondingly, the aforementioned obstacle information can be information used to characterize the features and attributes of the obstacle, including location, size, speed, category, confidence level, etc., which are not limited in this embodiment. In this embodiment, the system can use a lightweight detection model (such as YOLOv8n or MobileDet) to detect obstacles in the road. The model can then output the bounding box information and confidence level of the detected obstacles, thereby obtaining the obstacle information of the road obstacles.

[0040] Step S20: Determine whether the road obstacle meets the preset semantic reasoning triggering condition based on the obstacle information.

[0041] It should be noted that the aforementioned preset semantic reasoning trigger conditions can be trigger conditions for semantic reasoning of obstacles, such as the probability of the obstacle's existence, confidence level, etc. In this embodiment, the system can filter out obstacles that need further semantic analysis by using preset semantic reasoning trigger conditions, thereby avoiding complex semantic reasoning for all detected obstacles, thus saving computing resources and improving system efficiency.

[0042] In practical applications, the system can simultaneously collect multimodal environmental data of the road surface through sensors such as cameras, LiDAR, and millimeter-wave radar. A lightweight detection model processes this multimodal environmental data to output bounding box information and confidence scores for obstacles in the road. The system then checks whether the confidence score of an obstacle exceeds a preset threshold (e.g., 0.7). If the confidence score exceeds the threshold, the obstacle is considered to require further semantic analysis; that is, the obstacle is determined to meet the preset semantic reasoning triggering conditions. Furthermore, for certain specific categories of road obstacles (such as "animals" or "dangerous goods"), semantic reasoning can be triggered even if their confidence score is low.

[0043] Step S30: If satisfied, perform semantic reasoning on the multimodal environment data of the currently driving vehicle to determine the obstacle semantic label corresponding to the road obstacle. The obstacle semantic label is used to characterize the type and semantic information of the road obstacle.

[0044] It should be understood that the aforementioned multimodal environmental data can be information about the vehicle's surrounding environment collected by various sensors (such as cameras, LiDAR, millimeter-wave radar, etc.). This data includes features of different modalities, such as visual images, point cloud data, and radar data. Among them, visual data (RGB images) can be high-resolution RGB images collected by a front-facing camera, which can be used to extract the appearance features of obstacles; point cloud data can be three-dimensional point cloud data collected by LiDAR, which can be used to obtain the geometry and spatial position of obstacles; and radar data can be information such as the reflection intensity and speed of obstacles collected by millimeter-wave radar, which can be used to detect the motion state of obstacles.

[0045] It should be noted that the aforementioned obstacle semantic tags can be tags used to characterize the specific type and state of obstacles. In this embodiment, these tags can include not only the basic categories of obstacles (such as animals, dangerous goods, valuables, ordinary obstacles, etc.), but also more detailed semantic information (such as "injured animal", "leaked dangerous goods", "lost valuables", "ordinary obstacles", etc.).

[0046] In this embodiment, the system first extracts Regions of Interest (ROIs) from the multimodal environmental data of the currently driving vehicle (such as high-resolution RGB images, point cloud data, thermal imaging images, etc.). The ROI can be a local region in the multimodal data related to road obstacles, typically containing detailed features of the obstacles, such as appearance, shape, and motion state. Then, the system inputs these ROIs into a lightweight multimodal large-scale model on the vehicle to construct structured cue words containing high-resolution RGB images, thermal imaging images, LiDAR point cloud data, and vehicle context information. Based on these structured cue words, the multimodal large-scale model performs cross-modal understanding and reasoning to generate obstacle semantic labels containing risk level, obstacle type, and handling suggestions.

[0047] Step S40: Respond to the road obstacle based on the obstacle semantic label.

[0048] In practical applications, after determining the semantic label of an obstacle on the road, the system can perform corresponding response operations based on the semantic label. For example, if the semantic label of an obstacle is "ordinary obstacle", the system can perform automatic emergency braking (AEB) or perform route planning to avoid the obstacle; if the semantic label of an obstacle is "injured animal", the system can control the vehicle to slow down, alert the driver to the obstacle, and record the event; if the semantic label of an obstacle is "leaked hazardous materials", the system can perform emergency braking, display a red alert, and automatically contact the rescue department.

[0049] This embodiment provides an obstacle response method. The method discloses obstacle detection on the road where the vehicle is currently traveling, obtaining obstacle information of road obstacles in the road; determining whether the road obstacle meets a preset semantic reasoning trigger condition based on the obstacle information; if so, performing semantic reasoning on the multimodal environmental data of the current vehicle to determine the obstacle semantic label corresponding to the road obstacle, the obstacle semantic label being used to characterize the type of road obstacle; and responding to the road obstacle based on the obstacle semantic label. Compared to existing obstacle response methods with low obstacle detection accuracy, which makes it impossible to generate an optimal response strategy for obstacle response in complex scenarios, this embodiment can determine whether semantic reasoning is triggered based on obstacle information, and when semantic reasoning is triggered, determines the obstacle's semantic label through semantic reasoning, and finally responds to the obstacle based on the obstacle semantic label. This solves the technical problem that existing obstacle response methods can only achieve geometric-level obstacle detection, resulting in an inability to achieve accurate obstacle response, thereby improving driving safety.

[0050] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 , Figure 2 This is a flowchart illustrating Embodiment 2 of the obstacle response method of this application.

[0051] In this embodiment, step S10 includes steps S101 to S104: Step S101: Collect multimodal environmental data of the currently driving vehicle, including vehicle environment images, lidar point cloud data and radar target data.

[0052] It should be understood that the aforementioned vehicle environment images can be RGB images of the vehicle's surrounding environment captured by an onboard camera. These images provide visual information about obstacles, such as color, texture, and shape. The lidar point cloud data can be three-dimensional point cloud data of the vehicle's driving environment captured by a lidar sensor. Each point in this data can contain position coordinates (x, y, z) and reflection intensity, which can provide information about the three-dimensional geometry and spatial location of obstacles. The radar target data can be data related to the target (obstacle) collected by a millimeter-wave radar sensor, such as information about the obstacle's position, speed, and reflection intensity.

[0053] Step S102: Input the multimodal environment data into a preset target detection model, wherein the preset target detection model is provided with a spatiotemporal calibration submodule and a target detection network.

[0054] It should be noted that the aforementioned preset target detection model can be a deep learning model used to detect and identify obstacles on the road where vehicles are traveling, such as the lightweight YOLOv8n model. In this embodiment, the preset target detection model may include a spatiotemporal calibration submodule and a target detection network. The spatiotemporal calibration submodule can be a module used for spatiotemporal alignment of multimodal data; the target detection network can be a structure used to detect obstacles based on multimodal fusion data. By learning the features of multimodal data, it can output information such as the bounding box, category, and confidence score of the obstacle.

[0055] Step S103: The vehicle environment image, the lidar point cloud data and the radar target data are spatiotemporally aligned using the spatiotemporal calibration submodule to output multimodal fusion data.

[0056] It should be noted that spatiotemporal alignment processing can be the process of aligning data of different modalities in time and space; correspondingly, the aforementioned multimodal fusion data can be the data obtained after spatiotemporally aligning and fusing vehicle environment images, lidar point cloud data and radar target data, which can provide more comprehensive obstacle information.

[0057] In this embodiment, the spatiotemporal alignment of multimodal data through the spatiotemporal calibration submodule can ensure that the data of different modalities are consistent in time and space, so as to facilitate effective fusion and analysis in the future, thereby improving the accuracy of obstacle detection.

[0058] Step S104: The target detection network performs obstacle detection on the road of the currently driving vehicle based on the multimodal fusion data to obtain obstacle information of road obstacles in the road.

[0059] In its implementation, the system first acquires RGB images of the vehicle's surrounding environment from a front-facing camera, 3D point cloud data of the vehicle's surrounding environment from a LiDAR sensor, and velocity and position information of targets around the vehicle from a millimeter-wave radar sensor, thus obtaining multimodal environmental data. This multimodal environmental data is then input into a preset target detection model. (Refer to...) Figure 3 , Figure 3 This is a structural diagram of the preset target detection model in the obstacle response method of this application. (See diagram for example.) Figure 3As shown, after receiving multimodal environmental data, the preset target detection model can first perform spatiotemporal alignment processing on the vehicle environment image, LiDAR point cloud data, and radar target data in the multimodal environmental data through the spatiotemporal calibration submodule, and then fuse the aligned multimodal data together to generate and output multimodal fused data. Then, the target detection network can perform obstacle detection based on the multimodal fused data and output the detected obstacle information, including the obstacle's location, category, and confidence level. Specifically, the target detection network can first extract multi-level feature maps of obstacles from the input data through the backbone network. Shallow features capture the details of obstacles (such as edges and corners), while deep features capture more abstract semantic information of obstacles (such as object parts and overall shape). The extracted multi-scale features are then input into the neck network, which can then fuse the multi-scale features extracted by the backbone network to combine deep semantic information with shallow localization information, enhancing the model's ability to detect targets of different sizes (especially small targets). Finally, the detection head can perform detection on the fused feature map and output a list of detection results, which can include the location (bounding box, in the format of normalized coordinates [x_center, y_center, width, height]) of each potential obstacle, the confidence score (used to indicate the probability of the obstacle being in the box), and a coarse class label.

[0060] Furthermore, before step S20, the method further includes: determining the position information corresponding to all bounding boxes of the road obstacle based on the obstacle information; determining redundant bounding boxes from each bounding box based on the position information; filtering the redundant bounding boxes; and updating the obstacle information after filtering is completed.

[0061] It is understandable that the above location information can be parameters used to represent the specific location of the bounding box in the image or point cloud data, such as [x_center, y_center, width, height], where x_center represents the horizontal coordinate of the center point of the bounding box; y_center represents the vertical coordinate of the center point of the bounding box; width represents the width of the bounding box; and height represents the height of the bounding box.

[0062] It should be understood that the aforementioned redundant bounding boxes can be those that overlap significantly in position among all detected bounding boxes and may represent the same obstacle. In practical applications, redundant bounding boxes are usually generated by multiple detections or false detections by the target detection model, which increases the computational burden and may lead to misjudgments. Therefore, it is necessary to filter redundant bounding boxes to improve the accuracy and efficiency of detection results.

[0063] In this embodiment, the system can extract the position coordinates of all corresponding bounding boxes from the obstacle information of road obstacles, and use the Non-Maximum Suppression (NMS) algorithm to determine and remove redundant bounding boxes based on the position coordinates of the bounding boxes, thereby updating the obstacle information.

[0064] Further, step S20 includes: Step S201: Determine the bounding box confidence level of all bounding boxes corresponding to the road obstacles based on the updated obstacle information.

[0065] It should be understood that the bounding box mentioned above can be a rectangular region output by the object detection model, which can be used to identify the location of obstacles in the image or point cloud data; correspondingly, the bounding box confidence score can be the confidence assessment of the object detection model that there are obstacles in each bounding box, which is usually a value between 0 and 1.

[0066] Step S202: Determine the target bounding box from each bounding box based on the bounding box confidence level. The target bounding box is the bounding box with the highest confidence level among the bounding boxes.

[0067] It is understandable that the target bounding box can be the bounding box with the highest confidence among all bounding boxes. In this embodiment, the system can compare the confidence of all bounding boxes output by the model, and select the bounding box with the highest confidence as the target bounding box based on the comparison results.

[0068] Step S203: Determine whether the confidence level of the target bounding box exceeds a preset confidence threshold.

[0069] It should be noted that the aforementioned preset confidence threshold can be a numerical value used to determine whether the confidence of the bounding box is sufficient to trigger semantic reasoning.

[0070] Step S204: If the condition is exceeded, it is determined that the road obstacle meets the preset semantic reasoning triggering condition.

[0071] In practical applications, after processing multimodal fusion data using a preset target detection model, the model can output the detection results of all detected obstacles. Each detection result contains a confidence value. At this point, the system can select the bounding box with the highest confidence as the target bounding box and determine whether the confidence of the target bounding box exceeds the preset confidence threshold. If it does, the preset semantic reasoning condition is triggered.

[0072] This embodiment discloses the acquisition of multimodal environmental data of the currently driving vehicle; inputting the multimodal environmental data into a preset target detection model, which includes a spatiotemporal calibration submodule and a target detection network; performing spatiotemporal alignment processing on vehicle environment images, LiDAR point cloud data, and radar target data through the spatiotemporal calibration submodule, outputting multimodal fused data; and using the target detection network to detect obstacles on the vehicle's driving path based on the multimodal fused data, obtaining obstacle information of road obstacles in the vehicle's driving path. This embodiment, by fusing vehicle environment images, LiDAR point cloud data, and radar target data, can provide rich obstacle information for the preset target detection model, thereby improving the accuracy of obstacle detection. Simultaneously, this embodiment, through the spatiotemporal calibration submodule to align the multimodal data, can ensure the consistency of data from different modalities in time and space, thereby reducing errors caused by differences in sensor position and time, and improving the reliability of the detection results.

[0073] Based on the first and / or second embodiments of this application, in the third embodiment of this application, the content that is the same as or similar to the above embodiments can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 4 , Figure 4 This is a flowchart illustrating Embodiment 3 of the obstacle response method of this application.

[0074] In this embodiment, step S30 includes steps S301 to S304: Step S301: Determine the bounding box coordinates of the target bounding box based on the obstacle information.

[0075] It should be noted that the above bounding box coordinates can be the position coordinates of the target bounding box in the image or point cloud data.

[0076] Step S302: Map the bounding box coordinates to the original high-resolution image coordinate system to obtain the high-resolution bounding box coordinates of the target bounding box.

[0077] It should be noted that the above-mentioned original high-resolution image coordinate system can be the coordinate system under the original resolution of the vehicle environment image, which usually has a high resolution (such as 1920x1080); correspondingly, the above-mentioned high-resolution bounding box coordinates can be the coordinates obtained by mapping the low-resolution bounding box coordinates to the original high-resolution image coordinate system, which can be used to accurately locate obstacles in high-resolution images, thereby providing accurate regions for subsequent semantic reasoning.

[0078] Step S303: Determine the region of interest in the multimodal environment data based on the high-resolution bounding box coordinates. The region of interest is a local region in the multimodal environment data that is related to the road obstacle.

[0079] It should be understood that the aforementioned region of interest can be a local region in multimodal environmental data related to road obstacles, such as a high-resolution RGB image patch, a high-resolution thermal imaging image patch, or a subset of three-dimensional point clouds corresponding to the obstacle.

[0080] Step S304: Perform semantic reasoning on the region of interest to determine the obstacle semantic label corresponding to the road obstacle.

[0081] In practical applications, the system can extract the bounding box coordinates corresponding to the target bounding box from the obstacle information of road obstacles. These bounding box coordinates are low-resolution coordinates. Then, based on the size ratio of the low-resolution and high-resolution images, the system maps the low-resolution bounding box coordinates to the high-resolution image to obtain the high-resolution bounding box coordinates of the target bounding box. Based on these high-resolution bounding box coordinates, the system accurately crops the region containing the obstacle from all original high-resolution data (including high-resolution images, point cloud data, or radar data), thus obtaining the region of interest. Finally, the system can input the extracted region of interest into the semantic reasoning module to generate semantic labels for the obstacles.

[0082] Further, step S304 includes: Step S304a: Input the region of interest into a preset multimodal large model, wherein the preset multimodal large model is a lightweight multimodal large model for vehicle end.

[0083] It should be noted that the aforementioned preset multimodal large model can be a lightweight multimodal large model that can run on the vehicle side. This model can process data of multiple modalities (such as RGB images, thermal imaging images, LiDAR point cloud data, etc.) and perform semantic reasoning.

[0084] In this embodiment, the preset multimodal big model can process and understand data from different modalities simultaneously. By understanding the content and relationships of multimodal data, it can generate semantic labels for obstacles. These labels can include a detailed description of the obstacles, such as risk level, obstacle type, and handling suggestions.

[0085] Step S304b: Construct structured prompts based on the high-resolution RGB image of the region of interest, the high-resolution thermal imaging image, the lidar point cloud data, and the vehicle context information.

[0086] It should be understood that the aforementioned high-resolution RGB images can be RGB images extracted from the region of interest (ROI). These images have high resolution and provide rich visual details, allowing for the extraction of visual features of obstacles, such as color, texture, and shape. High-resolution thermal imaging images can be thermal images extracted from the ROI, providing temperature information about obstacles, which can be used to detect thermal features such as the body temperature of an animal or the temperature changes of a leaked liquid. LiDAR point cloud data can be 3D point cloud data extracted from the ROI, including the geometry and spatial location information of obstacles. This data can be used to extract 3D features of obstacles, such as size, shape, and surface contours. Vehicle context information can be vehicle-related information extracted from the ROI, including the vehicle's current state, such as speed, position, and direction of travel. This provides additional contextual information for semantic reasoning, helping the model to more accurately understand the potential risks posed by obstacles.

[0087] It should be noted that the structured prompts mentioned above can be prompts used to guide multimodal large models in reasoning. In this embodiment, the structured prompts may include visual information provided by high-resolution RGB images, temperature distribution information provided by thermal imaging images, three-dimensional geometric information provided by lidar point clouds, and contextual information such as the vehicle's current GPS location, speed, and time.

[0088] Step S304c: Generate obstacle description text corresponding to the road obstacle based on the structured prompt words using the preset multimodal large model.

[0089] It is understandable that the obstacle description text mentioned above can be a natural language description of the obstacles, including a detailed analysis and recommendations, output by a multimodal large model.

[0090] Step S304d: Determine the obstacle semantic label corresponding to the road obstacle based on the obstacle description text.

[0091] In this embodiment, after extracting the region of interest from the original high-resolution multimodal data, the system can combine the high-resolution RGB image, thermal imaging image, LiDAR point cloud data, and vehicle context information into structured prompts, which are then input into a preset multimodal large model. The large model then uses its internal cross-modal understanding mechanism to analyze image content, thermal imaging features, and point cloud geometric information, and combines this with vehicle up / down information to generate a natural language description containing detailed obstacle descriptions, i.e., obstacle description text. Risk level, obstacle type, and handling suggestions can then be extracted from the generated obstacle description text to form the final obstacle semantic label.

[0092] Further, step S40 includes: Step S401: Parse the semantic tags of the obstacles to generate obstacle response instructions.

[0093] It should be noted that the aforementioned obstacle response commands can be operational instructions used to guide the vehicle control system to execute corresponding response actions. In practical applications, the system can parse the semantic labels of obstacles (including risk level, obstacle type, and handling suggestions) into specific response commands, and control the vehicle to execute corresponding actions based on the generated response commands, so as to ensure that the vehicle can take appropriate measures against obstacles according to their type, state, and risk level, such as emergency braking, deceleration, avoidance, and alarm.

[0094] Further, step S401 includes: parsing the semantic tags of the obstacles to determine the risk level, obstacle type, and handling suggestions of the road obstacles; generating a response strategy corresponding to the road obstacles based on the risk level, obstacle type, and handling suggestions; and generating obstacle response instructions according to the response strategy.

[0095] It should be understood that the aforementioned risk level can be an indicator used to characterize the potential danger level of an obstacle. In this embodiment, the risk level of road obstacles can be divided into high risk, medium risk, and low risk. The aforementioned obstacle type can be a description of the specific category of the obstacle, such as "animals", "dangerous goods", "valuables", "ordinary obstacles", etc. The aforementioned handling suggestions can be specific operational suggestions generated based on the risk level and obstacle type, such as "emergency braking", "slowing down and avoiding", "alarm notification", etc.

[0096] It is understood that the above response strategies can be specific actions taken based on the different risk levels, types, and handling recommendations of obstacles, such as emergency braking, deceleration, avoidance, alarm, and driver alert, etc. This embodiment does not limit these actions.

[0097] Step S402: Respond to the road obstacle based on the obstacle response command.

[0098] Furthermore, the obstacle response command includes a vehicle obstacle avoidance command; if the obstacle response command is a vehicle obstacle avoidance command, then the current vehicle is replanned to obtain a replanned driving path; the current vehicle is controlled to drive according to the replanned driving path to avoid the road obstacle.

[0099] It should be noted that the aforementioned vehicle obstacle avoidance command can be an instruction to the vehicle control system to replan the vehicle's driving path to ensure the vehicle safely bypasses obstacles; correspondingly, the replanned driving path can be a vehicle driving path that avoids obstacles. In this embodiment, when the system detects that the road obstacle is a low-risk ordinary obstacle, it can generate a vehicle obstacle avoidance command to instruct the vehicle control system to replan the vehicle's driving path using a path planning algorithm (such as Dijkstra's algorithm or a sampling-based method), obtain a replanned driving path, and control the vehicle to drive according to the replanned driving path to avoid obstacles and ensure vehicle driving safety.

[0100] Furthermore, the obstacle response command also includes an obstacle warning command; if the obstacle response command is an obstacle warning command, then the obstacle warning method corresponding to the road obstacle is determined; and the road obstacle is warned through the obstacle warning method.

[0101] It should be understood that the aforementioned obstacle warning instructions can be instructions that alert the driver to potential hazards through visual, auditory, or other means. Correspondingly, the aforementioned obstacle warning methods can be methods used to alert the driver to the presence of obstacles on the road, such as visual cues, auditory cues, tactile cues, and automatic alarms. Specifically, visual cues can be displayed on the vehicle's dashboard, HUD (Head-Up Display), or central control screen; auditory cues can be issued by emitting an alarm sound through the vehicle horn or in-vehicle speakers; tactile cues can be issued by vibrating the seat or other tactile feedback; and automatic alarms can be sent to emergency rescue departments. In this embodiment, the system can select an appropriate warning method based on the obstacle's risk level and the specific driving scenario to ensure the driver can notice the obstacle's presence in a timely manner, thereby improving driving safety.

[0102] In the specific implementation, refer to Figure 5 , Figure 5 This is a schematic diagram illustrating the overall process of the obstacle response method in this application. Figure 5As shown, the system first acquires and fuses data from cameras, LiDAR, and millimeter-wave radar simultaneously through a multimodal perception module, and performs spatiotemporal alignment on this data to obtain multimodal fused data. Then, the system uses a target detection network to detect whether there are suspicious obstacles on the currently traveling road and outputs obstacle information. Based on this information, the system determines whether the obstacle meets preset semantic reasoning trigger conditions. When an obstacle meets these conditions, the semantic reasoning model is activated to perform semantic reasoning, outputting four semantic labels corresponding to the obstacle: live casualties, dangerous goods, valuables, and ordinary obstacles. Subsequently, based on the parsed risk level, obstacle type, and handling suggestions, the system selects the corresponding response priority, response method, and specific response action to provide differentiated responses to road obstacles, including vehicle control commands, human-machine interaction displays, and communication triggers.

[0103] This embodiment discloses a method for determining the bounding box coordinates of a target bounding box based on obstacle information; mapping the bounding box coordinates to the original high-resolution image coordinate system to obtain the high-resolution bounding box coordinates of the target bounding box; determining the region of interest (ROI) in the multimodal environmental data based on the high-resolution bounding box coordinates, where the ROI is a local region in the multimodal environmental data related to road obstacles; and performing semantic reasoning on the ROI to determine the obstacle semantic labels corresponding to the road obstacles. Since this embodiment can extract obstacle-related local regions from multimodal environmental data based on high-resolution bounding box coordinates, the amount of data processed can be reduced, thereby improving obstacle response efficiency. Furthermore, by performing semantic reasoning on the ROI, this embodiment can generate detailed obstacle semantic labels and perform differentiated responses to obstacles based on these semantic labels, thus enabling more flexible handling of various complex situations and improving driving safety and emergency response capabilities.

[0104] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the obstacle response method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0105] This application also provides an obstacle response device, please refer to... Figure 6 The obstacle response device includes: The obstacle detection module 10 is used to detect obstacles on the road where the vehicle is currently traveling and to obtain obstacle information of road obstacles in the road where the vehicle is traveling. The semantic reasoning triggering module 20 is used to determine whether the road obstacle meets the preset semantic reasoning triggering conditions based on the obstacle information; The semantic reasoning module 30 is used to perform semantic reasoning on the multimodal environment data of the currently driving vehicle if the conditions are met, and determine the obstacle semantic label corresponding to the road obstacle. The obstacle semantic label is used to characterize the type and semantic information of the road obstacle. The obstacle response module 40 is used to respond to the road obstacles based on the obstacle semantic tags.

[0106] The obstacle response device provided in this application, employing the obstacle response method in the above embodiments, can solve the technical problem that existing obstacle response methods can only achieve geometric-level obstacle detection, resulting in an inability to achieve accurate obstacle response. Compared with the prior art, the beneficial effects of the obstacle response device provided in this application are the same as those of the obstacle response method provided in the above embodiments, and other technical features in the obstacle response device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0107] This application provides an obstacle response device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the obstacle response method in Embodiment 1 above.

[0108] The following is for reference. Figure 7 The diagram illustrates a structural schematic suitable for implementing an obstacle response device according to embodiments of this application. The obstacle response device in these embodiments may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 7 The obstacle response device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of this application.

[0109] like Figure 7As shown, the obstacle response device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the obstacle response device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the obstacle response device to communicate wirelessly or wiredly with other devices to exchange data. While the figures show obstacle response devices with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0110] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0111] The obstacle response device provided in this application, employing the obstacle response method in the above embodiments, can solve the technical problem of obstacle response. Compared with the prior art, the beneficial effects of the obstacle response device provided in this application are the same as those of the obstacle response method provided in the above embodiments, and other technical features of the obstacle response device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0112] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0113] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0114] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the obstacle response method in the above embodiments.

[0115] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0116] The aforementioned computer-readable storage medium may be included in the obstacle response device; or it may exist independently and not assembled into the obstacle response device.

[0117] The aforementioned computer-readable storage medium carries one or more programs. When the one or more programs are executed by the obstacle response device, the obstacle response device performs the following actions: detects obstacles on the road where the currently traveling vehicle is traveling, and obtains obstacle information of road obstacles on the road; determines whether the road obstacles meet preset semantic reasoning trigger conditions based on the obstacle information; if they do, performs semantic reasoning on the multimodal environment data of the currently traveling vehicle, determines the obstacle semantic label corresponding to the road obstacle, the obstacle semantic label being used to characterize the type of road obstacle; and responds to the road obstacle based on the obstacle semantic label.

[0118] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0119] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0120] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0121] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described obstacle response method. This solves the technical problem that existing obstacle response methods can only achieve geometric-level obstacle detection, resulting in an inability to achieve accurate obstacle response. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the obstacle response method provided in the above embodiments, and will not be repeated here.

[0122] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the obstacle response method as described above.

[0123] The computer program product provided in this application solves the technical problem that existing obstacle response methods can only achieve geometric-level obstacle detection, resulting in an inability to achieve accurate obstacle response. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the obstacle response methods provided in the above embodiments, and will not be repeated here.

[0124] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

[0125] This invention discloses A1, an obstacle response method, the method comprising: Obstacle detection is performed on the road where the vehicle is currently traveling to obtain obstacle information of road obstacles in the road where the vehicle is traveling; Based on the obstacle information, determine whether the road obstacle meets the preset semantic reasoning trigger condition; If the conditions are met, semantic reasoning is performed on the multimodal environment data of the currently driving vehicle to determine the obstacle semantic label corresponding to the road obstacle. The obstacle semantic label is used to characterize the type and semantic information of the road obstacle. The system responds to the road obstacles based on the obstacle semantic tags.

[0126] A2. The method described in A1, wherein the step of detecting obstacles on the road where the currently traveling vehicle is traveling and obtaining obstacle information of road obstacles in the road where the vehicle is traveling includes: Collect multimodal environmental data of the currently driving vehicle, including vehicle environment images, lidar point cloud data, and radar target data; The multimodal environment data is input into a preset target detection model, which includes a spatiotemporal calibration submodule and a target detection network. The spatiotemporal calibration submodule performs spatiotemporal alignment processing on the vehicle environment image, the lidar point cloud data, and the radar target data to output multimodal fusion data. The target detection network performs obstacle detection on the road where the currently driving vehicle is traveling based on the multimodal fusion data, thereby obtaining obstacle information of road obstacles in the road where the vehicle is traveling.

[0127] A3. As described in A2, before the step of determining whether the road obstacle meets the preset semantic reasoning triggering condition based on the obstacle information, the method further includes: Based on the obstacle information, determine the location information corresponding to all bounding boxes of the road obstacles; Redundant bounding boxes are determined from each bounding box based on the location information; The redundant bounding boxes are filtered, and the obstacle information is updated after the filtering is completed.

[0128] A4. As described in A3, the step of determining whether the road obstacle meets the preset semantic reasoning triggering condition based on the obstacle information includes: Determine the bounding box confidence level of all bounding boxes corresponding to the road obstacles based on the updated obstacle information; The target bounding box is determined from each bounding box based on the bounding box confidence score, and the target bounding box is the bounding box with the highest confidence score among the bounding boxes. Determine whether the confidence level of the target bounding box exceeds a preset confidence threshold; If the condition is exceeded, the road obstacle is determined to meet the preset semantic reasoning trigger condition.

[0129] A5. As described in A4, the step of performing semantic reasoning on the multimodal environment data of the currently driving vehicle to determine the obstacle semantic label corresponding to the road obstacle includes: The bounding box coordinates of the target bounding box are determined based on the obstacle information; The bounding box coordinates are mapped to the original high-resolution image coordinate system to obtain the high-resolution bounding box coordinates of the target bounding box; The region of interest in the multimodal environment data is determined based on the high-resolution bounding box coordinates. The region of interest is a local region in the multimodal environment data that is related to the road obstacle. Semantic reasoning is performed on the region of interest to determine the obstacle semantic labels corresponding to the road obstacles.

[0130] A6. As described in A5, the step of performing semantic reasoning on the region of interest to determine the obstacle semantic label corresponding to the road obstacle includes: The region of interest is input into a preset multimodal large model, which is a lightweight multimodal large model for vehicle end. Structured prompts are constructed based on the high-resolution RGB image, high-resolution thermal image, LiDAR point cloud data, and vehicle context information of the region of interest. The preset multimodal large model generates obstacle description text corresponding to the road obstacles based on the structured prompt words; The obstacle semantic label corresponding to the road obstacle is determined based on the obstacle description text.

[0131] A7. The method described in A6, wherein the step of responding to the road obstacle based on the obstacle semantic label includes: The semantic tags of the obstacles are parsed to generate obstacle response commands; The road obstacle is responded to based on the obstacle response command.

[0132] A8. As described in A7, the step of parsing the obstacle semantic tags and generating obstacle response instructions includes: The semantic tags of the obstacles are parsed to determine the risk level, obstacle type, and handling recommendations for the road obstacles; Generate a response strategy for the road obstacle based on the risk level, the obstacle type, and the proposed handling; Obstacle response commands are generated according to the response strategy.

[0133] A9. As described in A7, the obstacle response command includes a vehicle obstacle avoidance command; the step of responding to the road obstacle based on the obstacle response command includes: If the obstacle response command is a vehicle obstacle avoidance command, then the current vehicle is replanned to obtain the replanned driving path. The vehicle is controlled to travel along the replanned path in order to avoid road obstacles.

[0134] A10. As described in A7, the obstacle response instruction further includes an obstacle warning instruction; the step of responding to the road obstacle based on the obstacle response instruction includes: If the obstacle response command is an obstacle warning command, then the obstacle warning method corresponding to the road obstacle is determined; The obstacle warning method is used to warn of road obstacles.

[0135] The present invention also discloses B11, an obstacle response device, the device comprising: The obstacle detection module is used to detect obstacles on the road where the vehicle is currently traveling and to obtain obstacle information of road obstacles in the road where the vehicle is traveling. The semantic reasoning triggering module is used to determine whether the road obstacle meets the preset semantic reasoning triggering conditions based on the obstacle information. The semantic reasoning module is used to perform semantic reasoning on the multimodal environment data of the currently driving vehicle if the conditions are met, and determine the obstacle semantic label corresponding to the road obstacle. The obstacle semantic label is used to characterize the type and semantic information of the road obstacle. An obstacle response module is used to respond to the road obstacles based on the obstacle semantic tags.

[0136] B12. The device as described in B11, wherein the obstacle detection module is further configured to collect multimodal environmental data of the currently driving vehicle, the multimodal environmental data including vehicle environment images, lidar point cloud data and radar target data; The obstacle detection module is also used to input the multimodal environment data into a preset target detection model, wherein the preset target detection model is provided with a spatiotemporal calibration submodule and a target detection network; The obstacle detection module is also used to perform spatiotemporal alignment processing on the vehicle environment image, the lidar point cloud data and the radar target data through the spatiotemporal calibration submodule, and output multimodal fusion data; The obstacle detection module is further configured to perform obstacle detection on the vehicle's driving road based on the multimodal fusion data through the target detection network, and obtain obstacle information of road obstacles in the vehicle's driving road.

[0137] B13. The device as described in B12, wherein the semantic reasoning triggering module is further configured to determine the bounding box confidence level corresponding to all bounding boxes of the road obstacle based on the updated obstacle information; The semantic reasoning triggering module is also used to determine the target bounding box from each bounding box based on the bounding box confidence, wherein the target bounding box is the bounding box with the highest confidence. The semantic reasoning triggering module is also used to determine whether the confidence level of the target bounding box exceeds a preset confidence threshold. The semantic reasoning triggering module is also used to determine that the road obstacle meets the preset semantic reasoning triggering conditions if the condition is exceeded.

[0138] B14. The apparatus as described in B11, wherein the semantic reasoning module is further configured to determine the bounding box coordinates of the road obstacle based on the obstacle information; The semantic reasoning module is also used to map the bounding box coordinates to the original high-definition image coordinate system to obtain high-resolution bounding box coordinates; The semantic reasoning module is further configured to determine the region of interest in the multimodal environment data based on the high-resolution bounding box coordinates, wherein the region of interest is a local region in the multimodal environment data that is related to the road obstacle; The semantic reasoning module is also used to perform semantic reasoning on the region of interest to determine the obstacle semantic label corresponding to the road obstacle.

[0139] B15. The device as described in B14, wherein the semantic reasoning module is further configured to input the region of interest into a preset multimodal large model, wherein the preset multimodal large model is a vehicle-side lightweight multimodal large model; The semantic reasoning module is also used to construct structured prompt words based on the high-resolution RGB image, high-resolution thermal imaging image, lidar point cloud data and vehicle context information of the region of interest; The semantic reasoning module is also used to generate obstacle description text corresponding to the road obstacle based on the structured prompt words using the preset multimodal large model; The semantic reasoning module is also used to determine the obstacle semantic label corresponding to the road obstacle based on the obstacle description text.

[0140] B16. In the apparatus described in B15, the obstacle response module is further configured to parse the obstacle semantic tags and generate obstacle response instructions; The obstacle response module is also used to respond to the road obstacle based on the obstacle response command.

[0141] B17. The device as described in B16, wherein the obstacle response module is further configured to parse the semantic tags of the obstacles to determine the risk level, obstacle type and handling recommendations of the road obstacles; The obstacle response module is also used to generate a response strategy corresponding to the road obstacle based on the risk level, the obstacle type, and the handling suggestion; The obstacle response module is also used to generate obstacle response instructions according to the response strategy.

[0142] The present invention also discloses C18, an obstacle response device, the device comprising: a memory, a processor, and an obstacle response program stored in the memory and executable on the processor, the obstacle response program being configured to implement the steps of the obstacle response method as described above.

[0143] The present invention also discloses D19, a storage medium storing an obstacle response program, wherein the obstacle response program, when executed by a processor, implements the steps of the obstacle response method as described above.

[0144] The present invention also discloses E20, a computer program product comprising a computer program that, when executed by a processor, implements the steps of the obstacle response method as described above.

Claims

1. An obstacle response method, characterized in that, The method includes: Obstacle detection is performed on the road where the vehicle is currently traveling to obtain obstacle information of road obstacles in the road where the vehicle is traveling; Based on the obstacle information, determine whether the road obstacle meets the preset semantic reasoning trigger condition; If the conditions are met, semantic reasoning is performed on the multimodal environment data of the currently driving vehicle to determine the obstacle semantic label corresponding to the road obstacle. The obstacle semantic label is used to characterize the type and semantic information of the road obstacle. The system responds to the road obstacles based on the obstacle semantic tags.

2. The method as described in claim 1, characterized in that, The step of detecting obstacles on the road where the currently traveling vehicle is traveling and obtaining obstacle information of road obstacles in the road includes: Collect multimodal environmental data of the currently driving vehicle, including vehicle environment images, lidar point cloud data, and radar target data; The multimodal environment data is input into a preset target detection model, which includes a spatiotemporal calibration submodule and a target detection network. The spatiotemporal calibration submodule performs spatiotemporal alignment processing on the vehicle environment image, the lidar point cloud data, and the radar target data to output multimodal fusion data. The target detection network performs obstacle detection on the road where the current vehicle is traveling based on the multimodal fusion data, thereby obtaining obstacle information of road obstacles in the road where the vehicle is traveling.

3. The method as described in claim 2, characterized in that, Before the step of determining whether the road obstacle meets the preset semantic reasoning triggering condition based on the obstacle information, the method further includes: Based on the obstacle information, determine the location information corresponding to all bounding boxes of the road obstacles; Redundant bounding boxes are determined from each bounding box based on the location information; The redundant bounding boxes are filtered, and the obstacle information is updated after the filtering is completed.

4. The method as described in claim 3, characterized in that, The step of determining whether the road obstacle meets the preset semantic reasoning triggering condition based on the obstacle information includes: Determine the bounding box confidence level of all bounding boxes corresponding to the road obstacles based on the updated obstacle information; The target bounding box is determined from each bounding box based on the bounding box confidence score, and the target bounding box is the bounding box with the highest confidence score among the bounding boxes. Determine whether the confidence level of the target bounding box exceeds a preset confidence threshold; If the condition is exceeded, the road obstacle is determined to meet the preset semantic reasoning trigger condition.

5. The method as described in claim 4, characterized in that, The step of performing semantic reasoning on the multimodal environment data of the currently driving vehicle to determine the obstacle semantic label corresponding to the road obstacle includes: The bounding box coordinates of the target bounding box are determined based on the obstacle information; The bounding box coordinates are mapped to the original high-resolution image coordinate system to obtain the high-resolution bounding box coordinates of the target bounding box; The region of interest in the multimodal environment data is determined based on the high-resolution bounding box coordinates. The region of interest is a local region in the multimodal environment data that is related to the road obstacle. Semantic reasoning is performed on the region of interest to determine the obstacle semantic labels corresponding to the road obstacles.

6. The method as described in claim 5, characterized in that, The step of performing semantic reasoning on the region of interest to determine the obstacle semantic labels corresponding to the road obstacles includes: The region of interest is input into a preset multimodal large model, which is a lightweight multimodal large model for vehicle end. Structured prompts are constructed based on the high-resolution RGB image, high-resolution thermal image, LiDAR point cloud data, and vehicle context information of the region of interest. The preset multimodal large model generates obstacle description text corresponding to the road obstacles based on the structured prompt words; The semantic tags corresponding to the road obstacles are determined based on the obstacle description text.

7. An obstacle response device, characterized in that, The device includes: The obstacle detection module is used to detect obstacles on the road where the vehicle is currently traveling and to obtain obstacle information of road obstacles in the road where the vehicle is traveling. The semantic reasoning triggering module is used to determine whether the road obstacle meets the preset semantic reasoning triggering conditions based on the obstacle information. The semantic reasoning module is used to perform semantic reasoning on the multimodal environment data of the currently driving vehicle if the conditions are met, and determine the obstacle semantic label corresponding to the road obstacle. The obstacle semantic label is used to characterize the type and semantic information of the road obstacle. An obstacle response module is used to respond to the road obstacles based on the obstacle semantic tags.

8. An obstacle response device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the obstacle response method as described in any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the obstacle response method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the obstacle response method as described in any one of claims 1 to 6.

Images