Observation state determination method and related device
By determining the observation status of the object to be observed, the problem of reduced accuracy of the perception module in the autonomous driving system is solved, and the accuracy of environmental perception is improved.
Patent Information
- Application Number
- CN202311819335.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-26
- Publication Date
- 2025-06-27
AI Technical Summary
In the existing autonomous driving system, the accuracy of the perception module is reduced because the sensor obtains the perception information not in the optimal observation area, and there is no information on whether the target is in the optimal observation area.
By obtaining the perceived results of the target vehicle, the preset target observation range is determined according to the type of the object to be observed, and the observation status of the object to be observed is determined based on the perceived results and the target observation range.
The accuracy of environmental perception in the autonomous driving system is improved, allowing downstream modules to perform special processing on perceived results in non-optimal observation states.
Smart Images
Figure CN120215478A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of autonomous driving technology, and in particular, to a method for determining an observation state and related devices. Background Art
[0002] An autonomous driving system includes three main modules: perception, decision-making, and control. Among them, the perception module is mainly responsible for detecting various moving and stationary obstacles (such as vehicles, pedestrians, buildings, etc.), and collecting various information on the road (such as drivable areas, lane lines, traffic signs, etc.), so that the vehicle can make reasonable decisions based on the perception results. Commonly used sensors include cameras, lidar, etc.
[0003] Under the related technology, due to the limited perception ability of a single sensor, the current mainstream environmental perception solution is a multi-sensor fusion solution. By complementing and enhancing different sensors, the advantages of each sensor are integrated to uniformly output more stable and comprehensive perception information for the downstream module, enabling the control module to achieve the ultimate safe driving of the vehicle based on accurate and stable perception results.
[0004] However, in the current fusion perception system, when performing environmental perception through each sensor, only the distance relationship between the sensor and the target is simply considered, resulting in the perception information obtained based on the sensor not being the information in the optimal observation area. The observation results in the non-optimal observation area often do not meet the usage requirements of the downstream module, and without information on whether the target is in the optimal observation area, the downstream module cannot perform special processing on the target detection results obtained in the non-optimal observation area, thereby reducing the accuracy of the perception module of the autonomous driving system. Summary of the Invention
[0005] This application provides a method for determining an observation state and related devices to improve the accuracy of environmental perception in an autonomous driving system.
[0006] In a first aspect, an embodiment of this application provides a method for determining an observation state, and the method includes:
[0007] Obtain the perception results of at least one object to be observed corresponding to the target vehicle, where the perception results include the target detection frames of at least one object to be observed;
[0008] For at least one object to be observed, perform the following operations respectively:
[0009] According to the type of an object to be observed, determine the preset target observation range for the object to be observed, where the target observation range is the ideal observation area preset for the object to be observed;
[0010] Determine the observation state of an object to be observed based on the perception result and the target observation range.
[0011] In a second aspect, an embodiment of the present application further provides an observation state determination device, including:
[0012] An acquisition module, configured to acquire the perception result of at least one object to be observed corresponding to the target vehicle, where the perception result includes at least one target detection box of the object to be observed;
[0013] A processing module, configured to perform the following operations respectively for at least one object to be observed:
[0014] Determine the preset target observation range for an object to be observed according to the type of the object to be observed, where the target observation range is an ideal observation area preset for the object to be observed;
[0015] Determine the observation state of an object to be observed based on the perception result and the target observation range.
[0016] Optionally, when determining the preset target observation range for an object to be observed according to the type of the object to be observed, the processing module is configured to:
[0017] Determine the observation distance range starting from a preset key point on the target vehicle;
[0018] Determine the observation angle range with the longitudinal direction of the target vehicle body as the reference direction.
[0019] Optionally, when determining the observation state of an object to be observed based on the perception result and the target observation range, the processing module is configured to:
[0020] Obtain the target distance between each point in the target detection box and the preset key point on the target vehicle, and the target angle between the target direction where each point in the target detection box is located and the longitudinal direction of the target vehicle body;
[0021] If there is at least one point in the target detection box whose corresponding target distance and target angle respectively satisfy the observation distance range and the observation angle range, determine the observation state of an object to be observed as the ideal observation state;
[0022] If there is no point in the target detection box whose corresponding target distance and target angle respectively satisfy the observation distance range and the observation angle range, determine the observation state of an object to be observed as the non-ideal observation state.
[0023] Optionally, the processing module is further configured to:
[0024] Based on the type of an object to be observed, a target feature region of the object to be observed is obtained, and the target feature region is the core area of concern of the object to be observed relative to the target vehicle;
[0025] If the target feature region is not within the preset range from the target vehicle, it indicates that the target feature region of an object to be observed is not observed;
[0026] If the target feature region is within the preset range from the target vehicle, it indicates that the target feature region of an object to be observed is observed.
[0027] Optionally, the combination of the observation state and the observation result of an object to be observed includes at least any one of the following:
[0028] The observation state is an ideal observation state, and the target feature region is observed;
[0029] The observation state is an ideal observation state, and the target feature region is not observed;
[0030] The observation state is a non-ideal observation state, and the target feature region is observed;
[0031] The observation state is a non-ideal observation state, and the target feature region is not observed.
[0032] In a third aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the method described in any item of the first aspect is implemented.
[0033] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in any item of the first aspect are implemented.
[0034] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product is called by a computer, the computer is enabled to execute the method described in the first aspect.
[0035] In an embodiment of the present application, a method and related device for determining an observation state are proposed. The server obtains the perception results of at least one object to be observed corresponding to a target vehicle, determines the preset target observation range for one object to be observed according to the type of one object to be observed, and finally determines the observation state of one object to be observed based on the perception results and the target observation range. In this way, after the perception system obtains the perception results of the objects to be observed, different optimal observation areas of sensors are set for different types of objects to be observed, and the observation states corresponding to the objects to be observed are recorded and output to the downstream module together with the perception results of the objects to be observed, providing more auxiliary information of the observation results for the downstream processing module, enabling the corresponding module to perform special processing on the perception results obtained in a non-optimal observation state, thereby improving the accuracy of the autonomous driving perception system.
[0036] Other features and advantages of the present application will be described in the following specification, and in part will be obvious from the specification, or will be understood by implementing the present application. The objectives and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written specification, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1 It is a schematic diagram of a possible application scenario in an embodiment of the present application;
[0038] Figure 2 It is a flowchart of a method for determining an observation state in an embodiment of the present application;
[0039] Figure 3 It is a first schematic diagram of a parking space observation scenario in an embodiment of the present application;
[0040] Figure 4 It is a flowchart of a method for obtaining the observation state of an object to be observed in an embodiment of the present application;
[0041] Figure 5 It is a second schematic diagram of a parking space observation scenario in an embodiment of the present application;
[0042] Figure 6 It is a flowchart of a method for obtaining the observation result corresponding to an observation object in an embodiment of the present application;
[0043] Figure 7 It is a third schematic diagram of a parking space observation scenario in an embodiment of the present application;
[0044] Figure 8 It is a fourth schematic diagram of a parking space observation scenario in an embodiment of the present application;
[0045] Figure 9 It is a schematic diagram of a vehicle observation scenario in an embodiment of the present application;
[0046] Figure 10 It is a schematic structural diagram of an observation state determination device in an embodiment of the present application;
[0047] Figure 11 It is a schematic structural diagram of an electronic device in an embodiment of the present application. Detailed implementation manners
[0048] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the following will clearly and completely describe the technical solutions of the present application in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solutions of the present application, rather than all of them. Based on the embodiments described in this application document, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the technical solutions of the present application.
[0049] Terms such as "first" and "second" in the description and claims of the present application and the above-mentioned accompanying drawings are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such data used can be interchanged under appropriate circumstances so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein.
[0050] The following explains some terms in the embodiments of the present application to facilitate the understanding of those skilled in the art.
[0051] (1) Bird's Eye View (BEV): The BEV network can convert the input data into a top view. In the field of autonomous driving, it is usually used to process sensor data such as lidar and cameras.
[0052] (2) Real-Time Models for object Detection (RTMDet): It can be used to efficiently complete object recognition tasks, such as instance segmentation and rotating object detection.
[0053] (3) Parking space line: It refers to the lines that form a parking space. A normal parking space has 4 parking space lines, which are functionally divided into an entrance line, a dividing line, and a tail line. The entrance line is the parking space line used to indicate the vehicle's entry when parking, and the tail line is the parking space line used to limit the vehicle's tail boundary. The remaining parking space lines are the dividing lines. Along the dividing lines, the direction from the entrance line to the tail line is the parking space direction.
[0054] (4) Parkability: There are no obstacles in the parking space affecting parking, that is, the parking space is not occupied, indicating that the parking space is parkable, otherwise the parking space is not parkable.
[0055] The following briefly introduces the design concept of the embodiments of the present application:
[0056] The perception system in autonomous driving technology is the "eyes" of autonomous vehicles, used to provide environmental information outside the vehicle for autonomous vehicles. It relies on the information input of various on-vehicle sensors, including cameras, ultrasonic radars, lidars, etc.
[0057] In the current perception system, when performing environmental perception through each sensor, only the distance relationship between the sensor and the target is simply considered, and the perspective relationship between the sensor and the target is not considered. As a result, the perception information obtained based on the sensor is not the information in the optimal observation area. The observation results in the non-optimal observation area often do not meet the usage requirements of the downstream module, and without the auxiliary information on whether the target is in the optimal observation area, the downstream module cannot perform special processing on the target detection results obtained in the non-optimal observation area, thus leading to a reduction in the accuracy of the perception module of the autonomous driving system.
[0058] In view of this, in the implementation of this application, an observation state determination method and related device are proposed.
[0059] In the embodiment of this application, the server obtains the perception results of at least one object to be observed corresponding to the target vehicle, determines the preset target observation range for one object to be observed according to the type of one object to be observed, and finally determines the observation state of one object to be observed based on the perception results and the target observation range.
[0060] In this way, after the perception system obtains the perception results of the object to be observed, different optimal observation areas of the sensor are set for different types of objects to be observed, and the observation state corresponding to the object to be observed is recorded and output to the downstream module together with the perception results of the object to be observed, providing more auxiliary information on the observation results for the downstream processing module, enabling the corresponding module to perform special processing on the perception results obtained in the non-optimal observation state, thereby improving the accuracy of the autonomous driving perception system.
[0061] The preferred embodiments of this application will be described below with reference to the accompanying drawings of the specification. It should be understood that the preferred embodiments described herein are only for the purpose of illustration and explanation of this application, and are not used to limit this application. And without conflict, the embodiments in this application and the features in the embodiments can be combined with each other.
[0062] Refer to Figure 1 As shown, it is a schematic diagram of a possible application scenario in the embodiment of this application.
[0063] This application scenario includes terminal devices 110 (including terminal devices 1101, 1102... 110n) and a server 120. The terminal devices 110 and the server 120 can communicate with each other through a communication network.
[0064] In an alternative embodiment, the communication network may be a wired network or a wireless network. Therefore, the terminal device 110 and the server 120 can be directly or indirectly connected through wired or wireless communication means. For example, the terminal device 110 can be indirectly connected to the server 120 through a wireless access point, or the terminal device 110 can be directly connected to the server 120 through the Internet. This application does not make any restrictions here.
[0065] In the embodiments of this application, the terminal device 110 includes, but is not limited to, devices such as mobile phones, tablet computers, laptop computers, desktop computers, e-book readers, intelligent voice interaction devices, smart home appliances, in-vehicle terminals, etc.; various clients can be installed on the terminal device, and the client can be an application program that supports functions such as video preview and video playback (such as browsers, game software, etc.), or it can be a web page, a small program, etc.
[0066] The server 120 is a background server corresponding to the client installed in the terminal device 110. The server 120 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or it can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, Content Delivery Network (CDN), and big data and artificial intelligence platforms.
[0067] It should be noted that the memory parking positioning method in the embodiments of this application can be executed by a computing device, and the computing device can be the server 120 or the terminal device 110, that is, the method can be executed independently by the server 120 or the terminal device 110, or can be jointly executed by the server 120 and the terminal device 110.
[0068] It should be noted that in the following text, mainly the example of the server executing alone is used for illustration, and no specific limitation is made here.
[0069] It should be noted that Figure 1 The above is only an example, and actually the number of the terminal device 110 and the server 120 is not limited, and no specific limitation is made in the embodiments of this application.
[0070] In the embodiments of this application, when the number of servers 120 is multiple, the multiple servers 120 can form a blockchain, and the server 120 is a node on the blockchain.
[0071] Refer to Figure 2 As shown, it is a flowchart of a method for determining an observation state in the embodiments of this application. The following will be combined with the attached Figure 2, a detailed description of the specific operations performed:
[0072] Step S201: Obtain the perception results of at least one object to be observed corresponding to the target vehicle.
[0073] The perception results include the target detection boxes of at least one object to be observed.
[0074] Specifically, in the embodiments of the present application, the server obtains sensor signals through various sensors installed on the target vehicle and inputs the sensor signals into a preset perception algorithm to obtain the perception results of at least one object to be observed in the surrounding environment of the target vehicle.
[0075] Among them, the sensors include but are not limited to cameras, lidars, ultrasonic sensors, etc., and the corresponding sensor signals include but are not limited to images, videos, laser point clouds, ultrasounds, etc.
[0076] For example, the server obtains multiple images of the surrounding environment through multiple fisheye cameras installed around the target vehicle, stitches the multiple images through a BEV network to obtain a BEV image, and finally inputs the BEV image into the RTMDet model to obtain the detection results of the objects to be observed in the BEV image, that is, the perception results in the embodiments of the present application.
[0077] Refer to Figure 3 As shown, it is a schematic diagram of the first scenario of parking space observation in the embodiments of the present application. Among them, the object to be observed is a parking space. The server inputs the BEV image containing the parking space into the RTMDet model to obtain the detection results of the parking space area, including the overall frame of the parking space (with the parkability of the parking space). Points BCDE are the 4 inner corner points of the parking space respectively, and line segment BE is the entrance line of the parking space. Along the forward direction of the target vehicle, the first parking space corner point encountered by the target vehicle is point B, and the other entrance corner point is point E.
[0078] Further, after the server obtains the perception results of at least one object to be observed, the following operations are respectively performed for at least one object to be observed:
[0079] Step S202: Determine the preset target observation range for one object to be observed according to the type of one object to be observed.
[0080] Among them, the target observation range is the ideal observation area preset for one object to be observed.
[0081] It should be noted that the objects to be observed can be vehicles, pedestrians, cones, road markings, traffic signs, etc., and the present application does not limit this.
[0082] In the embodiments of the present application, when the server determines the target observation range, it includes at least the following two aspects:
[0083] (1) Determine the observation distance range starting from a preset key point on the target vehicle.
[0084] Specifically, in the embodiment of the present application, when obtaining the 2D perception result through the fisheye camera, the server determines the target observation distance range starting from the fisheye camera corresponding to the target vehicle according to the type of the object to be observed. Among them, the target observation distance range corresponding to the 2D perception result includes two observation distance ranges in the x direction and the y direction.
[0085] For example, as shown in Figure 3 Since the parking space of the object to be observed is on the right side of the target vehicle and the fisheye camera closest to it is the fisheye camera on the right side of the vehicle body, the server takes the right-side fisheye camera as the starting point and determines that the observation distance range in the x direction for the parking space type of the object to be observed is 0.5m - 3m, and the observation distance range in the y direction is 0m - 1m.
[0086] It should be noted that in the embodiment of the present application, the preset key point on the target vehicle is the installation position of the corresponding sensor closest to the object to be observed.
[0087] (2) Determine the observation angle range with the longitudinal direction of the target vehicle body as the reference direction.
[0088] Specifically, in the embodiment of the present application, in order to better determine the observation state of the object to be observed, a limit on the observation angle between the object to be observed and the longitudinal direction of the target vehicle itself is added.
[0089] For example, the server takes the longitudinal direction of the target vehicle body as the reference direction and determines that the observation angle range for the parking space type of the object to be observed is 0° - 20°.
[0090] It should be noted that the target observation ranges corresponding to different types of objects to be observed are different, and their values are prior values obtained by those skilled in the art according to experience or according to data statistics rules. The present application does not limit this.
[0091] Step S203: Determine the observation state of an object to be observed based on the perception result and the target observation range.
[0092] Specifically, referring to Figure 4 shown, which is a flowchart of a method for obtaining the observation state of an object to be observed in the embodiment of the present application. The following combines the appendix Figure 4 , and details the specific steps to be executed:
[0093] Step S401: Obtain the target distance between each point in the target detection frame and the preset key point on the target vehicle, and the target angle between the target direction where each point in the target detection frame is located and the longitudinal direction of the target vehicle body.
[0094] In the embodiments of the present application, the server calculates the target distance between each point and the corresponding sensor installation position based on the coordinates of each point in the target detection box corresponding to the object to be observed, and the target angle formed between the longitudinal direction of the vehicle body of the target vehicle and the line segment where each point is located.
[0095] Among them, as shown in Figure 3 When calculating the target angle between the line segments where the four corner points BCDE in the parking space frame are located and the longitudinal direction of the vehicle body, there are two line segments in different directions. It can be understood that the one with a smaller angle between the line segment and the longitudinal direction of the vehicle body needs to be selected as the target angle.
[0096] Step S402: Determine whether there is at least one point corresponding to the target distance and the target angle that respectively satisfy the observation distance range and the observation angle range. If so, execute Step S403; otherwise, execute Step S404.
[0097] Furthermore, in the embodiments of the present application, the server sequentially determines whether each point in the target detection box corresponding to the object to be observed satisfies the corresponding observation distance range and the observation angle range.
[0098] Step S403: Determine the observation state of an object to be observed as an ideal observation state.
[0099] If there is at least one point corresponding to the target distance and the target angle in the target detection box that respectively satisfy the corresponding observation distance range and the observation angle range at the same time, the server determines the observation state of the current object to be observed as an ideal observation state.
[0100] For example, as shown in Figure 3 Assume that the 2D coordinates of the midpoint of the BE line segment and the distance from the fisheye camera on the right side of the vehicle body are 2m in the x - direction distance and 0.8m in the y - direction distance, and the angle between the longitudinal direction of the vehicle body and the BE line segment is 15°, that is, the target distance and the target angle of the midpoint of the BE line segment respectively satisfy the corresponding observation distance range and the observation angle range.
[0101] In addition, assume that the fan - shaped area is the target observation range determined by the fisheye camera for the parking space. The points in the parking space frame that simultaneously satisfy the observation distance range and the observation angle range all fall within the fan - shaped area. Therefore, the server determines the observation state of the parking space as an ideal observation state.
[0102] Step S404: Determine the observation state of an object to be observed as a non - ideal observation state.
[0103] In another alternative embodiment, if there is no point corresponding to the target distance and the target angle in the target detection box that respectively satisfy the observation distance range and the observation angle range, the observation state of an object to be observed is determined as a non - ideal observation state.
[0104] For example, referring to Figure 5 shown in the figure, which is a schematic diagram of the second scenario of parking space observation in an embodiment of the present application. If no point in the parking space frame falls within the fan-shaped area, it means that there is no target distance and target angle corresponding to any point that simultaneously satisfy the observation distance range and the observation angle range. Therefore, the server determines that the observation state of the parking space is a non-ideal observation state.
[0105] Furthermore, in an embodiment of the present application, when the server obtains the observation state of an object to be observed based on the perception result, it will also obtain the observation result corresponding to the object to be observed based on the perception result.
[0106] Referring to Figure 6 shown in the figure, which is a flowchart of a method for obtaining the observation result corresponding to the observation object in an embodiment of the present application. The following will be described in detail with reference to the attached Figure 6 , and the specific steps will be described in detail:
[0107] Step S601: Obtain the target feature region of an object to be observed based on the type of the object to be observed.
[0108] Among them, the target feature region is the core concerned region of the object to be observed relative to the target vehicle.
[0109] Specifically, in an embodiment of the present application, the target feature regions of different types of objects to be observed are different. Among them, the target observation region of the parking space is the 2D coordinate of the midpoint of the entrance line.
[0110] It should be noted that the target feature region of the object to be observed is preset by those skilled in the art according to experience. The target feature region can be a point, or a line segment or a plane composed of multiple points. The present application does not limit this.
[0111] Step S602: Determine whether the target feature region is within the preset range from the target vehicle. If so, execute Step S603; otherwise, execute Step S604.
[0112] Since the sensor may not obtain the complete contour of the object to be observed, and when the server obtains the perception result based on the sensor information, it will adaptively complete the target detection frame corresponding to the object to be observed. Therefore, taking the example of obtaining the BEV image of the object to be observed through a fish-eye camera, the target feature region in the target detection frame of the object to be observed may not appear in the corresponding image. Therefore, the server limits the distance of the target feature region relative to the target vehicle based on the display range of the BEV image. The specific value depends on the performance of the fish-eye camera.
[0113] Step S603: Determine that the observation result of an object to be observed is that the target feature region is observed.
[0114] Specifically, if the coordinates of the target feature region in the target detection box and the distance from the target vehicle meet the preset range limit, that is, the target feature region appears in the corresponding BEV image, it indicates that the target feature region of the object to be observed is observed.
[0115] Step S604: Determine that the observation result of an object to be observed is that the target feature region is not observed.
[0116] If the coordinates of the target feature region in the target detection box and the distance from the target vehicle do not meet the preset range limit, that is, the target feature region does not appear in the corresponding BEV image, it indicates that the target feature region of the object to be observed is not observed.
[0117] Specifically, in the embodiments of the present application, the observation state and the observation result of the object to be observed are independent of each other. Therefore, the combination of the observation state and the observation result of an object to be observed includes at least any one of the following:
[0118] 1. The observation state is an ideal observation state, and the target feature region is observed;
[0119] 2. The observation state is an ideal observation state, and the target feature region is not observed;
[0120] 3. The observation state is a non-ideal observation state, and the target feature region is observed;
[0121] 4. The observation state is a non-ideal observation state, and the target feature region is not observed.
[0122] Combined with the attached Figure 3 As shown, the black border represents the display range of the BEV image, where the target feature region of the parking space is located within the image and within the corresponding target observation range. Therefore, Figure 3 The scene shown corresponds to the above combination 1; the attached Figure 5 As shown, the target feature region of the parking space is within the image but not within the corresponding target observation range. Therefore, Figure 5 The scene shown corresponds to the above combination 3.
[0123] In addition, referring to Figure 7 As shown, it is a schematic diagram of a third parking space observation scene in the embodiments of the present application. Among them, the target feature region of the parking space is not within the image, but some points in the target detection box of the parking space are within the corresponding target observation range. Therefore, Figure 7 The scene shown corresponds to the above combination 2; referring to Figure 8 As shown, it is a schematic diagram of a fourth parking space observation scene in the embodiments of the present application. Among them, the target feature region of the parking space is not in the image, and there is no point in the target detection of the parking space within the corresponding target observation range. Therefore, Figure 8The scene shown corresponds to the above combination 4.
[0124] The above embodiments will be further described in detail below through a specific application scenario.
[0125] Refer to Figure 9 As shown, it is a schematic diagram of a vehicle observation scenario in an embodiment of the present application. The server obtains various sensor signals installed on the target vehicle. Figure 9 Still taking the fish-eye camera obtaining a fish-eye image as an example, the server inputs the spliced BEV image into a preset SMOKE 3D object detection model to obtain a perception result including a 3D detection box of the vehicle to be observed.
[0126] Among them, the 3D detection box surrounding the vehicle to be observed is specifically (cx, cy, cz, W, H, I, sin(θ), cos(θ)), where cx, cy, and cz are the coordinates of the center point of the 3D detection box of the vehicle to be observed, and W, H, and I are the length, width, and height of the 3D detection box, and θ is the heading angle of the vehicle to be observed.
[0127] The server determines the target observation range for the vehicle to be observed based on the type of the object to be observed (vehicle), specifically including: the observation distance range starting from the installation position of the fish-eye camera on the right side of the target vehicle body is: 0.5m - 4m in the x direction, 0m - 4m in the y direction, and 0.1m - 1.5m in the z direction; the observation angle range between the longitudinal direction of the target vehicle body and the longitudinal direction of the vehicle to be observed includes the included angle ranges of the three plane projections of the xy plane, xz plane, and yz plane, which are respectively: αxy ∈ [60°, 150°], αxz ∈ [-10°, 10°], αyz ∈ [-30°, 30°].
[0128] At the same time, the server determines the target feature area of the vehicle to be observed as the head or tail area based on the type of the object to be observed. Here, it is specifically the 3D coordinates of the center point of the head area corresponding to the 3D detection box of the vehicle to be observed.
[0129] Refer to Figure 9 As shown, assuming that the vector length from the installation position of the fish-eye camera on the right side of the target vehicle body to the 3D coordinates of the center point of the head area of the vehicle to be observed is: 2m in the x direction, 3m in the y direction, and 1.2m in the z direction, and the included angles between the longitudinal direction of the target vehicle body and the longitudinal direction of the vehicle to be observed in the xy plane, xz plane, and yz plane projections are respectively: αxy = 90°, αxz = 0°, αyz = 0°, then the server determines that the observation state corresponding to the vehicle to be observed is an ideal observation state, and the observation result is that the target feature area is observed.
[0130] Furthermore, the server outputs the observation state and perception result of the vehicle to be observed to the downstream module for processing.
[0131] It should be noted that the above content is only an example for the processing method of the image perception result obtained by the fish-eye sensor. The processing methods for the perception results such as the laser point cloud obtained by other types of sensors such as lidar are similar to the above method flow and will not be elaborated here.
[0132] In addition, since the target vehicle itself is in a driving state and the object to be observed may be a vehicle in motion, in order to further improve the accuracy of the perception result, when the server obtains the perception result through sensor information, it can also record information such as the corresponding time series, and comprehensively process the multiple perception results corresponding to multiple time series.
[0133] In summary, in the embodiment of the present application, during the process of environmental perception by the perception system of the autonomous driving system, the observation state of the object to be observed is judged and recorded, as well as the observation result of the target feature area of the object to be observed, and they are transmitted to the downstream module together with the perception result, which can enable the downstream module to perform feature processing on the perception result in a non-ideal state, making the perception result referred to by the decision-making system more accurate, and during the observation process, the distance relationship and perspective relationship between the target vehicle and the object to be observed are considered, thereby further improving the accuracy of environmental perception.
[0134] Based on the same technical concept, referring to Figure 10 As shown, the embodiment of the present application further provides an observation state determination device, which includes:
[0135] An acquisition module 1001, configured to acquire the perception result of at least one object to be observed corresponding to the target vehicle, where the perception result includes the target detection frame of at least one object to be observed;
[0136] A processing module 1002, configured to perform the following operations for each of the at least one object to be observed:
[0137] Determine the preset target observation range for one object to be observed according to the type of one object to be observed, where the target observation range is the preset ideal observation area for one object to be observed;
[0138] Based on the perception result and the target observation range, determine the observation state of one object to be observed.
[0139] Optionally, when determining the preset target observation range for one object to be observed according to the type of one object to be observed, the processing module 1002 is configured to:
[0140] Determine the observation distance range starting from the preset key point on the target vehicle;
[0141] Determine the observation angle range with the longitudinal direction of the target vehicle body as the reference direction.
[0142] Optionally, when determining the observation state of an object to be observed based on the perception result and the target observation range, the processing module 1002 is configured to:
[0143] Obtain the target distances between the points in the target detection frame and the preset key points on the target vehicle, and the target angles between the target directions where the points in the target detection frame are located and the longitudinal direction of the target vehicle body;
[0144] If there is at least one point in the target detection frame whose corresponding target distance and target angle respectively satisfy the observation distance range and the observation angle range, determine that the observation state of an object to be observed is an ideal observation state;
[0145] If there is no point in the target detection frame whose corresponding target distance and target angle respectively satisfy the observation distance range and the observation angle range, determine that the observation state of an object to be observed is a non-ideal observation state.
[0146] Optionally, the processing module 1002 is further configured to:
[0147] Based on the type of an object to be observed, obtain the target feature region of an object to be observed, where the target feature region is the core attention region of the object to be observed relative to the target vehicle;
[0148] If the target feature region is not within the preset range from the target vehicle, it indicates that the target feature region of an object to be observed has not been observed;
[0149] If the target feature region is within the preset range from the target vehicle, it indicates that the target feature region of an object to be observed has been observed.
[0150] Optionally, the combination of the observation state and the observation result of an object to be observed includes at least any one of the following:
[0151] The observation state is an ideal observation state, and the target feature region is observed;
[0152] The observation state is an ideal observation state, and the target feature region is not observed;
[0153] The observation state is a non-ideal observation state, and the target feature region is observed;
[0154] The observation state is a non-ideal observation state, and the target feature region is not observed.
[0155] Based on the same technical concept, an embodiment of the present application further provides an electronic device, and this electronic device can implement the method flow for determining the observation state provided in the above embodiments of the present application.
[0156] In one embodiment, the electronic device may be a server, a terminal device, or other electronic devices.
[0157] Referring to Figure 11 as shown, the electronic device may include:
[0158] At least one processor 1101 and a memory 1102 connected to the at least one processor 1101. In the embodiments of the present application, the specific connection medium between the processor 1101 and the memory 1102 is not limited. Figure 11 In this example, the processor 1101 and the memory 1102 are connected through a bus 1100. The bus 1100 is Figure 11 represented by a thick line in this example. The connection manners between other components are only for illustrative purposes and are not limited thereto. The bus 1100 may be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, Figure 11 only a thick line is used to represent it in this example, but it does not mean that there is only one bus or one type of bus. Alternatively, the processor 1101 may also be referred to as a controller, and the name is not limited.
[0159] In the embodiments of the present application, the memory 1102 stores instructions executable by the at least one processor 1101. By executing the instructions stored in the memory 1102, the at least one processor 1101 can execute an observation state determination method described above. The processor 1101 can implement Figure 10 the functions of each module in the device shown.
[0160] Among them, the processor 1101 is the control center of the device, and can connect various parts of the entire control device through various interfaces and lines. By running or executing the instructions stored in the memory 1102 and calling the data stored in the memory 1102, various functions of the device and process data, so as to monitor the device as a whole.
[0161] In a possible design, the processor 1101 may include one or more processing units. The processor 1101 may integrate an application processor and a modem processor. Among them, the application processor mainly processes the operating system, user interface, application programs, etc., and the modem processor mainly processes wireless communication. It can be understood that the above modem processor may not be integrated into the processor 1101. In some embodiments, the processor 1101 and the memory 1102 may be implemented on the same chip, and in some embodiments, they may also be separately implemented on independent chips.
[0162] The processor 1101 may be a general-purpose processor, such as a CPU, a digital signal processor, an application-specific integrated circuit, a field-programmable gate array, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of an observation state determination method disclosed in combination with the embodiments of the present application may be directly embodied as being executed and completed by a hardware processor, or executed and completed by a combination of hardware and software modules in the processor.
[0163] The memory 1102, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. The memory 1102 may include at least one type of storage medium. For example, it may include flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (RAM), a static random access memory (SRAM), a programmable read-only memory (PROM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic memory, a magnetic disk, an optical disc, and so on. The memory 1102 is any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 1102 in the embodiments of the present application may also be a circuit or any other device capable of implementing a storage function, for storing program instructions and / or data.
[0164] By designing and programming the processor 1101, the code corresponding to the observation state determination method introduced in the foregoing embodiments can be solidified into the chip, so that the chip can execute Figure 2 the steps of the observation state determination method of the embodiment shown. How to design and program the processor 1101 is well-known to those skilled in the art and will not be elaborated here.
[0165] Based on the same inventive concept, the embodiments of the present application also provide a storage medium storing computer instructions, which, when run on a computer, cause the computer to execute an observation state determination method described above.
[0166] In some possible embodiments, various aspects of the method for determining an observation state provided in this application can also be implemented in the form of a program product, which includes program code. When the program product runs on a device, the program code is used to cause the control device to execute the steps in the method for determining an observation state according to various exemplary embodiments of this application described above in this specification.
[0167] It should be noted that although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of this application, the features and functions of the two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.
[0168] In addition, although the operations of the method of this application are described in a specific order in the drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step for execution, and / or one step may be decomposed into multiple steps for execution.
[0169] Those skilled in the art should understand that the embodiments of this application can be provided as a method, a system, or a computer program product. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0170] This application is described with reference to the flowcharts and / or block diagrams of the method, device (system), and computer program product according to this application. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate a device for implementing the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.
[0171] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce a manufacture including instruction means that implement the function specified in the flowchart(s) Figure 1 a flowchart or flowcharts and / or block(s) Figure 1 a block or blocks.
[0172] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing steps for implementing the function specified in the flowchart(s) Figure 1 a flowchart or flowcharts and / or block(s) Figure 1 a block or blocks.
[0173] It will be apparent to those skilled in the art that various modifications and variations can be made to the present application without departing from the spirit and scope of the application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application is also intended to cover these modifications and variations.
Claims
1. A method for determining an observation state, characterized in that Including: Obtain the perception results of at least one object to be observed corresponding to the target vehicle, where the perception results include the target detection frames of the at least one object to be observed; For each of the at least one object to be observed, perform the following operations respectively: Determine the preset target observation range for the one object to be observed according to the type of the one object to be observed, where the target observation range is the preset ideal observation area for the one object to be observed; Based on the perception results and the target observation range, determine the observation state of the one object to be observed.
2. The method according to claim 1, wherein Determining the preset target observation range for the one object to be observed according to the type of the one object to be observed includes: Determine the observation distance range starting from a preset key point on the target vehicle; Determine the observation angle range with the longitudinal direction of the target vehicle body as the reference direction.
3. The method according to claim 2, wherein The determining the observation state of the one object to be observed based on the perception results and the target observation range includes: Obtain the target distances between the points in the target detection frame and the preset key point on the target vehicle, and the target angles between the target directions where the points in the target detection frame are located and the longitudinal direction of the target vehicle body; If there is at least one point in the target detection frame whose corresponding target distance and target angle respectively satisfy the observation distance range and the observation angle range, determine that the observation state of the one object to be observed is the ideal observation state; If there is no point in the target detection frame whose corresponding target distance and target angle respectively satisfy the observation distance range and the observation angle range, determine that the observation state of the one object to be observed is the non-ideal observation state.
4. The method according to any one of claims 1 to 3, characterized in that, Further including: Based on the type of the one object to be observed, obtain the target feature area of the one object to be observed, where the target feature area is the core attention area of the object to be observed relative to the target vehicle; If the target feature area is not within the preset range from the target vehicle, it indicates that the target feature area of the one object to be observed has not been observed; If the target feature area is within the preset range from the target vehicle, it indicates that the target feature area of the one object to be observed has been observed.
5. The method according to any one of claims 1 to 3, characterized in that, The combination of the observation state and the observation result of the one object to be observed includes at least any one of the following: The observation state is the ideal observation state, and the target feature area is observed; The observation state is the ideal observation state, and the target feature area is not observed; The observation state is the non-ideal observation state, and the target feature area is observed; The observation state is the non-ideal observation state, and the target feature area is not observed.
6. An observation state determination device, characterized in that, Including: An acquisition module, configured to obtain the perception results of at least one object to be observed corresponding to the target vehicle, where the perception results include the target detection frames of the at least one object to be observed; A processing module, configured to perform the following operations respectively for each of the at least one object to be observed: Determine a preset target observation range for the one object to be observed according to the type of the one object to be observed, where the target observation range is an ideal observation area preset for the one object to be observed; Determine the observation state of the one object to be observed based on the perception result and the target observation range.
7. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, the method described in any one of claims 1-5 is implemented.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, the steps of the method described in any one of claims 1-5 are implemented.
9. A computer program product, characterized in that, When the computer program product is called by a computer, the computer is caused to execute the method described in any one of claims 1-5.