Method, apparatus and computer program product for position awareness of a target object
By combining object detection networks and 3D attribute prediction networks, the problem of converting 2D obstacles to 3D obstacles in autonomous driving was solved, and more accurate object location perception was achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING BAIDU NETCOM SCI & TECH CO LTD
- Filing Date
- 2023-12-20
- Publication Date
- 2026-07-21
Smart Images
Figure CN117746386B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of artificial intelligence technology, specifically to the field of machine vision technology, and particularly to a method, device, electronic device, storage medium, and computer program product for perceiving the position of a target object, which can be applied to autonomous driving scenarios. Background Technology
[0002] As research into autonomous driving technology deepens, the limitations of single-vehicle intelligent autonomous driving have become apparent. Simultaneously, considering factors such as the safety redundancy of vehicle-side sensors and sensor costs, vehicle-to-infrastructure (V2I) solutions have been re-emphasized. V2X (Vehicle-to-Everything) roadside perception is a crucial technology for achieving V2I. Perception information obtained from roadside sensors can be transmitted to autonomous vehicles via V2I (Vehicle-to-Infrastructure) communication. After extracting obstacles of interest from images using image recognition technology, a key step in V2X roadside perception technology is to reconstruct the detected 2D obstacles into 3D obstacles in a three-dimensional coordinate system. Summary of the Invention
[0003] This disclosure provides a method, apparatus, electronic device, storage medium, and computer program product for sensing the location of a target object.
[0004] According to the first aspect, a method for sensing the location of a target object is provided, comprising: determining the target object in the acquired roadside sensing image through a target detection network; predicting the three-dimensional attribute information of the target object through a three-dimensional attribute prediction network; determining the location determination method corresponding to the target object based on the three-dimensional attribute information; and determining the physical location information of the target object based on the three-dimensional attribute information using the location determination method.
[0005] According to a second aspect, a target object location sensing device is provided, comprising: a detection unit configured to determine a target object in an acquired roadside sensing image through a target detection network; a prediction unit configured to predict three-dimensional attribute information of the target object through a three-dimensional attribute prediction network; a first determination unit configured to determine a location determination method corresponding to the target object based on the three-dimensional attribute information; and a second determination unit configured to determine the physical location information of the target object based on the three-dimensional attribute information using the location determination method.
[0006] According to a third aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a method as described in any implementation of the first aspect.
[0007] According to a fourth aspect, a non-transitory computer-readable storage medium is provided that stores computer instructions for causing a computer to perform the method described in any implementation of the first aspect.
[0008] According to a fifth aspect, a computer program product is provided, comprising: a computer program that, when executed by a processor, implements the method as described in any implementation of the first aspect.
[0009] According to the technology disclosed herein, a method for sensing the position of a target object is provided, which combines a target detection network and a three-dimensional attribute prediction network to predict the three-dimensional attribute information of the target object; determines the position determination method corresponding to the target object based on the three-dimensional attribute information; and uses the position determination method to determine the physical position information of the target object, thereby using different position determination methods to determine the physical position information of different target objects, thus improving the accuracy of the determined physical position information.
[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description
[0011] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:
[0012] Figure 1 This is an exemplary system architecture diagram that can be applied to an embodiment of this disclosure;
[0013] Figure 2 This is a flowchart of an embodiment of the target object location sensing method according to the present disclosure;
[0014] Figure 3 This is a schematic diagram of the target detection network according to this embodiment;
[0015] Figure 4 This is a schematic diagram of the structure of the three-dimensional attribute prediction network according to this embodiment;
[0016] Figure 5 This is a schematic diagram illustrating the process of determining grounding point information according to this embodiment;
[0017] Figure 6 This is a schematic diagram illustrating an application scenario of the target object location awareness method according to this embodiment;
[0018] Figure 7 This is a flowchart of yet another embodiment of the target object location sensing method according to the present disclosure;
[0019] Figure 8 This is a structural diagram of one embodiment of the target object position sensing device according to the present disclosure;
[0020] Figure 9 This is a schematic diagram of the structure of a computer system suitable for implementing the embodiments of the present disclosure. Detailed Implementation
[0021] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.
[0022] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0023] Figure 1 An exemplary architecture 100 is shown for which the location-aware method and apparatus for the target object of this disclosure can be applied.
[0024] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. The communication connections between terminal devices 101, 102, and 103 form a network topology. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.
[0025] Terminal devices 101, 102, and 103 can be hardware or software that supports network connectivity for data interaction and processing. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices supporting network connectivity, information acquisition, interaction, display, and processing functions, including but not limited to image acquisition devices, in-vehicle computers, smartphones, tablets, e-book readers, laptops, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the aforementioned electronic devices. They can be implemented as multiple software programs or software modules, for example, to provide distributed services, or as a single software program or software module. No specific limitations are imposed here.
[0026] Server 105 can be a server that provides various services, such as acquiring roadside perception images collected by terminal devices 101, 102, and 103, combining them with a target detection network and a 3D attribute prediction network to predict the 3D attribute information of the target object; determining the location determination method corresponding to the target object based on the 3D attribute information; and using the location determination method to determine the physical location information of the target object. As an example, server 105 can be a cloud server.
[0027] It should be noted that a server can be either hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software programs or software modules (such as software programs or software modules used to provide distributed services), or as a single software program or software module. No specific limitations are made here.
[0028] It should also be noted that the target object location sensing method provided in the embodiments of this disclosure can be executed by a server, by a terminal device, or by a combination of both. Accordingly, the various parts (e.g., various units) of the target object location sensing device can be all located in the server, all located in the terminal device, or located in both the server and the terminal device.
[0029] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included. When the electronic device on which the target object's location awareness method runs does not need to transmit data with other electronic devices, the system architecture may only include the electronic device (e.g., a terminal device or server) on which the target object's location awareness method runs.
[0030] Please refer to Figure 2 , Figure 2 A flowchart illustrating a method for sensing the location of a target object according to an embodiment of this disclosure. Flowchart 200 includes the following steps:
[0031] Step 201: Use the target detection network to identify the target objects in the acquired roadside perception images.
[0032] In this embodiment, the execution body of the target object's position awareness method (e.g., Figure 1 The terminal device or server in the system can acquire roadside perception images remotely or locally through wired or wireless network connections, and determine the target objects in the acquired roadside perception images through a target detection network.
[0033] Roadside perception images are acquired by roadside perception devices (e.g., image acquisition devices installed on the roadside) and characterize the surrounding environment information at the location of the roadside perception device. For example, in an autonomous driving scenario, roadside perception images are driving environment images acquired by roadside perception devices installed on the road where the autonomous vehicle is operating.
[0034] As an example, the aforementioned execution entity can input roadside sensing images into a pre-trained object detection model, which then determines the target objects in the roadside sensing images and their location regions within the images. The object detection model is used to characterize the correspondence between the roadside sensing images, the target objects within them, and their location regions.
[0035] Object detection models can be single-stage or two-stage. Single-stage object detection models aim to directly output the detection box and category label of the target object from the roadside perception image. Examples include the YOLO (You Only Look Once) model, the SSD (Single Shot Multi-Box Detector) model, and the RetinaNet model.
[0036] Two-stage object detection models typically perform object detection in two stages.
[0037] Phase 1 (Candidate Box Generation): The model extracts image features using a feature extraction network. Then, a candidate box generation algorithm is used to generate a series of potential candidate bounding boxes, which may contain the target object. Examples of such algorithms include Selective Search (SS) or Region Proposal Network (RPN).
[0038] Phase Two (Object Classification and Bounding Box Regression): Candidate object boxes and their overlap with the ground truth bounding boxes (i.e., IoU values) are input into the classification network for object classification and bounding box regression. The classification network determines whether each candidate object box is foreground or background (usually using a softmax activation function) and assigns a class label to foreground objects (such as people, vehicles, etc.). The bounding box regression network fine-tunes the position of the candidate object boxes to more accurately locate them.
[0039] Two-stage object detection models include Faster R-CNN (Region Convolutional Neural Networks), R-CNN (Region Convolutional Neural Networks), and Mask R-CNN (Mask Region Convolutional Neural Networks).
[0040] Continue to refer to Figure 3 The diagram shows a schematic of the target detection network.
[0041] In some optional implementations of this embodiment, the execution entity can perform step 201 as follows: determine the category, detection box, and feature data of the target object in the roadside perception image through the target detection network.
[0042] As an example, the aforementioned execution entity can input roadside sensing images into a target detection network, which determines the detection box for each target object in the roadside sensing image, and then determines the category and feature data of the target object in each detection box.
[0043] In this implementation, the target object category, detection box, and feature data are determined based on the target detection network, which provides rich input data for the subsequent 3D attribute prediction network and helps to improve the prediction accuracy of 3D attribute information.
[0044] In some optional implementations of this embodiment, the execution entity can perform the process of determining the category, detection box, and feature data in the following manner:
[0045] First, feature extraction is performed on the roadside perception image through the feature extraction subnetwork in the object detection network to obtain image features.
[0046] Continue to refer to Figure 3 The object detection network 300 includes a feature extraction subnetwork 301 and an object detection subnetwork 302.
[0047] The feature extraction subnetwork 301 serves as the backbone network in the object detection network, used to extract image features from roadside perception images. Examples of feature extraction subnetworks include residual networks, convolutional neural networks, and recurrent neural networks.
[0048] Then, through the target detection subnetwork in the target detection network, target objects in the roadside perception image are detected at different feature scales based on image features, resulting in multiple candidate detection boxes.
[0049] The object detection sub-network 302 can employ a feature pyramid network. Based on image features, the feature pyramid network detects target objects in the roadside perception image at different feature scales, obtaining multiple candidate detection boxes.
[0050] Then, duplicate candidate detection boxes are removed to determine the final detection box.
[0051] As an example, the aforementioned execution entity can use the NMS (Non-Maximum Suppression) method to deduplicate multiple candidate detection boxes. Specifically, the NMS method's processing procedure is as follows:
[0052] 1. Identify multiple candidate bounding boxes in the roadside sensing image. Each candidate bounding box includes a bounding box (a rectangle surrounding the target object) and its corresponding confidence score.
[0053] 2. Based on actual application requirements, set a confidence threshold and filter out all candidate detection boxes with confidence scores lower than the confidence threshold to remove unreliable prediction results.
[0054] 3. For the remaining candidate detection boxes, sort them according to their confidence scores, usually in descending order of scores.
[0055] 4. Starting with the candidate detection box with the highest score, iterate through each candidate detection box sequentially. For the current candidate detection box, check its overlap with subsequent candidate detection boxes. The overlap is generally represented by IoU (Intersection over Union).
[0056] 5. If the overlap between two candidate detection boxes exceeds the set overlap threshold, delete the candidate detection box with the lower score. Otherwise, retain the current candidate detection box.
[0057] By repeating steps 4 and 5 above, until all candidate detection boxes have been traversed, the remaining candidate detection boxes are determined as the deduplicated detection boxes.
[0058] Finally, the category and feature data of the target object corresponding to the detection box are determined.
[0059] For each detection box after deduplication, the aforementioned execution entity can determine the category and feature data of the target object within it.
[0060] This implementation provides the specific structure of the target detection network and the data processing procedure based on the target detection network, which helps to improve the accuracy of the obtained categories, detection boxes, and feature data.
[0061] Step 202: Predict the three-dimensional attribute information of the target object using a three-dimensional attribute prediction network.
[0062] In this embodiment, the aforementioned execution entity can predict the three-dimensional attribute information of the target object through a three-dimensional attribute prediction network.
[0063] The 3D attribute information includes, but is not limited to, the length, width, and height of the target object; key point information such as the vertices and the center point of the bottom surface of the target object; the orientation angle information of the target object; and whether the target object is occluded or truncated. Occlusion indicates that part of the target object is obscured by other objects in the roadside perception image; truncation indicates that part of the target object extends beyond the edge of the roadside perception image and is not presented in the roadside perception image.
[0064] As an example, the aforementioned execution entity can input the roadside perception image, the target object in the roadside perception image, and the location region of the target object in the roadside perception image into a pre-trained 3D attribute prediction model. The 3D attribute prediction model extracts features from each target object in the roadside perception image based on the target object and the location region of the target object in the roadside perception image, and predicts 3D attribute information based on the extracted features to obtain the 3D attribute information of each target object.
[0065] A 3D attribute prediction network can be a unified 3D attribute prediction network with the ability to predict various 3D attribute information of the target object; or it can be a 3D attribute prediction network composed of multiple sub-models, each of which has the ability to predict one 3D attribute information of the target object.
[0066] In some optional implementations of this embodiment, the execution entity can perform step 202 as follows: predict the three-dimensional attribute information of the target object in the roadside perception image based on the category, detection box and feature data through a three-dimensional attribute prediction network.
[0067] In this implementation, for each target object in the roadside perception image, the three-dimensional attribute information of the target object is predicted based on the corresponding category, detection box, and feature data. Based on rich input data, the prediction accuracy of the three-dimensional attribute information is improved.
[0068] In some optional implementations of this embodiment, the execution entity can perform the prediction process of three-dimensional attribute information in the following manner:
[0069] By using multiple attribute prediction heads in a 3D attribute prediction network, the 3D attribute information corresponding to each attribute prediction head is predicted based on the category, detection box, and feature data.
[0070] Continue to refer to Figure 4The diagram illustrates the structure of a three-dimensional attribute prediction network. The three-dimensional attribute prediction network 400 includes a feature scale unification module 401, a feature extraction module 402, and multiple attribute prediction heads 403.
[0071] The feature scale unification module 401 is used to scale features of different sizes to a uniform scale using the ROI Align (Region of Interest Align) operation; the feature extraction module 402 includes two cascaded bottleneck structures for further processing of the unified scale features; each attribute prediction head 403 is used to predict the corresponding three-dimensional attribute information based on the processed features.
[0072] This implementation provides the specific structure of the 3D attribute prediction network and the processing procedure based on the 3D attribute prediction network, which further improves the prediction accuracy of 3D attribute information.
[0073] Step 203: Determine the location of the target object based on the three-dimensional attribute information.
[0074] In this embodiment, the aforementioned execution entity can determine the location determination method corresponding to the target object based on the three-dimensional attribute information.
[0075] Roadside sensing images typically include multiple target objects. For each target object in the roadside sensing image, the aforementioned execution entity can determine the location determination method corresponding to that target object based on its three-dimensional attribute information.
[0076] As an example, the aforementioned execution entity can determine the target key points that best represent the target object based on the three-dimensional attribute information of the target object, and then determine a location determination method suitable for the target key points, so as to determine the physical location information of the target key points based on the location determination method, and use it as the physical location information of the target object.
[0077] In some optional implementations of this embodiment, the execution entity can perform step 203 as follows:
[0078] First, based on the three-dimensional attribute information, determine the target type of the target object under the preset type classification standard.
[0079] The preset type classification criteria can be set according to the actual situation. For example, the preset type classification criteria can divide the target into the first preset type, the second preset type, and the third preset type.
[0080] The first preset type represents a target object with relatively obvious and regular vertex features. For example, motor vehicles, electric tricycles, and four-wheeled vehicles can be approximated as cuboids with relatively obvious eight-vertex features. The second preset type represents a target object that is occluded and / or truncated. The third preset type represents a target object that is a small-volume object and generally does not have relatively obvious and regular vertex features, such as traffic cones, triangular signs, and pedestrians.
[0081] For each target object in the roadside perception image, the aforementioned execution entity can analyze the three-dimensional attribute information of the target object, and then determine the target type of the target object under the preset type classification criteria.
[0082] Second, determine the location determination method corresponding to the target object based on the target type.
[0083] As an example, the aforementioned execution entity or the electronic device communicatively connected to the aforementioned execution entity may pre-set a correspondence between target type and location determination method. Consequently, the aforementioned execution entity may determine a location determination method applicable to the target object based on the target type of the target object under a preset type classification standard.
[0084] This implementation provides a specific method for determining the type of a target object based on a preset type classification standard, and then determining the appropriate location method for the target object based on the type. This improves the compatibility between the location determination method and the target object, and helps to improve the accuracy of the physical location information of the target object determined by the location determination method.
[0085] In some optional implementations of this embodiment, the execution entity can perform the second step as follows:
[0086] In response to determining that the target type is the first preset type, the method for determining the location of the target object is the first location determination method.
[0087] The first position determination method is characterized by determining the physical position information of the target object's vertices based on the vertex information in the image coordinate system of the ground equation and three-dimensional attribute information.
[0088] The ground equation, also known as the ground plane equation, can be understood as follows: for each point [Xi, Yi, Zi] in the camera coordinate system, it is mapped to pixel coordinates [ui, vi, 1] in the image coordinate system through the intrinsic parameters, extrinsic parameters, and perspective projection of the image acquisition device. Simultaneously, the ground equation describes the geometric properties of the ground through parameters a, b, c, and d. When a point is located on the ground, substituting these parameters into the ground equation should satisfy the following equations.
[0089] By solving the ground equation, we can obtain the point [Xi, Yi, Zi] in the camera coordinate system corresponding to the point [ui, vi, 1] in the image coordinate system; or conversely, we can calculate the pixel position [ui, vi, 1] in the image coordinate system using the known point [Xi, Yi, Zi] in the camera coordinate system.
[0090] For each vertex information in the image coordinate system, the vertex coordinates are substituted into the ground equation to calculate the physical location information of that vertex information in the camera coordinate system.
[0091] In this implementation, for a target object of the first preset type, the method for determining its position is to determine the physical position information of the target object's vertices based on the vertex information in the image coordinate system of the ground equation and the three-dimensional attribute information, which helps to improve the accuracy of determining the physical position information of the target object of the first preset type.
[0092] In some optional implementations of this embodiment, the execution entity can perform the second step as follows:
[0093] In response to determining that the target type is the second preset type, the method for determining the location of the target object is the second location determination method.
[0094] The second location determination method involves determining the physical location of the target object's bottom center point based on the ground equation and the bottom center point information in the image coordinate system from the three-dimensional attribute information. The bottom surface of the target object is generally the side that contacts the ground.
[0095] For the coordinates of the bottom center point represented by the bottom center point information in the image coordinate system, the bottom center point information is substituted into the ground equation to calculate the physical position information of the bottom center point information in the camera coordinate system.
[0096] In this implementation, for target objects of the second preset type, the method for determining their location is to determine the physical location information of the bottom center point of the target object based on the ground equation and the bottom center point information in the image coordinate system of the three-dimensional attribute information. This helps to improve the accuracy of determining the physical location information of target objects of the second preset type.
[0097] In some optional implementations of this embodiment, the execution entity can perform the second step as follows:
[0098] In response to determining that the target type is the third preset type, the method for determining the location of the target object is the third location determination method.
[0099] The third location determination method is characterized by determining the physical location information of the grounding point of the target object based on the ground equation and the grounding point information in the image coordinate system of the three-dimensional attribute information. The grounding point is the point on the target object that is closest to the acquisition device of the roadside sensing image.
[0100] For the grounding point coordinates represented by the grounding point information in the image coordinate system, the grounding point coordinates are substituted into the ground equation to calculate the physical location information of the grounding point in the camera coordinate system.
[0101] In this implementation, for target objects of the third preset type, the method for determining their location is to determine the physical location information of the grounding point of the target object based on the ground equation and the grounding point information in the image coordinate system of the three-dimensional attribute information. This helps to improve the accuracy of determining the physical location information of target objects of the third preset type.
[0102] Step 204: Using a location determination method, determine the physical location information of the target object based on the three-dimensional attribute information.
[0103] In this embodiment, the aforementioned execution entity can use a location determination method to determine the physical location information of the target object based on three-dimensional attribute information.
[0104] Roadside sensing images typically include multiple target objects. For each target object in the roadside sensing image, the aforementioned execution entity can use the location determination method corresponding to that target object to determine the physical location information of the target object based on its three-dimensional attribute information.
[0105] After determining the physical location information of the target object in the roadside perception image collected by the roadside perception device, the aforementioned execution entity can fuse the roadside perception results obtained based on the roadside perception image and the perception results obtained based on the autonomous driving image collected by the autonomous driving vehicle.
[0106] In some optional implementations of this embodiment, for a target object of the first preset type, the execution entity can perform step 204 as follows:
[0107] First, the vertex information of multiple vertices of the target object in the image coordinate system is determined from the three-dimensional attribute information.
[0108] For a target object of the first preset type, the three-dimensional attribute information obtained by the execution subject through step 202 includes vertex information of multiple vertices of the target object in the image coordinate system.
[0109] Then, for each of the multiple vertices, the physical location information of that vertex is determined based on the ground equation and the vertex information of that vertex.
[0110] As an example, the ground equation is as follows:
[0111]
[0112] Where K is the pre-calibrated intrinsic parameter matrix of the image acquisition device, with a size of 3×3. X i ,Y i Z i It is the three-dimensional coordinate of the vertex in the camera coordinate system, u i ,v i λ represents the coordinates of the vertex in the image coordinate system, and λ is a coefficient. a, b, c, and d are the coefficients of the ground equation expressed in the camera coordinate system.
[0113] For each of the multiple vertices, the aforementioned execution entity can substitute the vertex information (coordinates in the image coordinate system) of that vertex into the ground equation to determine the physical location information (three-dimensional coordinates in the camera coordinate system) of that vertex.
[0114] In this implementation, for target objects of the first preset type, a specific implementation method is provided to determine the physical location information of the vertices through ground equations as the physical location information of the target object, thereby improving the accuracy of the determined physical location information of the target object.
[0115] In some optional implementations of this embodiment, for a target object of the second preset type, the execution entity can perform step 204 as follows:
[0116] First, the center point of the bottom surface of the target object in the image coordinate system is determined from the three-dimensional attribute information.
[0117] For the target object of the second preset type, the three-dimensional attribute information obtained by the execution subject through the above step 202 includes the bottom center point information of the target object in the image coordinate system.
[0118] Then, based on the ground equation and the information of the center point of the bottom surface, the physical location information of the center point of the bottom surface is determined.
[0119] As an example, the aforementioned execution entity can substitute the information of the bottom center point (the coordinates of the bottom center point in the image coordinate system) into the ground equation to determine the physical location information of the bottom center point (three-dimensional coordinates in the camera coordinate system).
[0120] In this implementation, for target objects of the second preset type, a specific implementation method is provided to determine the physical location information of the bottom center point through ground equations, which is used as the physical location information of the target object, thereby improving the accuracy of the determined physical location information of the target object.
[0121] In some optional implementations of this embodiment, for a target object of the first preset type, the execution entity can perform step 204 as follows:
[0122] First, based on the three-dimensional attribute information, determine the grounding point information of the target object in the image coordinate system.
[0123] For the target object of the third preset type, the three-dimensional attribute information obtained by the above-mentioned execution subject through the above-mentioned step 202 includes the grounding point information of the target object in the image coordinate system.
[0124] Then, based on the ground equation and grounding point information, the physical location information of the grounding point is determined.
[0125] As an example, the aforementioned execution entity can substitute the grounding point information (the coordinates of the grounding point in the image coordinate system) into the ground equation to determine the physical location information of the grounding point (three-dimensional coordinates in the camera coordinate system).
[0126] In this implementation, for the target object of the third preset type, a specific implementation method is provided to determine the physical location information of the grounding point through ground equations as the physical location information of the target object, thereby improving the accuracy of the determined physical location information of the target object.
[0127] In some optional implementations of this embodiment, the execution entity can determine the grounding point information of the target object in the image coordinate system based on the three-dimensional attribute information in the following manner:
[0128] Based on the size and orientation angle information of the target object in the three-dimensional attribute information, the grounding point information in the image coordinate system is determined.
[0129] Continue to refer to Figure 5 The diagram illustrates the process of determining grounding point information. A and C represent the lower left and lower right endpoints of the detection bounding box of the target object in the roadside sensing image, respectively, and B is the grounding point.
[0130] The calculation process for grounding point information is as follows:
[0131] Assumption: B→b(X) b Z b )
[0132] Given: ∠boZ=θ, ∠ebf=α, the length of the target object is L, the width is W, A→a(X a Z a )
[0133] Then we can obtain: ∠ebc=ry, ry=α-θ, where ∠ebc is the orientation angle information.
[0134] Therefore, grounding point B can be obtained:
[0135]
[0136] This implementation provides a specific method for determining grounding point information, which improves the accuracy of grounding point information determination.
[0137] See also Figure 6 , Figure 6 This is a schematic diagram 600 illustrating an application scenario of the target object location sensing method according to this embodiment. Figure 6 In the application scenario, the roadside sensing device 601 collects surrounding environmental information in real time, obtains a roadside sensing image 602, and uploads the roadside sensing image to the server 603. The server 603 first uses a target detection network 604 to determine the target object in the acquired roadside sensing image; then, it uses a 3D attribute prediction network 605 to predict the 3D attribute information of the target object; then, it determines the location determination method corresponding to the target object based on the 3D attribute information; finally, it uses the location determination method to determine the physical location information of the target object based on the 3D attribute information.
[0138] In this embodiment, a target object location perception method is provided. The method combines a target detection network and a 3D attribute prediction network to predict the 3D attribute information of the target object. Based on the 3D attribute information, a location determination method corresponding to the target object is determined. The location determination method is then used to determine the physical location information of the target object. Thus, different location determination methods are used to determine the physical location information of different target objects, thereby improving the accuracy of the determined physical location information.
[0139] Continue to refer to Figure 7 This illustrates a schematic flow 700 of yet another embodiment of the target object location-aware method according to the present disclosure. Flow 700 includes the following steps:
[0140] Step 701: Determine the category, detection box, and feature data of the target object in the roadside perception image through the target detection network.
[0141] Step 702: Using a 3D attribute prediction network, predict the 3D attribute information of the target object in the roadside perception image based on the category, detection box, and feature data.
[0142] Step 703: Determine the target type of the target object under the preset type classification standard based on the three-dimensional attribute information.
[0143] Step 704: In response to determining the target type as the first preset type, determine the location determination method corresponding to the target object as the first location determination method.
[0144] The first position determination method is characterized by determining the physical position information of the target object's vertices based on the vertex information in the image coordinate system of the ground equation and three-dimensional attribute information.
[0145] Step 705: Determine the vertex information of multiple vertices of the target object in the image coordinate system from the three-dimensional attribute information.
[0146] Step 706: For each of the multiple vertices, determine the physical location information of the vertex based on the ground equation and the vertex information of that vertex.
[0147] Step 707: In response to determining that the target type is the second preset type, the location determination method corresponding to the target object is determined to be the second location determination method.
[0148] The second position determination method is characterized by determining the physical position information of the bottom center point of the target object based on the ground equation and the bottom center point information in the image coordinate system of the three-dimensional attribute information.
[0149] Step 708: Determine the bottom center point information of the target object in the image coordinate system from the three-dimensional attribute information.
[0150] Step 709: Determine the physical location information of the bottom center point based on the ground equation and the bottom center point information.
[0151] Step 710: In response to determining that the target type is the third preset type, the location determination method corresponding to the target object is determined to be the third location determination method.
[0152] The third location determination method is characterized by determining the physical location information of the grounding point of the target object based on the ground equation and the grounding point information in the image coordinate system of the three-dimensional attribute information. The grounding point is the point on the target object that is closest to the acquisition device of the roadside sensing image.
[0153] Step 711: Determine the grounding point information in the image coordinate system based on the size information and orientation angle information of the target object in the three-dimensional attribute information.
[0154] Step 712: Determine the physical location information of the grounding point based on the ground equation and grounding point information.
[0155] As can be seen from this embodiment, with Figure 2 Compared to the corresponding embodiments, the flow 700 of the target object location perception method in this embodiment specifically illustrates the process of determining the physical location information of various types of target objects, further improving the accuracy of the physical location information.
[0156] Continue to refer to Figure 8 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a target object position sensing device, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.
[0157] like Figure 8 As shown, the target object location sensing device 800 includes: a detection unit 801 configured to determine a target object in an acquired roadside sensing image through a target detection network; a prediction unit 802 configured to predict the three-dimensional attribute information of the target object through a three-dimensional attribute prediction network; a first determination unit 803 configured to determine a location determination method corresponding to the target object based on the three-dimensional attribute information; and a second determination unit 804 configured to determine the physical location information of the target object based on the three-dimensional attribute information using the location determination method.
[0158] In some optional implementations of this embodiment, the first determining unit 803 is further configured to: determine the target type of the target object under a preset type classification standard based on the three-dimensional attribute information; and determine the location determination method corresponding to the target object based on the target type.
[0159] In some optional implementations of this embodiment, the first determining unit 803 is further configured to: in response to determining the target type as a first preset type, determine the position determination method corresponding to the target object as a first position determination method, wherein the first position determination method represents determining the physical position information of the vertex of the target object based on the vertex information in the image coordinate system of the ground equation and the three-dimensional attribute information.
[0160] In some optional implementations of this embodiment, the second determining unit 804 is further configured to: determine the vertex information of multiple vertices of the target object in the image coordinate system from the three-dimensional attribute information; and for each of the multiple vertices, determine the physical position information of the vertex according to the ground equation and the vertex information of the vertex.
[0161] In some optional implementations of this embodiment, the first determining unit 803 is further configured to: in response to determining the target type as a second preset type, determine the position determination method corresponding to the target object as a second position determination method, wherein the second position determination method represents determining the physical position information of the bottom center point of the target object based on the bottom center point information in the image coordinate system of the ground equation and the three-dimensional attribute information.
[0162] In some optional implementations of this embodiment, the second determining unit 804 is further configured to: determine the bottom center point information of the target object in the image coordinate system from the three-dimensional attribute information; and determine the physical location information of the bottom center point based on the ground equation and the bottom center point information.
[0163] In some optional implementations of this embodiment, the first determining unit 803 is further configured to: in response to determining the target type as a third preset type, determine the location determination method corresponding to the target object as a third location determination method, wherein the third location determination method represents that, based on the ground equation and the ground point information in the image coordinate system of the three-dimensional attribute information, the physical location information of the ground point of the target object is determined, and the ground point is the point on the target object that is closest to the acquisition device of the roadside sensing image.
[0164] In some optional implementations of this embodiment, the second determining unit 804 is further configured to: determine the grounding point information of the grounding point on the target object in the image coordinate system based on the three-dimensional attribute information; and determine the physical location information of the grounding point based on the ground equation and the grounding point information.
[0165] In some optional implementations of this embodiment, the second determining unit 804 is further configured to: determine the grounding point information in the image coordinate system based on the size information and orientation angle information of the target object in the three-dimensional attribute information.
[0166] In some optional implementations of this embodiment, the detection unit 801 is further configured to: determine the category, detection box, and feature data of the target object in the roadside perception image through a target detection network.
[0167] In some optional implementations of this embodiment, the detection unit 801 is further configured to: extract features from the roadside perception image through the feature extraction subnetwork in the target detection network to obtain image features; detect target objects in the roadside perception image at different feature scales based on the image features through the target detection subnetwork in the target detection network to obtain multiple candidate detection boxes; deduplicate the multiple candidate detection boxes to determine the detection box; and determine the category and feature data of the target object corresponding to the detection box.
[0168] In some optional implementations of this embodiment, the prediction unit 802 is further configured to: predict the three-dimensional attribute information of the target object in the roadside perception image based on the category, detection box and feature data through a three-dimensional attribute prediction network.
[0169] In some optional implementations of this embodiment, the prediction unit 802 is further configured to: predict the three-dimensional attribute information corresponding to each of the multiple attribute prediction heads in the three-dimensional attribute prediction network based on the category, detection box and feature data.
[0170] In this embodiment, a target object position sensing device is provided. It combines a target detection network and a three-dimensional attribute prediction network to predict the three-dimensional attribute information of the target object; determines the position determination method corresponding to the target object based on the three-dimensional attribute information; and uses the position determination method to determine the physical position information of the target object. Thus, different position determination methods are used to determine the physical position information of different target objects, thereby improving the accuracy of the determined physical position information.
[0171] According to embodiments of this disclosure, this disclosure also provides an electronic device, the electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to implement the target object location sensing method described in any of the above embodiments.
[0172] According to embodiments of this disclosure, this disclosure also provides a readable storage medium storing computer instructions that, when executed by a computer, enable the implementation of the target object location awareness method described in any of the above embodiments.
[0173] This disclosure provides a computer program product that, when executed by a processor, can implement the target object location awareness method described in any of the above embodiments.
[0174] Figure 9 A schematic block diagram of an example electronic device 900 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.
[0175] like Figure 9As shown, device 900 includes a computing unit 901, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 902 or a computer program loaded from storage unit 908 into random access memory (RAM) 903. RAM 903 may also store various programs and data required for the operation of device 900. The computing unit 901, ROM 902, and RAM 903 are interconnected via bus 904. Input / output (I / O) interface 905 is also connected to bus 904.
[0176] Multiple components in device 900 are connected to I / O interface 905, including: input unit 906, such as keyboard, mouse, etc.; output unit 907, such as various types of monitors, speakers, etc.; storage unit 908, such as disk, optical disk, etc.; and communication unit 909, such as network card, modem, wireless transceiver, etc. Communication unit 909 allows device 900 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.
[0177] The computing unit 901 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above, such as the target object location awareness method. For example, in some embodiments, the target object location awareness method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed on device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by the computing unit 901, one or more steps of the target object location awareness method described above can be performed. Alternatively, in other embodiments, the computing unit 901 can be configured to perform the target object location awareness method by any other suitable means (e.g., by means of firmware).
[0178] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.
[0179] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.
[0180] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0181] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).
[0182] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.
[0183] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, also known as cloud computing servers or cloud hosts, which are hosting products within the cloud computing service system to address the management difficulties and weak business scalability inherent in traditional physical hosts and Virtual Private Servers (VPS) services; they can also be servers for distributed systems or servers integrated with blockchain technology.
[0184] According to the technical solution of the embodiments of this disclosure, a method for sensing the position of a target object is provided. The method combines a target detection network and a three-dimensional attribute prediction network to predict the three-dimensional attribute information of the target object; a position determination method corresponding to the target object is determined based on the three-dimensional attribute information; and the position determination method is used to determine the physical position information of the target object. Thus, different position determination methods are used to determine the physical position information of different target objects, thereby improving the accuracy of the determined physical position information.
[0185] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution provided in this disclosure can be achieved, and this is not limited herein.
[0186] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.
Claims
1. A method for position-aware targeting of a target object, comprising: The target objects in the acquired roadside perception images are identified through the target detection network. The three-dimensional attribute information of the target object is predicted using a three-dimensional attribute prediction network. Based on the three-dimensional attribute information, the target type of the target object under the preset type classification standard is determined; The method for determining the location of the target object according to the target type includes: in response to determining that the target type is a second preset type representing that the target object is occluded and / or truncated, determining the location determination method corresponding to the target object as a second location determination method, wherein the second location determination method represents determining the physical location information of the bottom center point of the target object according to the ground equation and the bottom center point information in the image coordinate system of the three-dimensional attribute information; in response to determining that the target type is a third preset type representing that the target object does not have regular vertex features, determining the location determination method corresponding to the target object as a third location determination method, wherein the third location determination method represents determining the physical location information of the ground point of the target object according to the ground equation and the ground point information in the image coordinate system of the three-dimensional attribute information, wherein the ground point is the point on the target object closest to the acquisition device of the roadside sensing image; Using the aforementioned location determination method, the physical location information of the target object is determined based on the three-dimensional attribute information.
2. The method according to claim 1, wherein, The method for determining the location of the target object based on the target type further includes: In response to determining that the target type is a first preset type that characterizes the target object as having regular vertex features, the method for determining the position of the target object is a first position determination method, wherein the first position determination method characterizes the determination of the physical position information of the vertex of the target object based on the ground equation and the vertex information in the image coordinate system of the three-dimensional attribute information.
3. The method according to claim 2, wherein, The method of determining the location of the target object using the three-dimensional attribute information includes: In response to the target type being the first preset type, vertex information of multiple vertices of the target object in the image coordinate system is determined from the three-dimensional attribute information; For each of the plurality of vertices, the physical location information of the vertex is determined based on the ground equation and the vertex information of that vertex.
4. The method according to claim 1, wherein, The method of determining the location of the target object using the three-dimensional attribute information includes: In response to the target type being the second preset type, the bottom center point information of the target object in the image coordinate system is determined from the three-dimensional attribute information; Based on the ground equation and the bottom center point information, the physical location information of the bottom center point is determined.
5. The method according to claim 1, wherein, The method of determining the location of the target object using the three-dimensional attribute information includes: In response to the target type being the third preset type, the grounding point information of the grounding point on the target object in the image coordinate system is determined based on the three-dimensional attribute information; Based on the ground equation and the grounding point information, the physical location information of the grounding point is determined.
6. The method according to claim 5, wherein, The step of determining the grounding point information of the grounding point on the target object in the image coordinate system based on the three-dimensional attribute information includes: Based on the size information and orientation angle information of the target object in the three-dimensional attribute information, the grounding point information in the image coordinate system is determined.
7. The method according to claim 1, wherein, The step of determining the target object in the acquired roadside perception image through the target detection network includes: The target detection network is used to determine the category, detection box, and feature data of the target object in the roadside perception image.
8. The method according to claim 7, wherein, The step of determining the category, detection box, and feature data of the target object in the roadside perception image through the target detection network includes: The feature extraction subnetwork in the target detection network is used to extract features from the roadside perception image to obtain image features; The target detection subnetwork in the target detection network detects target objects in the roadside perception image at different feature scales based on the image features, and obtains multiple candidate detection boxes. The candidate detection boxes are deduplicated to determine the detection boxes; Determine the category and feature data of the target object corresponding to the detection box.
9. The method according to claim 7, wherein, The step of predicting the three-dimensional attribute information of the target object through a three-dimensional attribute prediction network includes: A 3D attribute prediction network is used to predict the 3D attribute information of target objects in the roadside perception image based on the category, the detection box, and the feature data.
10. The method according to claim 9, wherein, The method of predicting the three-dimensional attribute information of the target object in the roadside perception image through a three-dimensional attribute prediction network based on the category, the detection box, and the feature data includes: Using multiple attribute prediction heads in the 3D attribute prediction network, the 3D attribute information corresponding to each of the multiple attribute prediction heads is predicted based on the category, the detection box, and the feature data.
11. A location sensing device for a target object, comprising: The detection unit is configured to identify target objects in the acquired roadside perception images through a target detection network; The prediction unit is configured to predict the three-dimensional attribute information of the target object through a three-dimensional attribute prediction network; The first determining unit is configured to determine the target type of the target object under a preset type classification standard based on the three-dimensional attribute information; The method for determining the location of the target object according to the target type includes: in response to determining that the target type is a second preset type representing that the target object is occluded and / or truncated, determining the location determination method corresponding to the target object as a second location determination method, wherein the second location determination method represents determining the physical location information of the bottom center point of the target object according to the ground equation and the bottom center point information in the image coordinate system of the three-dimensional attribute information; in response to determining that the target type is a third preset type representing that the target object does not have regular vertex features, determining the location determination method corresponding to the target object as a third location determination method, wherein the third location determination method represents determining the physical location information of the ground point of the target object according to the ground equation and the ground point information in the image coordinate system of the three-dimensional attribute information, wherein the ground point is the point on the target object closest to the acquisition device of the roadside sensing image; The second determining unit is configured to use the position determining method to determine the physical position information of the target object based on the three-dimensional attribute information.
12. The apparatus according to claim 11, wherein, The first determining unit is further configured to: In response to determining that the target type is a first preset type that characterizes the target object as having regular vertex features, the method for determining the position of the target object is a first position determination method, wherein the first position determination method characterizes the determination of the physical position information of the vertex of the target object based on the ground equation and the vertex information in the image coordinate system of the three-dimensional attribute information.
13. The apparatus according to claim 12, wherein, The second determining unit is further configured to: In response to the target type being the first preset type, vertex information of multiple vertices of the target object in the image coordinate system is determined from the three-dimensional attribute information; For each of the plurality of vertices, the physical location information of the vertex is determined based on the ground equation and the vertex information of that vertex.
14. The apparatus according to claim 11, wherein, The second determining unit is further configured to: In response to the target type being the second preset type, the bottom center point information of the target object in the image coordinate system is determined from the three-dimensional attribute information; the physical location information of the bottom center point is determined according to the ground equation and the bottom center point information.
15. The apparatus according to claim 11, wherein, The second determining unit is further configured to: In response to the target type being the third preset type, the grounding point information of the grounding point on the target object in the image coordinate system is determined according to the three-dimensional attribute information; the physical location information of the grounding point is determined according to the ground equation and the grounding point information.
16. The apparatus according to claim 15, wherein, The second determining unit is further configured to: Based on the size information and orientation angle information of the target object in the three-dimensional attribute information, the grounding point information in the image coordinate system is determined.
17. The apparatus according to claim 11, wherein, The detection unit is further configured to: The target detection network is used to determine the category, detection box, and feature data of the target object in the roadside perception image.
18. The apparatus according to claim 17, wherein, The detection unit is further configured to: The feature extraction subnetwork in the target detection network extracts features from the roadside sensing image to obtain image features; the target detection subnetwork in the target detection network detects target objects in the roadside sensing image at different feature scales based on the image features to obtain multiple candidate detection boxes; the multiple candidate detection boxes are deduplicated to determine the detection box; Determine the category and feature data of the target object corresponding to the detection box.
19. The apparatus according to claim 17, wherein, The prediction unit is further configured to: A 3D attribute prediction network is used to predict the 3D attribute information of target objects in the roadside perception image based on the category, the detection box, and the feature data.
20. The apparatus according to claim 19, wherein, The prediction unit is further configured to: Using multiple attribute prediction heads in the 3D attribute prediction network, the 3D attribute information corresponding to each of the multiple attribute prediction heads is predicted based on the category, the detection box, and the feature data.
21. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method of any one of claims 1-10.
22. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-10.
23. A computer program product comprising: A computer program that, when executed by a processor, implements the method according to any one of claims 1-10.