Object position determination method and device, controller, vehicle and program product

Abstract

The embodiment of the invention relates to an object position determination method and device, a controller, a vehicle and a program product. The method includes determining a first depth, a three-dimensional (3D) size, and a 3D center point of an object in a traffic environment based on a two-dimensional (2D) image corresponding to the traffic environment. The method further includes calculating a first position of the object based on the 3D size and the 3D center point of the object, and parameters of a camera capturing the image. The method further includes determining a second position of the object based on the first depth and a parameter of the camera. The method further includes determining a 3D position of the object based on the first position and the second position. In this way, the position of the object in the 3D space can be accurately and flexibly estimated based on the 2D visual information shot by the camera and the constraint of the geometrical relationship under the condition that the number of training data for the object detection model is insufficient or the quality is poor.

Description

Technical Field Embodiments of the present disclosure generally relate to the field of computers, and more particularly, to methods and devices, controllers, vehicles, and program products for determining a position of an object. Background Art Object detection is a computer vision technology that can be used to locate and identify objects of interest in visual information such as images or videos. For example, vehicles, pedestrians, and road signs in a traffic environment. Object detection is a key component of assisted driving and autonomous driving technologies. Such technologies can help vehicle driving systems (such as advanced driver assistance systems (ADAS) and autonomous driving (AD) systems) better understand the traffic environment, making safer and more effective decisions and taking appropriate actions. In addition, object detection also helps with traffic scheduling and management, accident identification, and other aspects. Object detection can be used to determine an object's category and location, among other things. Artificial intelligence (AI) / machine learning (ML) models used for object detection are capable of extracting features representing elements from visual information and, after training, identifying objects. With the continuous advancement of technologies such as computer vision and machine learning, the accuracy and real-time performance of object detection have continued to improve. Summary of the Invention Embodiments of the present disclosure provide a method, apparatus, device, and medium for determining a location of an object. According to a first aspect of the present disclosure, a method for determining the position of an object is provided. The method includes determining a first depth, a three-dimensional (3D) size, and a 3D center point of an object in a traffic environment based on a two-dimensional (2D) image corresponding to the traffic environment. The method also includes calculating a first position of the object based on the 3D size and 3D center point of the object and parameters of a camera that captured the image. The method also includes determining a second position of the object based on the first depth and the camera parameters. The method also includes determining the 3D position of the object based on the first position and the second position. According to a second aspect of the present disclosure, a device for determining the position of an object is provided. The device includes a first determination module configured to determine a first depth, a three-dimensional (3D) size, and a 3D center point of an object in a traffic environment based on a two-dimensional (2D) image corresponding to the traffic environment. The device also includes a calculation module configured to calculate the first position of the object based on the 3D size and 3D center point of the object and parameters of a camera that captured the image. The device also includes a second determination module configured to determine the second position of the object based on the first depth and the parameters of the camera. The device also includes a position determination module configured to determine the 3D position of the object based on the first position and the second position. According to a third aspect of the present disclosure, a controller is provided. The electronic device includes at least one processor. The electronic device also includes a memory coupled to the at least one processor and having instructions stored therein, which, when executed by the at least one processor, cause the device to perform the method of the first aspect of the present disclosure. According to a fourth aspect of the present disclosure, a vehicle is provided, comprising the controller according to the third aspect of the present disclosure. According to a fifth aspect of the present disclosure, a computer program product is provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions that, when executed by a processor of a computer, cause the computer to implement the method of the first aspect of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS By describing the exemplary embodiments of the present disclosure in more detail with reference to the accompanying drawings, the above and other purposes, features and advantages of the present disclosure will become more apparent. In the exemplary embodiments of the present disclosure, the same or similar reference numerals generally represent the same or similar parts, components, etc. Figure 1 A schematic diagram illustrating an example environment in which methods and / or apparatus according to embodiments of the present disclosure may be implemented; Figure 2 A flowchart of a method for determining a position of an object according to an embodiment of the present disclosure is illustrated; Figure 3 A diagram illustrating a process of determining an object position according to an embodiment of the present disclosure; Figure 4 FIG2 illustrates a schematic diagram of identifying an object in a 3D space based on its height according to an embodiment of the present disclosure; Figure 5 A schematic diagram illustrating an example architecture of an object detection model according to an embodiment of the present disclosure; Figure 6A A diagram illustrating an example of the 3D size and 3D center point of an object, and Figure 6B yes Figure 6A A partial enlarged view of Figure 7 A diagram illustrating a real-world test of visualization of 3D object detection according to an embodiment of the present disclosure; and Figure 8 A diagram illustrating a verification process of a position determination result according to an embodiment of the present disclosure; Figure 9 A schematic diagram illustrating an apparatus for determining a position of an object according to an embodiment of the present disclosure; and Figure 10 A schematic block diagram of an example device suitable for implementing embodiments of the present disclosure is illustrated. In the various drawings, the same or corresponding reference numerals denote the same or corresponding parts. DETAILED DESCRIPTION

[0001] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0002] In the description of the embodiments of the present disclosure, the term "including" and its variations should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. can refer to different or the same objects, unless explicitly indicated to be different.

[0003] Object detection is a computer vision technology that plays a key role in many usage scenarios. It can identify the location of objects based on visual information (for example, in the form of rectangular boxes, etc.), and can also determine the category, identity and more information of the identified objects. Object detection technology helps promote scene understanding, behavior analysis, etc. For example, in the field of intelligent driving, by performing object detection, key elements such as pedestrians, other vehicles, road signs, etc. can be identified, so as to make safe and effective driving decisions. Accurate object detection functionality is conducive to improving the performance of ADAS and AD systems, improving blind spot detection, automatic parking assistance (APA), adaptive cruise control (ACC), lane keeping assistance (LKA), automatic emergency braking, etc. In addition, for example, in the field of intelligent security, by performing object detection, abnormal objects and behaviors, such as intruder intrusion, fire, etc., can be monitored in real time, and alarms can be issued in time.

[0004] After training, AI / ML models used for object detection can be configured to extract key features that characterize objects of interest from visual information and further determine the location of one or more objects. Building a well-performing and stable model requires a sufficient amount of high-quality training data to ensure that the inference behavior meets expectations, which is crucial for both model accuracy and generalization capabilities.

[0005] For example, in the application of object detection in the driving field, geographic data can be used to train the object detection model in order to accurately output the object location. However, collecting geographic data is often time-consuming and labor-intensive, especially in long-tail scenarios that are rarely accessed or difficult to reproduce. In addition, geographic data is sensitive, its acquisition process is strictly regulated, and its use is subject to restrictions. If the amount of training data available for training is insufficient or the quality is poor, the object location determination results output by the object detection model will be degraded.

[0006] To improve the accuracy of object location determination, solutions may rely on data augmentation strategies to augment existing training data that meets the requirements. However, simply focusing on sample size alone cannot fully address accuracy issues. For example, if the augmented training data is too similar or lacks sufficient diversity, it may lead to overfitting of the model. This means that the model performs well on the training data but has poor generalization ability in real-world applications. At least to address the above and other potential problems, an embodiment of the present disclosure provides a solution for determining the position of an object. The solution for determining the position of an object according to an embodiment of the present disclosure includes determining a first depth, a three-dimensional 3D size, and a 3D center point of an object in a traffic environment based on a two-dimensional 2D image corresponding to the traffic environment. The solution also includes calculating the first position of the object based on the 3D size and 3D center point of the object, and the parameters of the camera that captured the image. The solution also includes determining the second position of the object based on the first depth and the parameters of the camera. The solution also includes determining the 3D position of the object based on the first position and the second position. In this way, when the amount of training data for the object detection model is insufficient or the quality is poor, the position of the object in 3D space can be accurately and flexibly estimated based on the constraints of the 2D visual information captured by the camera and the geometric relationship.

[0007] Reference below Figures 1 to 10 It should be understood that these exemplary embodiments are provided only to enable those skilled in the art to better understand and implement the embodiments of the present disclosure, and are not intended to limit the scope of the present disclosure in any way.

[0008] Figure 1 A schematic diagram of an example environment 100 in which methods and / or processes according to embodiments of the present disclosure may be implemented is shown. For ease of understanding, the following description uses object detection applied to a driving context. It should be understood that this is not limiting, and that methods according to embodiments of the present disclosure may also be applied to other diverse contexts, such as the security context discussed above. Object detection applications in such security contexts may be deployed in schools, banks, and the like.

[0009] like Figure 1 As shown in FIG, the example environment 100 may include a camera 110, an image 111 about a traffic environment, a computing device 120, and a storage device 130, and these components may be coupled to each other for interaction, as shown in FIG. Figure 1 As shown in . It should be understood that limited components are shown in the example environment 100 for implementing the embodiments of the present disclosure for ease of understanding and illustration only, and the embodiments of the present disclosure are not limited thereto. For example, the example environment 100 may further include a display (not shown) configured to display the image 111, the determined position information of the object, etc.

[0010] According to embodiments of the present disclosure, camera 110 can be configured to monitor and record the traffic environment. Camera 110 can capture various elements involved in traffic, such as vehicles, pedestrians, and obstacles on the road. In some embodiments, camera 110 may comprise an infrastructure monocular camera. Such cameras are typically installed at major traffic arteries, intersections, or other key areas. With a fixed viewing angle and focal length, they enable continuous and stable monitoring of a specific area. Infrastructure monocular cameras offer advantages in reliability and stability, capable of operating normally in various weather and lighting conditions, ensuring the accuracy and integrity of traffic data. Furthermore, infrastructure monocular cameras can interact with traffic dispatch systems, enabling real-time data transmission and sharing to support instant analysis of traffic conditions, enabling, for example, rapid dispatch decisions to improve road efficiency and reduce the risk of traffic accidents. In other embodiments, camera 110 may comprise a vehicle-mounted camera, which is mounted on a vehicle and moves with the vehicle. It should be understood that the embodiments of the present disclosure do not specify the specific model and configuration of camera 110; the appropriate camera can be selected based on specific usage needs.

[0011] According to an embodiment of the present disclosure, the image 111 of the traffic environment may be a 2D image captured by the camera 110 of the traffic environment, which visually indicates various objects in the traffic environment, such as various elements involved in traffic, such as vehicles, pedestrians, obstacles, etc. The vehicle described herein refers to a motorized or non-motorized vehicle that can carry people and / or objects and has the ability to move. Figure 1 As shown in FIG, the vehicle may include, for example, a car, a motorcycle, etc. In some embodiments, the type of the image 111 may include, but is not limited to, an RGB image, an infrared image, etc.

[0012] In some embodiments, the image 111 of the traffic environment may be pre-processed before being input into the object detection model 121 to determine the location of the object. Examples of such pre-processing may include, but are not limited to, normalization to unify the image format, cropping to remove redundant background, scaling to adjust the image size, flipping or rotating to present a desired angle and direction, and brightness and contrast adjustment to optimize the image visual effect.

[0013] According to an embodiment of the present disclosure, the computing device 120 may include an on-board computing device (which means that the computing device 120 may be located inside the vehicle 100), an off-board computing device, or a combination thereof. The computing device 120 may have computing capabilities suitable for performing object detection, so that the AI / ML model 121 applied to object detection (hereinafter, also referred to as the object detection model 121) can run thereon. The computing device 120 is configured to perform object position determination according to an embodiment of the present disclosure. During execution, the computing device 120 may process an image 111 corresponding to a traffic environment captured by the camera 110, and may access a training data set 131 stored in the storage device 130, which is utilized to train the object detection model 121. It should be understood that the computing device 120 Figure 1 1 is shown as one computing device, but this is only illustrative and not limiting, and a greater number of computing devices may exist in the example environment 100. In the following, the corresponding operations of the computing device 120 will be described in further detail.

[0014] By way of example and not limitation, the computing device 120 may include, but is not limited to, an in-vehicle computer, a personal computer, a laptop computer, a server computer, a mobile device (such as a smartphone, a tablet computer, etc.), a wearable electronic device, a multimedia player, a personal digital assistant (PDA), a smart home device, a consumer electronic product, or a distributed computing environment including any one or more of the above devices.

[0015] According to an embodiment of the present disclosure, the storage device 130 may be configured to store a training data set 131, an object detection model 121 to be run on the computing device 120 and its parameters, intermediate and final results of object position determination, etc. It should be understood that the embodiment of the present disclosure does not specify the specific type and architecture of the object detection model 121, and a suitable model can be selected according to specific usage needs, examples of which may include but are not limited to convolutional neural networks, support vector machines, etc. In addition, the storage device 130 may store the training data set 131, the object detection model 121 to be run on the computing device 120, and the parameters thereof, the intermediate and final results of object position determination, etc. Figure 1 One storage device is shown in the example environment 100 , but this is only illustrative and not restrictive, and a greater number of storage devices may exist in the example environment 100 .

[0016] like Figure 1 As shown in , training dataset 131 may be stored in storage device 130 and accessed by computing device 120. In some embodiments, training dataset 131 may include sample data in the form of images, videos containing image frames, etc., and these sample data may be diverse (e.g., different driving scenarios, or acquired via different devices). In training dataset 131, these sample data may be classified into subsets having the same or similar sample data based on a clustering strategy.

[0017] By way of example and not limitation, the storage device 130 may include, but is not limited to, a local storage device, a remote storage device, and combinations thereof. In some embodiments, the multiple storage devices in the storage device 130 may include, but are not limited to, a mechanical hard disk drive (HDD), a solid-state drive (SSD), and the like, and some of the multiple storage devices may be locally located, while others may be remotely located, for example, coupled together via a line or network.

[0018] Combined with the above Figure 1 An example environment 100 is described in which methods and / or processes according to embodiments of the present disclosure may be implemented. Figure 2 , a flowchart of method 200 for determining the position of an object according to an embodiment of the present disclosure is described. Through this method 200, even when the amount of training data for object detection model 121 is limited or the quality is unsatisfactory, the position of the object of interest in 3D space can still be determined with high accuracy and flexibility by relying on the 2D visual information captured by camera 110 and combined with ground constraints. This process not only fully utilizes existing visual data resources but also effectively compensates for the limitations of insufficient or poor data quality through geometric analysis.

[0019] Figure 2A flowchart of a method 200 for determining the position of an object according to an embodiment of the present disclosure is illustrated. At block 210, a first depth, 3D size, and 3D center point of an object in the traffic environment are determined based on a 2D image 111 corresponding to the traffic environment. By way of example and not limitation, the image 111 corresponding to the traffic environment, captured by the camera 110, is input to an object detection model 121 running on a computing device 120. The object detection model 121 has been trained based on a training dataset 131 stored in a storage device 130. The trained object detection model 121 has the expected reasoning capabilities and is configured to infer the depth, 3D size, and 3D center point of an object of interest in the traffic environment based on the 2D image 111 corresponding to the traffic environment. For ease of reference and descriptive purposes, this inferred depth will be referred to as the first depth below. In some embodiments, the inferred 3D size and 3D center point may be presented in a UV coordinate system (where U represents horizontal and V represents vertical) and may indicate the spatial size and geometric center of the object, respectively.

[0020] According to an embodiment of the present disclosure, at block 220, a first position of the object is calculated based on the 3D size and 3D center point of the object and the parameters of the camera 110 that captured the image 111. The camera parameters may include intrinsic and extrinsic parameters of the camera. The intrinsic parameters of the camera may be intrinsic parameters, primarily including focal length, principal point coordinates, distortion coefficients, etc. These intrinsic parameters describe the optical and geometric characteristics of the camera, as well as the relationship between the image sensor and the lens. The extrinsic parameters of the camera may be the pose parameters of the camera, which describe the position and orientation of the camera in 3D space, i.e., the relative relationship between the camera coordinate system and the world coordinate system. These parameters typically include a rotation matrix and a translation vector, which are used to transform points in the camera coordinate system into the world coordinate system. In some embodiments, the intrinsic and extrinsic parameters of the camera 110 include a KRT matrix, where K represents the intrinsic parameter matrix of the camera 110, R represents the rotation matrix, and T represents the translation vector.

[0021] In some embodiments, the calculation of box 220 can also be combined with ground mapping, which corresponds to a constraint of geometric relationship or is called a ground constraint. For example, the ground mapping can be configured to restrict each object in each scene to the ground based on the ground plane equation. In this way, based on the above, when the 3D center point and 3D size of the target object are known, its position and depth can be calculated. It should be understood that restricting the object to the ground is one of the many implementations of the present disclosure, and other forms of restrictions, such as wall constraints, can also be used. For ease of reference and reference, the calculated position is referred to as the first position hereinafter. The position calculation process will be further described in detail below.

[0022] According to an embodiment of the present disclosure, at block 230, a second position of the object is determined based on the first depth of the object and the parameters of camera 110. As described above, at block 230, a 2D image 111 captured by camera 110 of a traffic environment is input into a trained object detection model 121, and the model output includes an inferred first depth of the object. Based on this inferred first depth and the parameters of camera 110, the position of the object can be determined. For ease of reference, this determined position will be referred to as the second position below. This process will be described in further detail below.

[0023] According to an embodiment of the present disclosure, at box 240, the 3D position of the object is determined based on the calculated first position and the obtained second position. As described above, the first position of the object is calculated at 220, and the second position of the object is inferred at 230. Based on the calculation result and the inference result of the object position, the 3D position of the object can be determined. The object position determination process based on the calculation result and the inference result of the object position will be further described in detail below. Through the method 200 for determining the position of an object according to an embodiment of the present disclosure, even in the face of the challenge of limited amount or poor quality of training data for training the object detection model 121, accurate and flexible object detection can be achieved with the help of 2D visual information captured by the camera and ground constraints.

[0024] Figure 3 FIG. 3 illustrates a diagram of an object position determination process 300 according to an embodiment of the present disclosure. Figure 3 As exemplarily shown in FIG, the object position determination process 300 may include a model training subprocess 310, a 2D-3D reasoning subprocess 320, a first position calculation subprocess 330, a second position determination subprocess 340, a position difference comparison subprocess 350, and a coordinate value processing subprocess 360. The object position determination process 300 and its subprocesses may be abstracted into an object position determination unit and corresponding subunits for each subprocess (e.g., a model training subunit, a 2D-3D reasoning subunit, etc.). These units and subunits may be software-based components or systems for determining the position of an object and may be run on a device having computing capabilities (such as the computing device 120).

[0025] According to an embodiment of the present disclosure, in the model training sub-process 310, the object detection model 121 can be trained so that it has the expected reasoning ability for object position determination. The image 111 taken by the camera 110 for the traffic environment and input to the object detection model 121 may include a specific image. Here, the term "specific image" may refer to an image having a specific attribute that can help identify the image and images that are the same or similar to it in the entire data set, or can be distinguished from other images in the entire data set. For example, the captured image indicates a specific scene of the traffic environment, such as driving in rainy and snowy weather, driving in congested road conditions, driving in tunnel sections, etc. In some embodiments, the specific image may be associated with at least one of the following: a predetermined scene, the posture of the camera 110 for capturing the specific image (for example, the shooting angle, shooting distance and height of the camera 110, its viewing angle, etc.), and the hardware configuration of the camera 110.

[0026] By way of example and not limitation, when the camera 110 captures an image of a vehicle traveling in a tunnel scene, if the object detection model 121 is expected to have high-performance position determination capabilities, a large amount of training data for tunnel scenes (a collection of which is referred to as a specific dataset) may be required to train the model. A dataset that includes specific training data can certainly promote the performance improvement of the model for specific problems. However, obtaining a specific dataset is not as easy as imagined. In the field of driving, the collection of geographic data is often time-consuming, labor-intensive, and sensitive, and its use is subject to strict supervision and many restrictions. To overcome such limitations, the model training subprocess 310 can use a non-specific training dataset relative to the specific image to train the object detection model 121. Here, in addition to the attributes of the specific image, the non-specific training dataset may include non-specific images with other different attributes.

[0027] The non-specific training dataset can be an open-source or public training dataset, which is relatively easy to obtain and use. In some embodiments, the non-specific training dataset can include at least one of the following: training data for scenes different from the scene of the specific image; training data for camera 110 in poses different from the pose used to capture the specific image; and training data for various cameras with hardware configurations different from that of camera 110. The various sub-processes of object position determination process 300 described below will ensure that the model achieves comparable or even superior position determination performance even when the amount of training data is insufficient or the quality is poor.

[0028] According to an embodiment of the present disclosure, in the 2D-3D inference sub-process 320, the trained object detection model 121 may be configured to infer a first depth, a 3D center point of the object in the UV coordinate system, and a 3D size of the object from a 2D image 111 captured by the camera 110 of the traffic environment. The coordinate system value of the 3D center point of the object in the UV coordinate system may be calculated as a value uniquely corresponding to a point.

[0029] According to an embodiment of the present disclosure, in the first position calculation subprocess 330, based on the 3D center point and 3D size of the object in the UV coordinate system inferred in the 2D-3D reasoning subprocess 320, and based on the parameters of the camera 110 (e.g., intrinsic parameters and extrinsic parameters), the computing device 120 can calculate the first position of the object. As described above, the intrinsic parameters of the camera describe the optical and geometric characteristics of the camera, as well as the relationship between the image sensor and the lens. The extrinsic parameters of the camera describe the position and orientation of the camera in 3D space, that is, the relative relationship between the camera coordinate system (e.g., UV coordinate system) and the world coordinate system (e.g., ENU coordinate system). These parameters generally include a rotation matrix and a translation vector, which are used to transform a point in the camera coordinate system into the world coordinate system.

[0030] In some embodiments, the calculation process of the first position can also be based on a ground map, which can restrict each object in each scene to the ground through the ground plane equation. In addition, based on the inferred 3D center point and 3D size of the object in the UV coordinate system, the intrinsic and extrinsic parameters of the camera 110, and the ground map, the depth of the object can be calculated. For ease of reference and reference, the calculated depth will be referred to as the second depth below. In some embodiments, based on the 3D size of the object, the computing device 120 can determine the height of the 3D center point from the ground. Based on the determined height, the object of interest can be identified in the ENU coordinate system, such as its position in the ENU coordinate system (i.e., the first position calculated above), and its depth (the second depth calculated above). The following will be combined with Figure 4 Let's explain this in more detail.

[0031] Figure 4 FIG. 1 illustrates a schematic diagram of identifying an object in a 3D space based on its height according to an embodiment of the present disclosure. Figure 4As shown in FIG, camera 410 can capture traffic conditions, projection 420 is the optical path from camera 410 to the object of interest, 3D size 401 indicates the spatial size of the object of interest, 3D center point 402 indicates the geometric center of the object of interest, ground projection point 403 is obtained by vertically projecting 3D center point 402 onto the ground, and coordinate system point 404 can be a reference point (e.g., the origin) in the ENU coordinate system. After determining the 3D center point 402 of the object, there may be more than one matching point on projection 420, and it may be difficult to identify the object based on the 3D center point. The inferred 3D size of the object (e.g., via the trained object detection model 121) is often relatively accurate, and in some cases, is even identical to the true value. Therefore, the height of 3D center point 402 from the ground (i.e., the distance from 3D center point 402 to ground projection point 403) can be determined based on 3D size 401. In this way, by determining whether the height and its 3D center point are met on the projection 420, the object can be identified in the ENU coordinate system, such as its type, position, depth, etc.

[0032] return Figure 3 According to an embodiment of the present disclosure, in the second position determination sub-process 340, the second position of the object can be inferred or determined based on the first depth inferred in the 2D-3D inference sub-process 320 and the intrinsic and extrinsic parameters of the camera 110 that captured the input image 111, for example, via the object detection model 121. In some embodiments, the coordinates of the object in the ENU coordinate system can be determined based on the first depth of the object inferred via the trained object detection model 121 and the intrinsic and extrinsic parameters of the camera 110. In some embodiments, the 3D depth of the object can be determined based on the first depth inferred in the 2D-3D inference sub-process 320 and the second depth calculated in the first position calculation sub-process 330.

[0033] According to an embodiment of the present disclosure, in the position difference comparison sub-process 350, the first position calculated in the first position calculation sub-process 330 may be compared with the second position inferred in the second position determination sub-process 340 to obtain a position difference between the first and second positions. The obtained position difference may then be compared with a difference threshold. In some embodiments, such a difference threshold may be predetermined or may dynamically change, for example, based on predefined conditions.

[0034] If the obtained difference threshold is not greater than the difference threshold, it indicates that the first position calculated and the second position inferred in the previous sub-process are consistent with expectations, or that their errors are within an acceptable range. At this point, the process proceeds to the coordinate value processing sub-process 360 to determine the 3D position of the object of interest based on the inferred position result and the calculated position result. If the obtained difference threshold is greater than the difference threshold, these position results are discarded and subsequent object position determination is performed, for example, returning to the 2D-3D reasoning sub-process 320.

[0035] In addition to directly comparing the two as described above, other implementations can be used to determine the difference between the inferred and calculated position results. In other embodiments, a weighted comparison can be used, or they can be compared to predetermined thresholds. For example, the calculated first position and the inferred or determined second position can be compared to predetermined thresholds. If at least one of the comparison results does not meet expectations, the calculated and inferred results can be discarded.

[0036] According to an embodiment of the present disclosure, in the coordinate value processing sub-process 360, if the acquired position difference is not greater than the difference threshold, for example, the computing device 120 averages or weighted averages the first coordinate system value in the ENU coordinate system of the first position and the second coordinate system value in the ENU coordinate system of the second position to obtain a third coordinate system value in the ENU coordinate system after the average or weighted average. The coordinates with the third coordinate system value in the ENU coordinate system can then be determined as the 3D position of the object. If the acquired position difference is greater than the difference threshold, the first coordinate system value in the ENU coordinate system of the first position and the second coordinate system value in the ENU coordinate system of the second position will be discarded. The object position determination according to an embodiment of the present disclosure will be further described in detail below with reference to the accompanying drawings.

[0037] Figure 5 A schematic diagram of an example architecture 500 of an object detection model according to an embodiment of the present disclosure is illustrated. An image 511 corresponding to a traffic environment can be input into a trained object detection model, and features are then extracted from the image 511 via a feature extraction component 520 (also referred to as a backbone) of the model. The extracted features can then be fed into corresponding regression components in a plurality of regression components (also referred to as regression heads) of the model for reasoning. In some embodiments, these regression components may include at least one of the following: a heatmap regression component 521, an offset regression component 522, a size regression component 523, an orientation regression component 524, and a depth regression component 525. It should be understood that more or fewer regression heads may also be used depending on the specific needs of use.

[0038] The inference results, via each regression head, may include the 3D size 540 and 3D center point 550 of the object, as well as the first depth of the object (e.g., via the depth regression component 525). As described above, the position calculation result 570 may include the calculated second depth and first position of the object, and the calculation process may be based on the ground map 530, the inferred 3D size 540 and 3D center point 550 of the object, and the intrinsic and extrinsic parameters 560 of the camera. In addition, the position inference result 580 may include the second position of the object, and the inference process may be based on the first depth of the object and the intrinsic and extrinsic parameters 560 of the camera. Based on the position calculation result 570 and the position inference result 580, the 3D position 590 of the object may be determined.

[0039] Figure 6A A diagram illustrating an example of the 3D size and 3D center point of an object, and Figure 6B yes Figure 6A A partial enlarged view of Figure 6A As shown in , each object in the traffic environment is identified in the form of a frame, where the frame can indicate the size of the identified object. It should be noted that Figure 6A The box in is the ground truth result. Figure 6B , point 610 refers to the object of interest ( Figure 6B ), point 620 refers to the 3D center point of the object inferred by the model, and box 630 refers to the 3D size of the ground truth of the object.

[0040] Figure 7 The figure shows a diagram of a real-world test of visualization of 3D object detection according to an embodiment of the present disclosure. By determining the position of the object according to an embodiment of the present disclosure, each element involved in the traffic can be identified in the form of a box. It should be understood that it is possible to choose whether to show more position identification information (e.g., the 3D center point of the object, etc.) as needed. The position determination result can be further converted into coordinates (e.g., in the ENU coordinate system) to indicate the position information of the object of interest in the form of coordinate values.

[0041] Figure 8 FIG. 8 is a diagram illustrating a verification process 800 of a position determination result according to an embodiment of the present disclosure. Figure 8As shown in FIG, point 810 is the true center point of the object, and point 820 is the detected center point of the object determined according to an embodiment of the present disclosure. In some embodiments, the distance between the true center point of the object and the detected center point of the object can be determined. If the distance between the two points is not greater than a predetermined distance threshold, the result of the object position determination can be stored or recorded (e.g., in storage device 130). If the distance is greater than the predetermined distance threshold, the result of the object position determination is discarded.

[0042] Figure 9 FIG2 shows a schematic diagram of an apparatus 900 for determining a position of an object according to an embodiment of the present disclosure. The apparatus 900 may include multiple units or modules for performing the following steps: Figure 2 The corresponding steps in the method 200 discussed in Figure 9 As shown in , the device 900 includes: a first determination module 910, configured to determine a first depth, a three-dimensional 3D size and a 3D center point of an object in the traffic environment based on a two-dimensional 2D image corresponding to the traffic environment; a calculation module 920, configured to calculate a first position of the object based on the 3D size and the 3D center point of the object and parameters of a camera that captures the image; a second determination module 930, configured to determine (230) a second position of the object based on the first depth and the parameters of the camera; and a position determination module 940, configured to determine (240) the 3D position of the object based on the first position and the second position.

[0043] In some embodiments, the image may include a specific image, which is associated with at least one of: a predetermined scene, the posture of the camera capturing the specific image, and the hardware configuration of the camera, and the first depth, the three-dimensional 3D size and the 3D center point of the object may be output by an object detection model based on the input image, and the object detection model is trained based on a non-specific training dataset relative to the specific image.

[0044] In some embodiments, the non-specific training data set may include at least one of the following: training data for additional scenes different from the predetermined scene; training data for additional postures of the camera different from the posture; and training data for various types of cameras with additional hardware configurations different from the hardware configuration.

[0045] In some embodiments, the coordinate system value of the 3D center point of the object in the UV coordinate system can be calculated as a value uniquely corresponding to a point, the parameters of the camera may include intrinsic parameters and extrinsic parameters, and the calculation of the first position can also be based on a ground mapping, which restricts each object in each scene to the ground through a ground plane equation.

[0046] In some embodiments, the calculation module 920 may be further configured to: determine the height of the 3D center point of the object from the ground based on the 3D size of the object; and identify the object in the ENU coordinate system based on the determined height.

[0047] In some embodiments, the first determination module 910 can also be configured to: extract features from the specific image via a feature extraction component of the trained object detection model; and feed the extracted features into a corresponding regression component among multiple regression components of the trained object detection model for inference, wherein the multiple regression components include at least one of the following: a heat map regression component, an offset regression component, a size regression component, an orientation regression component, and a depth regression component. In some embodiments, the second determination module 930 can also be configured to: infer the first depth of the object via the depth regression component; and determine the coordinates of the object in the ENU coordinate system based on the inferred first depth and the intrinsic and extrinsic parameters of the camera.

[0048] In some embodiments, the calculation module 920 can also be configured to calculate the second depth of the object based on the three-dimensional 3D size and the 3D center point of the object, the intrinsic parameters and the extrinsic parameters of the camera, and the ground mapping, and the device 900 can also include a depth determination module configured to determine the 3D depth of the object based on the inferred first depth and the calculated second depth.

[0049] In some embodiments, the device 900 may also include: a position difference acquisition module, configured to obtain the position difference between the first position and the second position by comparing the calculated first position with the obtained second position; and a judgment module, configured to determine whether the obtained position difference is greater than a difference threshold.

[0050] In some embodiments, the determination module 940 can also be configured to: in response to determining that the acquired position difference is not greater than the difference threshold, obtain a third coordinate system value in the ENU coordinate system after weighted averaging by taking the first coordinate system value in the ENU coordinate system of the first position and the second coordinate system value in the ENU coordinate system of the second position; and determine the coordinate having the third coordinate system value in the ENU coordinate system as the 3D position of the object.

[0051] In some embodiments, the device 900 may further include a discarding module configured to: discard the first coordinate system value in the ENU coordinate system of the first position and the second coordinate system value in the ENU coordinate system of the second position in response to determining that the acquired position difference is greater than the difference threshold. Figure 10 The schematic block diagram of an example device 1000 suitable for implementing an embodiment of the present disclosure is illustrated. The controller above can be implemented using device 1000. As shown, device 1000 includes a processor 1001, which can be loaded into a computer program instruction in a random access memory (RAM) 1003 according to the computer program instruction stored in a read-only memory (ROM) 1002, to perform various appropriate actions and processes. In RAM 1003, various programs and data required for the operation of device 1000 can also be stored. Processor 1001, ROM 1002 and RAM 1003 are connected to each other via bus 1004. Input / output (I / O) interface 1005 is also connected to bus 1004. The various processes and processing described above, such as method 200, can be executed by processor 1001. For example, in some embodiments, method 200 can be implemented as a computer software program, which is tangibly contained in a machine-readable medium. In some embodiments, part or all of the computer program can be loaded and / or installed on device 1000 via ROM 1002. When the computer program is loaded into RAM 1003 and executed by processor 1001, one or more actions of method 200 described above can be performed. According to an embodiment of the present disclosure, a vehicle is provided, which may include device 1000 as described above for performing various aspects of the present disclosure. The present disclosure may be a method, an apparatus, an electronic device, a vehicle, a computer-readable storage medium, and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present disclosure. Computer-readable storage media can be a tangible device that can hold and store the instructions used by the instruction execution device. Computer-readable storage media can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove having instructions stored thereon, and any suitable combination thereof. Computer-readable storage media used herein is not interpreted as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagated by waveguides or other transmission media (for example, light pulses by fiber optic cables), or electrical signals transmitted by wires. The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device. The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and conventional procedural programming languages ​​such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., utilizing an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions. The electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure. Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions. These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine such that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram. Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram. The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and the part of the module, program segment or instruction includes one or more executable instructions for realizing the prescribed logical function. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the prescribed function or action, or can be implemented with a combination of dedicated hardware and computer instructions. While various embodiments of the present disclosure have been described above, the foregoing description is intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technical improvements to existing technologies, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method (200) for determining a position of an object, comprising: Determining (210) a first depth, a three-dimensional (3D) size, and a 3D center point of an object in the traffic environment based on a two-dimensional (2D) image corresponding to the traffic environment; calculating (220) a first position of the object based on the 3D size and the 3D center point of the object and parameters of a camera that captured the image; determining (230) a second position of the object based on the first depth and the parameters of the camera; as well as Based on the first position and the second position, a 3D position of the object is determined (240).

2. The method according to claim 1, wherein The image includes a specific image, the specific image being associated with at least one of: a predetermined scene, a posture of the camera for capturing the specific image, and a hardware configuration of the camera, and The first depth, the three-dimensional 3D size, and the 3D center point of the object are output by an object detection model based on the input image, and the object detection model is trained based on a non-specific training dataset relative to the specific image.

3. The method according to claim 2, wherein the non-specific training data set comprises at least one of the following: training data for another scenario different from the predetermined scenario; training data for additional poses of the camera that are different from the pose; and Training data for various types of cameras with additional hardware configurations different from the described hardware configuration.

4. The method according to claim 1, wherein The coordinate system value of the 3D center point of the object in the UV coordinate system is calculated as a value uniquely corresponding to a point. The parameters of the camera include intrinsic parameters and extrinsic parameters, and The calculation of the first position is also based on a ground map, which constrains each object in each scene to be on the ground through a ground plane equation.

5. The method of claim 4, wherein calculating the first position comprises: Determining a height of the 3D center point of the object from the ground based on the 3D size of the object; as well as Based on the determined height, the object is identified in an ENU coordinate system.

6. The method according to claim 2, wherein determining the first depth, three-dimensional 3D size, and 3D center point of the object further comprises: extracting features from the specific image via a feature extraction component of the trained object detection model; as well as feeding the extracted features into corresponding regression components among a plurality of regression components of the trained object detection model for inference, The multiple regression components include at least one of the following: a heat map regression component, an offset regression component, a size regression component, a directional regression component, and a depth regression component.

7. The method of claim 6, wherein determining the second position comprises: inferring the first depth of the object via the depth regression component; The coordinates of the object in an ENU coordinate system are determined based on the inferred first depth and intrinsic and extrinsic parameters of the camera.

8. The method according to claim 7, further comprising: Calculating a second depth of the object based on the three-dimensional (3D) size and the 3D center point of the object, the intrinsic and extrinsic parameters of the camera, and a ground map; as well as A 3D depth of the object is determined based on the inferred first depth and the calculated second depth.

9. The method according to claim 1, further comprising: obtaining a position difference between the first position and the second position by comparing the calculated first position with the obtained second position; as well as It is determined whether the acquired position difference is greater than a difference threshold.

10. The method of claim 9, wherein determining the 3D position of the object comprises: In response to determining that the acquired position difference is not greater than the difference threshold, obtaining a weighted average third coordinate system value in the ENU coordinate system by weighted averaging a first coordinate system value in the ENU coordinate system of the first position and a second coordinate system value in the ENU coordinate system of the second position; as well as The coordinate having the third coordinate system value in the ENU coordinate system is determined as the 3D position of the object.

11. The method according to claim 10, further comprising: In response to determining that the acquired position difference is greater than the difference threshold, a first coordinate system value in the ENU coordinate system of the first position and a second coordinate system value in the ENU coordinate system of the second position are discarded.

12. An apparatus (800) for determining a position of an object, comprising: A first determining module is configured to determine a first depth, a three-dimensional 3D size, and a 3D center point of an object in the traffic environment based on a two-dimensional 2D image corresponding to the traffic environment; a calculation module configured to calculate a first position of the object based on the 3D size and the 3D center point of the object and parameters of a camera that captures the image; a second determination module configured to determine ( 230 ) a second position of the object based on the first depth and the parameters of the camera; A position determination module is configured to determine (240) a 3D position of the object based on the first position and the second position.

13. A controller comprising: at least one processor; as well as A memory is coupled to the at least one processor and has instructions stored thereon, which, when executed by the at least one processor, cause the controller to perform the method according to any one of claims 1-11.

14. A vehicle comprising the controller according to claim 13.

15. A computer program product tangibly stored on a non-transitory machine-readable medium and comprising machine-executable instructions which, when executed by a processor of a machine, cause the machine to perform the method according to any one of claims 1 to 11.

Images