Model training and obstacle distance determination method and apparatus, electronic device, and medium

Abstract

The present disclosure provides a model training and obstacle distance determination method and device, electronic equipment, computer readable storage medium and computer program product, relating to the field of artificial intelligence, and particularly to the field of automatic driving technology. The implementation scheme is: obtaining a sample image containing an obstacle and a distance label between a preset collection device and the obstacle; inputting the sample image into a feature extraction network to obtain an image feature; inputting the image feature into a first neural network to obtain a first height of a 2D detection box corresponding to the obstacle; inputting the image feature into a second neural network to obtain a second height of a 3D detection box corresponding to the obstacle, a first distance between the preset collection device and the obstacle, a first uncertainty corresponding to the second height, and a second uncertainty corresponding to the first distance, so as to determine a value of a first loss function; and adjusting parameter values of each network based on the value of the first loss function.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of artificial intelligence, in particular to the field of automatic driving, and more particularly to a model training and obstacle distance determination method and device, electronic equipment, computer readable storage medium and computer program product. BACKGROUND

[0002] The current target detection technology based on images is increasingly mature. In the field of automatic driving, in order to ensure the safety of the automatic driving vehicle, the distance of the target obstacle needs to be estimated according to the image of the captured obstacle, so as to help understand the current state of the target obstacle, and at the same time, to assist in the subsequent prediction, tracking, obstacle avoidance and path planning of the automatic driving vehicle. SUMMARY

[0003] The present disclosure provides a model training and obstacle distance determination method, device, electronic equipment, computer readable storage medium and computer program product.

[0004] According to an aspect of the present disclosure, a model training method is provided, comprising: obtaining a sample image containing an obstacle based on a preset acquisition device, and a distance label between the preset acquisition device and the obstacle; inputting the sample image into a feature extraction network to obtain an image feature; inputting the image feature into a trained first neural network to obtain a first height of a 2-dimensional detection box corresponding to the obstacle; inputting the image feature or an image feature determined based on an output result of the first neural network into a trained second neural network to obtain a second height of a 3-dimensional detection box corresponding to the obstacle, a first distance between the preset acquisition device and the obstacle, a first uncertainty corresponding to the second height, and a second uncertainty corresponding to the first distance; determining a second distance between the preset acquisition device and the obstacle based on the first height and the second height; determining a value of a first loss function based on the first distance, the second uncertainty, the second distance, the first uncertainty and the distance label; and adjusting a parameter value of at least one of the feature extraction network, the first neural network and the second neural network based on the value of the first loss function.

[0005] According to another aspect of the present disclosure, a method for determining an obstacle distance is provided, including: obtaining an image including an obstacle based on a preset acquisition device; inputting the image into a feature extraction network to obtain an image feature; inputting the image feature into a trained first neural network to obtain a first height of a 2-dimensional detection box corresponding to the obstacle; inputting the image feature or an image feature determined based on an output result of the first neural network into a trained second neural network to obtain a second height of a 3-dimensional detection box corresponding to the obstacle, a first distance between the preset acquisition device and the obstacle, a first uncertainty corresponding to the second height, and a second uncertainty corresponding to the first distance; determining a second distance between the preset acquisition device and the obstacle based on the first height and the second height; and fusing the first distance and the second distance based on the first uncertainty and the second uncertainty to take a fused distance as a distance between the preset acquisition device and the obstacle.

[0006] According to another aspect of the present disclosure, a model training device is provided, including: a first obtaining unit configured to obtain a sample image including an obstacle based on a preset acquisition device, and a distance label between the preset acquisition device and the obstacle; a first feature module configured to input the sample image into a feature extraction network to obtain an image feature; a first determining unit configured to input the image feature into a trained first neural network to obtain a first height of a 2-dimensional detection box corresponding to the obstacle; a second determining unit configured to input the image feature or an image feature determined based on an output result of the first neural network into a trained second neural network to obtain a second height of a 3-dimensional detection box corresponding to the obstacle, a first distance between the preset acquisition device and the obstacle, a first uncertainty corresponding to the second height, and a second uncertainty corresponding to the first distance; a third determining unit configured to determine a second distance between the preset acquisition device and the obstacle based on the first height and the second height; a fourth determining unit configured to determine a value of a first loss function based on the first distance, the second uncertainty, the second distance, the first uncertainty, and the distance label; and a parameter adjusting unit configured to adjust a parameter value of at least one of the feature extraction network, the first neural network, and the second neural network based on the value of the first loss function.

[0007] According to another aspect of the present disclosure, there is provided an obstacle distance determination apparatus, comprising: a second acquisition unit configured to acquire an image including an obstacle obtained based on a preset acquisition device; a second feature unit configured to input the image into a feature extraction network to obtain image features; a fifth determination unit configured to input the image features into a trained first neural network to obtain a first height of a 2D detection box corresponding to the obstacle; a sixth determination unit configured to input the image features or image features determined based on an output result of the first neural network into a trained second neural network to obtain a second height of a 3D detection box corresponding to the obstacle, a first distance between the preset acquisition device and the obstacle, a first uncertainty corresponding to the second height, and a second uncertainty corresponding to the first distance; a seventh determination unit configured to determine a second distance between the preset acquisition device and the obstacle based on the first height and the second height; and a distance fusion unit configured to fuse the first distance and the second distance based on the first uncertainty and the second uncertainty to take a fused distance as a distance between the preset acquisition device and the obstacle.

[0008] According to another aspect of the present disclosure, there is provided an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the model training method or the obstacle distance determination method as described above.

[0009] According to another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the model training method or the obstacle distance determination method as described above.

[0010] According to another aspect of the present disclosure, there is provided a computer program product comprising a computer program which, when executed by a processor, implements the model training method or the obstacle distance determination method as described above.

[0011] According to one or more embodiments of the present disclosure, in the training process of the obstacle distance prediction model, different network structures are used to process sample images to obtain 2D height information, 3D height information and distance information between a preset acquisition device and an obstacle of the obstacles in the sample images, so as to determine a value of a first loss function of the model by combining height ranging method and direct ranging method according to the above information, and adjust parameter values of the respective network structures based on the value of the first loss function, thereby solving the problem that loss functions corresponding to different network structures will adversely affect each other in the training process, and improving the prediction accuracy of the prediction model obtained by training.

[0012] It is to be understood that the details set forth herein do not limit the scope of the embodiments of the present disclosure but merely constitute exemplary designations of various alternative embodiments. Other features of the present disclosure will become apparent in the course of the following descriptions and / or in the appended claims. BRIEF DESCRIPTION OF DRAWINGS

[0013] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the principles of the application. The drawings illustrate only exemplary embodiments of the application and therefore, should not be considered to narrow the scope of the application. In the drawings:

[0014] Figure 1 shows a schematic diagram of an exemplary system in which various methods described herein can be implemented according to embodiments of the present disclosure;

[0015] Figure 2 shows a flowchart of a model training method according to embodiments of the present disclosure;

[0016] Figure 3 shows a flowchart of an obstacle distance determination method according to embodiments of the present disclosure;

[0017] Figure 4 shows a structural block diagram of a model training apparatus according to embodiments of the present disclosure;

[0018] Figure 5 shows a structural block diagram of an obstacle distance determination apparatus according to embodiments of the present disclosure; and

[0019] Figure 6 shows a structural block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure. DETAILED DESCRIPTION

[0020] Exemplary embodiments of the present disclosure are described herein with reference to the accompanying drawings in order to be able to provide a clear and complete understanding of the present disclosure. It should be understood that the exemplary embodiments described herein are only for illustrative purposes and are not intended to limit the scope of the present disclosure. It will be readily understood by those skilled in the art that various changes in form and details can be made therein without departing from the scope of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in order to make the present disclosure clear and concise.

[0021] In the present disclosure, the terms “first”, “second”, etc. used in the description of various described examples are not intended to limit the positional relationship, the timing relationship or the importance relationship of the elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element can refer to the same instance of the element, and in some cases, based on the description of the context, they can also refer to different instances.

[0022] The terms used in the description of various described examples in the present disclosure are only for the purpose of describing the specific examples, and are not intended to be limiting. Unless the number of elements is specifically limited by the context, the element can be one or more. In addition, the term “and / or” used in the present disclosure encompasses any one of the listed items and all possible combinations.

[0023] Embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0024] Figure 1 A schematic diagram of an example system 100 in which various methods and apparatus described herein can be implemented according to embodiments of the present disclosure is shown. Referring to Figure 1 , the system 100 comprises one or more client devices 101, 102, 103, 104, 105 and 106, a server 120 and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105 and 106 can be configured to execute one or more application programs.

[0025] In embodiments of the present disclosure, the server 120 can run one or more services or software applications that enable the execution of a model training method and / or an obstacle distance determination method.

[0026] In certain embodiments, the server 120 can also provide other services or software applications, which can include non-virtual environments and virtual environments. In certain embodiments, these services can be provided as web-based services or cloud services, for example to users of the client devices 101, 102, 103, 104, 105 and / or 106 under a software as a service (SaaS) model.

[0027] In Figure 1In the illustrated configuration, the server 120 can include one or more components that implement functionality performed by the server 120. These components can include software components that are executable by one or more processors, hardware components, or a combination thereof. Users operating the client devices 101, 102, 103, 104, 105, and / or 106 can in turn utilize one or more client applications to interact with the server 120 to utilize the services provided by these components. It should be understood that various different system configurations are possible, which can differ from the system 100. Thus, Figure 1 is one example of a system for implementing the various methods described herein and is not intended to be limiting.

[0028] A user can use the client devices 101, 102, 103, 104, 105, and / or 106 to conduct model training or obstacle ranging. The client devices can provide an interface that enables a user of the client device to interact with the client device. The client devices can also output information to the user via the interface. Although Figure 1 Only six client devices are depicted, but one of skill in the art will appreciate that the present disclosure can support any number of client devices.

[0029] The client devices 101, 102, 103, 104, 105, and / or 106 can include various types of computer devices, such as portable handheld devices, general purpose computers (such as personal computers and laptop computers), workstation computers, wearable devices, smart screen devices, self-service kiosk devices, service robots, gaming systems, thin clients, various messaging devices, sensors or other sensing devices, and the like. These computer devices can run various types and versions of software applications and operating systems, such as MICROSOFT Windows, APPLE iOS, UNIX-like operating systems, Linux, or Linux-like operating systems (such as GOOGLE Chrome OS); or include various mobile operating systems, such as MICROSOFT Windows Mobile OS, iOS, Windows Phone, Android. Portable handheld devices can include cellular telephones, smartphones, tablet computers, personal digital assistants (PDAs), and the like. Wearable devices can include head-mounted displays (such as smart glasses) and other devices. Gaming systems can include various handheld gaming devices, Internet-enabled gaming devices, and the like. The client devices are capable of executing various different applications, such as various Internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and can use various communication protocols.

[0030] Network 110 can be any type of network familiar to those skilled in the art, which can support data communications using any of a variety of available protocols, including without limitation TCP / IP, SNA, IPX, etc. As examples only, one or more of networks 110 can be a LAN, an Ethernet network, a Token Ring network, a WAN, the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., a Bluetooth network, a WIFI network), and / or any combination of these and / or other networks.

[0031] Server 120 can include one or more general purpose computers, special purpose server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers, large mainframe computers), server clusters, or any other appropriate arrangement and / or combination. Server 120 can include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that can be virtualized to maintain virtual storage devices for servers). In various embodiments, server 120 can be adapted to run one or more services or software applications providing the functionality described below.

[0032] Computing units in server 120 can run one or more operating systems, including any of the operating systems described above, as well as any commercially available server operating systems. Server 120 can also run any of a variety of additional server applications and / or mid-tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0033] In some embodiments, server 120 can include one or more applications to analyze and consolidate data feeds and / or event updates from users of client devices 101, 102, 103, 104, 105, and 106. Server 120 can also include one or more applications to display the data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105, and 106.

[0034] In some embodiments, server 120 can be a server of a distributed system, or a server in combination with a blockchain. Server 120 can also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. The cloud server is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and virtual private server (VPS, Virtual Private Server) service.

[0035] The system 100 can also include one or more databases 130. In certain embodiments, these databases can be used to store data and other information. For example, one or more of the databases 130 can be used to store distance and uncertainty information for obstacles. The databases 130 can reside in various locations. For example, databases used by the server 120 can be local to the server 120 or can be remote from the server 120 and can communicate with the server 120 via a network- based or dedicated connection. The databases 130 can be of different types. In certain embodiments, databases used by the server 120 can be, for example, relational databases. One or more of these databases can store, update, and retrieve data to and from the databases in response to commands.

[0036] In certain embodiments, one or more of the databases 130 can also be used by applications to store application data. Databases used by applications can be different types of databases, such as key-value stores, object stores, or regular stores backed by file systems.

[0037] Figure 1 The system 100 of FIG. 1 can be configured and operated in various ways to enable the various methods and apparatuses described in accordance with the present disclosure to be applied.

[0038] In an autonomous driving scenario, the information such as the category, size, orientation, and distance of a 3D obstacle can be predicted based on visual image data collected by a camera by training a prediction model. In the related art, a CNN network structure is usually taken as a basic framework of an algorithm model, and multiple data such as an obstacle image collected by a camera, distance information of the obstacle, height information of the obstacle, and camera intrinsic parameters are taken as training data to train the model at the same time, so as to improve the prediction accuracy of the obtained model. However, the uncertainty loss formula used in the traditional prediction algorithm model is not robust enough to abnormal data and is unstable in scale. When the model is trained based on multiple prediction data, the losses corresponding to different training data will affect each other, resulting in a prediction model with low prediction accuracy.

[0039] To this end, Figure 2 A schematic flowchart of a model training method according to an embodiment of the present disclosure is shown. The client or server implementation shown in FIG. 1 can be utilized to implement the method 200 shown in FIG. 2. Figure 1 The method 200 shown in FIG. 2. Figure 2 The method 200 shown in FIG. 2.

[0040] As Figure 2As shown, the embodiment according to the present disclosure provides a model training method 200, comprising: obtaining a sample image containing an obstacle obtained based on a preset acquisition device, and a distance label between the preset acquisition device and the obstacle (step 210); inputting the sample image into a feature extraction network to obtain an image feature (step 220); inputting the image feature into a trained first neural network to obtain a first height of a 2D detection box corresponding to the obstacle (step 230); inputting the image feature or the image feature determined based on the output result of the first neural network into a trained second neural network to obtain a second height of a 3D detection box corresponding to the obstacle, a first distance between the preset acquisition device and the obstacle, a first uncertainty corresponding to the second height, and a second uncertainty corresponding to the first distance (step 240); determining a second distance between the preset acquisition device and the obstacle based on the first height and the second height (step 250); determining a value of a first loss function based on the first distance, the second uncertainty, the second distance, the first uncertainty, and the distance label (step 260); and adjusting a parameter value of at least one of the feature extraction network, the first neural network, and the second neural network based on the value of the first loss function (step 270).

[0041] The model training method according to the embodiment of the present disclosure, in the training process of the obstacle distance prediction model, utilizes different network structures to process sample images to obtain 2D height information, 3D height information, and distance information between a preset acquisition device and an obstacle of the obstacle in the sample images, so as to determine a value of a first loss function of the model by combining height ranging and direct ranging according to the above information, and adjust a parameter value of the network structure based on the value of the first loss function, thereby solving the problem that loss functions corresponding to different network structures will adversely affect each other in the training process, and improving the prediction accuracy of the prediction model obtained by training.

[0042] In step 210, a sample image containing an obstacle obtained based on a preset acquisition device, and a distance label between the preset acquisition device and the obstacle are obtained.

[0043] In some embodiments, the preset acquisition device can be arranged on an autonomous vehicle, or can be arranged on a roadside of a road on which the autonomous vehicle travels, and the present disclosure does not limit this.

[0044] In some embodiments, the preset acquisition device includes but is not limited to an image or video acquisition device such as a video camera and a camera, and the present disclosure does not limit this. When the preset acquisition device is a video acquisition device, the sample image can be a corresponding video frame.

[0045] In an example, a preset acquisition device can be arranged on the autonomous vehicle to obtain a sample image containing an obstacle, and a ranging device such as a laser radar can be used to obtain a distance between the autonomous vehicle and the obstacle, so as to generate the distance label.

[0046] It should be understood that the description of the preset acquisition device herein is only illustrative and does not limit the preset acquisition device to this.

[0047] According to some embodiments, step 210 comprises: obtaining a distance reference value between the preset acquisition device and the obstacle, and a parameter of the preset acquisition device; and determining the distance label based on a ratio of the distance reference value and the parameter. Based on this, the camera intrinsic parameter can be avoided to be introduced when generating the distance label, so as to decouple the distance information and the camera intrinsic parameter, thereby improving the generality of the obtained algorithm model among different hardware platforms.

[0048] According to some embodiments, the first height, the second height, the first distance, and the second distance are all represented in a logarithmic form. Based on this, the gradient scale invariance of the log domain can be fully utilized to optimize a plurality of loss functions corresponding to a plurality of network structures, so as to have a positive effect in the quantization process and reduce the degree of mutual influence among the plurality of loss functions.

[0049] In an example, the preset acquisition device comprises a camera, and the distance ground truth between the preset acquisition device and the obstacle is d real . The distance ground truth d real may be subjected to log domain transformation and camera intrinsic parameter decoupling to obtain a distance label d gt . For example, the distance label d gt is determined based on the following formula:

[0050]

[0051] wherein f y represents a focal length of the camera in the height direction.

[0052] In step 220, the sample image is input into a feature extraction network to obtain image features.

[0053] In some embodiments, the feature extraction network comprises but is not limited to a convolutional neural network, etc. It can be understood that the feature extraction network can adopt other types of network structures, which are not specifically limited herein.

[0054] In step 230, the image features are input into the trained first neural network to obtain a first height of a 2D detection box corresponding to the obstacle.

[0055] In some embodiments, the first neural network described above can be constructed based on a deep learning neural network such as a DNN, a CNN, an LSTM, a Conformer, a TDNN, etc. It can be understood that the first neural network can adopt other types of network structures, which are not specifically limited herein.

[0056] In step 240, the image features or the image features determined based on the output results of the first neural network are input into the trained second neural network to obtain a second height of the 3D detection box corresponding to the obstacle, a first distance between the preset acquisition device and the obstacle, a first uncertainty corresponding to the second height, and a second uncertainty corresponding to the first distance.

[0057] In some embodiments, when the second neural network is used to realize 3D detection of the obstacle, the input of the second neural network can be the image features of the sample image extracted, or the 2D detection result obtained through the first neural network, which is not limited in the present disclosure. When the input of the second neural network is the image features of the 2D detection result obtained through the first neural network, such as the image features determined based on the output 2D detection box, the second neural network further fuses the 2D detection result, so as to more accurately determine the information of the 3D detection box thereof.

[0058] In some embodiments, the 2D detection result output by the first neural network can be processed through an algorithm such as Non-Maximum Suppression (NMS) or ROIAlign, so as to input the processed image features into the second neural network, and further obtain the 3D detection result of the obstacle.

[0059] In some embodiments, the second neural network described above can be constructed based on a deep learning neural network such as a DNN, a CNN, an LSTM, a Conformer, a TDNN, etc. It can be understood that the second neural network can adopt other types of network structures, which are not specifically limited herein.

[0060] In step 250, a second distance between the preset acquisition device and the obstacle is determined based on the first height and the second height.

[0061] According to some embodiments, the second distance d between the preset acquisition device and the obstacle can be determined based on the following formula through the height ranging method geo :

[0062] d geo = log(H) - log(h)

[0063] wherein log(H) is the second height in logarithmic form, and log(h) is the first height in logarithmic form.

[0064] In some embodiments, the second neural network comprises a height ranging branch and a direct ranging branch. And a size sub-branch.

[0065] In an example, the output of the size sub-branch comprises size information of the 3D bounding box of the obstacle. For example, the output dimension of the size sub-branch is 4, and the output components comprise a second height log(H) of the 3D bounding box corresponding to the obstacle, a width log(W) of the 3D bounding box corresponding to the obstacle, and a length log(L) of the 3D bounding box corresponding to the obstacle.

[0066] In an example, the height ranging sub-branch is configured to output a first uncertainty corresponding to the second height log(H).

[0067] In an example, the output of the direct ranging branch comprises distance information between the preset acquisition device and the obstacle. For example, the output dimension of the direct ranging branch is 2, and the output components comprise a first distance d direct and a second uncertainty u direct corresponding to the first distance.

[0068] It should be understood that the above description of the branches in the second neural network is only for illustration and does not limit it thereto.

[0069] In step 260, a value of a first loss function is determined based on the first distance, the second uncertainty, the second distance, the first uncertainty, and the distance label.

[0070] According to some embodiments, the first loss function comprises a second loss function and a third loss function. The second loss function is determined based on the first distance, the second uncertainty, and the distance label; and the third loss function is determined based on the second distance, the first uncertainty, and the distance label.

[0071] For example, the second loss function and the third loss function can be added to obtain the first loss function. Alternatively, the second loss function and the third loss function can also be added according to corresponding weights to obtain the first loss function, which is not limited herein.

[0072] According to some embodiments, at least one of the second loss function and the third loss function can be determined based on the following formula:

[0073]

[0074] wherein, λ is a preset parameter value, σ is a corresponding uncertainty in the first uncertainty and the second uncertainty, d pred is one of the first distance and the second distance corresponding to σ, and d gtThe distance label. The parameter λ can be related to factors such as camera focal length and training data, for example, set to an empirical value of 0.001.

[0075] In some examples, multiple obstacles can be included. Therefore, in the above loss function determination formula, d pred is a tensor including distance prediction values of each of the multiple obstacles; d gt is a tensor including distance labels of each of the multiple obstacles; and σ is a tensor including corresponding respective uncertainties of each of the multiple obstacles.

[0076] At this time, the parameter w can be set as shown below:

[0077]

[0078] where N is the number of obstacles, and σ i is the corresponding respective uncertainty of the i th obstacle.

[0079] Generally, at least one of the second loss function and the third loss function can also be represented based on the following formula:

[0080]

[0081] However, the above loss function can have the following problems: (1) the first part of the function the derivative at 0 does not exist; (2) the uncertainty σ will continuously decrease during the optimization process, causing scale instability, which will adversely affect other loss functions; (3) there is a lack of weight adjustment between the first part and the second part (logσ) of the function, resulting in suboptimal optimization results.

[0082] The form of the loss function in the above embodiment avoids the problem of non-existent derivative, expands the scope of application; secondly, for each term in the function, a regularization balance parameter λ is added as a weight, so that the importance of each part of the function can be balanced, and the optimization effect based on the loss function is improved.

[0083] When multiple obstacles are included, the average value of the uncertainty is used to normalize the entire function, so that the first loss function will not cause scale expansion as the uncertainty decreases during the optimization process, effectively improving the prediction accuracy of the model obtained by training.

[0084] In step 270, the parameter value of at least one of the feature extraction network, the first neural network, and the second neural network is adjusted based on the value of the first loss function.

[0085] Figure 3FIG. 1 shows a flowchart of a method for determining an obstacle distance according to an embodiment of the present disclosure.

[0086] Referring to Figure 3 According to an embodiment of the present disclosure, there is provided a method 300 for determining an obstacle distance, comprising: obtaining an image including an obstacle based on a preset acquisition device (step 310); inputting the image into a feature extraction network to obtain image features (step 320); inputting the image features into a trained first neural network to obtain a first height of a 2D detection box corresponding to the obstacle (step 330); inputting the image features or image features determined based on an output result of the first neural network into a trained second neural network to obtain a second height of a 3D detection box corresponding to the obstacle, a first distance between the preset acquisition device and the obstacle, a first uncertainty corresponding to the second height, and a second uncertainty corresponding to the first distance (step 340); determining a second distance between the preset acquisition device and the obstacle based on the first height and the second height (step 350); and fusing the first distance and the second distance based on the first uncertainty and the second uncertainty to take a fused distance as a distance between the preset acquisition device and the obstacle (step 360).

[0087] The method for determining an obstacle distance according to an embodiment of the present disclosure processes an image including an obstacle by using different network structures to obtain 2D height information, 3D height information, and distance information between a preset acquisition device and the obstacle of the obstacle in the image, and determines a first distance and a second distance based on the above information, combines height ranging and direct ranging to obtain a result after fusing the first distance and the second distance as an obstacle ranging result, thereby improving the prediction accuracy of the distance of the obstacle in the automatic driving perception process, increasing the recall rate of the obstacle in the automatic driving perception, providing more reliable perception results for the downstream, and improving the accuracy of the automatic driving and the user experience.

[0088] It should be understood that steps 310-350 are similar to steps 210-250 described above, and will not be described again here.

[0089] According to some embodiments, the feature extraction network, the first neural network, and the second neural network are trained by the model training method described above. Therefore, reference can be made to the detailed description of the model training method above, and will not be described again here.

[0090] According to some embodiments, step 360 comprises: normalizing the first uncertainty and the second uncertainty; and taking the normalized first uncertainty and the second uncertainty as coefficients to perform weighted summation on the first distance and the second distance to determine the fused distance based on a result after the summation.

[0091] In an example, the summed result d fusion :

[0092]

[0093] where w direct = exp(-u direct )(log is in base e), w geo = exp(-u geo )(log is in base e), d direct is a first distance between the preset acquisition device and the obstacle, u direct is a second uncertainty corresponding to the first distance, d geo is a second distance between the preset acquisition device and the obstacle, u geo is a first uncertainty corresponding to the second height.

[0094] In some embodiments, the preset acquisition device includes a camera. Since the summed result d fusion is in the log domain and the camera intrinsic parameter decoupling is achieved, the fused distance d final may be further determined as:

[0095] d final = exp(d fusion )*f y

[0096] where f y is the focal length of the camera in the height direction. Based on this, the accuracy and precision of the obtained fused distance are effectively improved.

[0097] Figure 4 A structural block diagram of a model training device according to an embodiment of the present disclosure is shown.

[0098] As Figure 4As shown, according to an embodiment of the present disclosure, a model training apparatus 400 is provided, comprising: a first acquisition unit 410 configured to acquire a sample image containing an obstacle obtained based on a preset acquisition device and a distance label between the preset acquisition device and the obstacle; a first feature module 420 configured to input the sample image into a feature extraction network to obtain an image feature; a first determination unit 430 configured to input the image feature into a trained first neural network to obtain a first height of a 2D detection box corresponding to the obstacle; a second determination unit 440 configured to input the image feature or an image feature determined based on an output result of the first neural network into a trained second neural network to obtain a second height of a 3D detection box corresponding to the obstacle, a first distance between the preset acquisition device and the obstacle, a first uncertainty corresponding to the second height, and a second uncertainty corresponding to the first distance; a third determination unit 450 configured to determine a second distance between the preset acquisition device and the obstacle based on the first height and the second height; a fourth determination unit 460 configured to determine a value of a first loss function based on the first distance, the second uncertainty, the second distance, the first uncertainty, and the distance label; and a parameter adjustment unit 470 configured to adjust a parameter value of at least one of the feature extraction network, the first neural network, and the second neural network based on the value of the first loss function.

[0099] Here, the operations of the above-mentioned units 410-470 of the model training apparatus 400 are similar to the operations of the steps 210-270 described above, and will not be described here again.

[0100] Figure 5 A structural block diagram of an obstacle distance determination apparatus according to an embodiment of the present disclosure is shown.

[0101] As Figure 5As shown, according to an embodiment of the present disclosure, a barrier distance determination apparatus 500 is provided, comprising: a second acquisition unit 510 configured to acquire an image including a barrier obtained based on a preset acquisition device; a second feature unit 520 configured to input the image into a feature extraction network to obtain an image feature; a fifth determination unit 530 configured to input the image feature into a trained first neural network to obtain a first height of a 2D detection box corresponding to the barrier; a sixth determination unit 540 configured to input the image feature or the image feature determined based on an output result of the first neural network into a trained second neural network to obtain a second height of a 3D detection box corresponding to the barrier, a first distance between the preset acquisition device and the barrier, a first uncertainty corresponding to the second height, and a second uncertainty corresponding to the first distance; a seventh determination unit 550 configured to determine a second distance between the preset acquisition device and the barrier based on the first height and the second height; and a distance fusion unit 560 configured to fuse the first distance and the second distance based on the first uncertainty and the second uncertainty to take a fused distance as a distance between the preset acquisition device and the barrier.

[0102] Here, the operations of the above-mentioned units 510-560 of the barrier distance determination apparatus 500 are similar to the operations of the steps 310-360 described above, and will not be repeated here.

[0103] According to an embodiment of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the model training method or the barrier distance determination method as described above.

[0104] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to perform the model training method or the barrier distance determination method as described above.

[0105] According to an embodiment of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program, when executed by a processor, implements the model training method or the barrier distance determination method as described above.

[0106] Reference Figure 6The present invention describes a structural block diagram of an electronic device 600 that can serve as a server or client of the present disclosure, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0107] like Figure 6 As shown, the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. The RAM 603 may also store various programs and data required for the operation of the electronic device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0108] Multiple components in electronic device 600 are connected to I / O interface 605, including: input unit 606, output unit 607, storage unit 608, and communication unit 609. Input unit 606 can be any type of device capable of inputting information to electronic device 600. Input unit 606 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of electronic device, and can include, but is not limited to, a mouse, keyboard, touchscreen, trackpad, trackball, joystick, microphone, and / or remote control. Output unit 607 can be any type of device capable of presenting information, and can include, but is not limited to, a monitor, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 608 can include, but is not limited to, disk and optical disk. Communication unit 609 allows electronic device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and can include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth devices, 802.11 devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0109] The computing unit 601 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs various methods and processes described above, such as the method 666. For example, in some embodiments, the method 666 can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded onto the RAM 603 and executed by the computing unit 601, one or more steps of the method 666 described above can be performed. Alternatively, in other embodiments, the computing unit 601 can be configured to perform the method 666 by any other suitable means, such as by means of firmware.

[0110] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0111] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces a means for implementing the functions / acts specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0112] In the context of this disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0113] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0114] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0115] The computer system can include clients and servers. This relationship can be remote or on-site. The servers can be cloud servers, servers of a distributed system, or servers combined with a blockchain.

[0116] It should be understood that the various forms of flow shown above can be used to reorder, add, or remove steps. For example, the steps described in the present disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, which are not limited herein.

[0117] Although embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-described methods, systems, and devices are merely exemplary embodiments or examples, and the scope of the present disclosure is not limited by these embodiments or examples, but only by the granted claims and their equivalent scope. Various elements in the embodiments or examples can be omitted or replaced by equivalent elements. In addition, each step can be performed in an order different from that described in the present disclosure. Further, various elements in the embodiments or examples can be combined in various ways. It is important that many of the elements described herein can be replaced by equivalent elements that appear after the present disclosure as technology evolves.

Claims

1. A model training method, comprising: obtaining a sample image containing an obstacle and a distance label between a preset acquisition device and the obstacle; inputting the sample image into a feature extraction network to obtain image features; inputting the image features into a trained first neural network to obtain a first height of a 2D bounding box corresponding to the obstacle; inputting the image features or image features determined based on an output result of the first neural network into a trained second neural network to obtain a second height of a 3D bounding box corresponding to the obstacle, a first distance between the preset acquisition device and the obstacle, a first uncertainty corresponding to the second height, and a second uncertainty corresponding to the first distance; determining a second distance between the preset acquisition device and the obstacle based on the first height and the second height; determining a value of a first loss function based on the first distance, the second uncertainty, the second distance, the first uncertainty, and the distance label; and adjusting a parameter value of at least one of the feature extraction network, the first neural network, and the second neural network based on the value of the first loss function. The first height, the second height, the first distance, and the second distance are represented in a logarithmic form.

2. The method of claim 1, wherein, Obtaining a sample image containing an obstacle and a distance label between a preset acquisition device and the obstacle comprises:

3. The method of claim 1, wherein, obtaining a distance reference value between the preset acquisition device and the obstacle, and a parameter of the preset acquisition device; and determining the distance label based on a ratio of the distance reference value and the parameter. Wherein, log(H) is the second height in a logarithmic form, and log(h) is the first height in a logarithmic form.

4. The method of claim 2, wherein, A second distance d between the preset collection device and the obstacle is determined based on the following formula geo : d geo = log(H) - log(h) The first loss function comprises a second loss function and a third loss function, and wherein, 5. The method of claim 1, wherein, the second loss function is determined based on the first distance, the second uncertainty, and the distance label; the third loss function is determined based on the second distance, the first uncertainty, and the distance label. At least one of the second loss function and the third loss function is determined based on the following formula:

6. The method of claim 5, wherein, 7.A method for determining an obstacle distance, comprising: wherein, λ is a preset parameter value, σ is a respective one of the first and second uncertainties, d pred is one of the first and second distances corresponding to σ, d gt is the distance label. obtaining an image containing an obstacle based on a preset acquisition device; inputting the image into a feature extraction network to obtain image features; inputting the image features into a trained first neural network to obtain a first height of a 2D bounding box corresponding to the obstacle; inputting the image features or image features determined based on an output result of the first neural network into a trained second neural network to obtain a second height of a 3D bounding box corresponding to the obstacle, a first distance between the preset acquisition device and the obstacle, a first uncertainty corresponding to the second height, and a second uncertainty corresponding to the first distance; determining a second distance between the preset acquisition device and the obstacle based on the first height and the second height; and ​ ​ fuse the first distance and the second distance based on the first uncertainty and the second uncertainty to obtain a fused distance as the distance between the preset acquisition device and the obstacle.

8. The method of claim 7, wherein, The feature extraction network, the first neural network and the second neural network are trained by the method in any one of claims 1-6.

9. The method of claim 7, wherein, The fusing the first distance and the second distance based on the first uncertainty and the second uncertainty comprises: normalizing the first uncertainty and the second uncertainty; and performing weighted summation on the first distance and the second distance by taking the normalized first uncertainty and the normalized second uncertainty as coefficients to determine the fused distance based on a result of the summation.

10. A model training apparatus, comprising: a first obtaining unit configured to obtain a sample image containing an obstacle and a distance label between a preset acquisition device and the obstacle, which are obtained based on the preset acquisition device; a first feature module configured to input the sample image into a feature extraction network to obtain image features; a first determining unit configured to input the image features into a trained first neural network to obtain a first height of a 2-dimensional detection box corresponding to the obstacle; a second determining unit configured to input the image features or image features determined based on an output result of the first neural network into a trained second neural network to obtain a second height of a 3-dimensional detection box corresponding to the obstacle, a first distance between the preset acquisition device and the obstacle, a first uncertainty corresponding to the second height, and a second uncertainty corresponding to the first distance; a third determining unit configured to determine a second distance between the preset acquisition device and the obstacle based on the first height and the second height; a fourth determining unit configured to determine a value of a first loss function based on the first distance, the second uncertainty, the second distance, the first uncertainty and the distance label; and a parameter adjusting unit configured to adjust a parameter value of at least one of the feature extraction network, the first neural network and the second neural network based on the value of the first loss function. The first height, the second height, the first distance and the second distance are represented in a logarithmic form.

11. The apparatus of claim 10, wherein, The first obtaining unit comprises:

12. The apparatus of claim 10, wherein, a parameter obtaining sub-module configured to obtain a distance reference value between the preset acquisition device and the obstacle and parameters of the preset acquisition device; and a distance label determining sub-module configured to determine the distance label based on a ratio of the distance reference value and the parameters. Wherein, log(H) is the second height in a logarithmic form, and log(h) is the first height in a logarithmic form.

13. The apparatus of claim 12, wherein, A second distance d between the preset collection device and the obstacle is determined based on the following formula geo : d geo = log(H) - log(h) The first loss function comprises a second loss function and a third loss function, and wherein, 14. The apparatus of claim 10, wherein, the second loss function is determined based on the first distance, the second uncertainty and the distance label; the third loss function is determined based on the second distance, the first uncertainty and the distance label. ​ 15. The apparatus of claim 14, wherein, At least one of the second loss function and the third loss function is determined based on the following formula: wherein, λ is a preset parameter value, σ is the first uncertainty and the second uncertainty, d pred is one of the first distance and the second distance corresponding to σ, d gt is the distance label.

16. An obstacle distance determination apparatus, comprising: a second acquisition unit configured to acquire an image including an obstacle obtained based on a preset acquisition device; a second feature unit configured to input the image into a feature extraction network to obtain an image feature; a fifth determination unit configured to input the image feature into a trained first neural network to obtain a first height of a 2-dimensional detection box corresponding to the obstacle; a sixth determination unit configured to input the image feature or an image feature determined based on an output result of the first neural network into a trained second neural network to obtain a second height of a 3-dimensional detection box corresponding to the obstacle, a first distance between the preset acquisition device and the obstacle, a first uncertainty corresponding to the second height, and a second uncertainty corresponding to the first distance; a seventh determination unit configured to determine a second distance between the preset acquisition device and the obstacle based on the first height and the second height; and a distance fusion unit configured to fuse the first distance and the second distance based on the first uncertainty and the second uncertainty to obtain a fused distance as a distance between the preset acquisition device and the obstacle. The feature extraction network, the first neural network, and the second neural network are trained by the method of any one of claims 1-6.

17. The apparatus of claim 16, wherein, The distance fusion unit comprises:

18. The apparatus of claim 16, wherein, a normalization unit configured to normalize the first uncertainty and the second uncertainty; and a distance determination unit configured to weight and sum the first distance and the second distance by using the normalized first uncertainty and the normalized second uncertainty as coefficients to determine the fused distance based on a result of the weighting and summing.

19. An electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-9. The computer instructions are used to enable the computer to perform the method of any one of claims 1-9.

20. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer program, when executed by a processor, implements the method of any one of claims 1-9.

21. A computer program product comprising a computer program, wherein, The computer program, when executed by a processor, implements the method of any one of claims 1-9.

Images