Model evaluation method, apparatus, device, medium, and product

Abstract

The application discloses a model evaluation method, device, equipment, medium and product. The method comprises the following steps: acquiring a plurality of image frames and corresponding laser point cloud data; projecting the laser point cloud data corresponding to the image frames to a two-dimensional image plane where the image frames are located, to obtain a projection image corresponding to the image frames; inputting the plurality of image frames into a monocular three-dimensional time sequence model to be tested, to obtain a prediction result of depth prediction for each image frame; calculating a scaling scale according to a prediction depth, a reference depth and a depth uncertainty corresponding to a plurality of effective pixel points in each image frame; correcting the prediction depth corresponding to the plurality of effective pixel points according to the scaling scale, to obtain a corrected depth corresponding to each effective pixel point; and determining an evaluation result of the monocular three-dimensional time sequence model according to the corrected depth corresponding to the effective pixel points and the reference depth corresponding to the effective pixel points, so as to realize quantitative evaluation of the performance of the monocular three-dimensional time sequence model.

Description

Technical Field

[0001] This application belongs to the field of image technology, and in particular relates to a model evaluation method, apparatus, device, medium and product. Background Technology

[0002] In the development of full-stack autonomous driving systems, the verification of intelligent transportation digital twins, and advanced driver assistance systems, constructing high-fidelity 3D simulation environments using real-world road data has become a core component. In recent years, with the explosion of deep learning and implicit neural representation technologies, generative artificial intelligence algorithms, represented by 3D Gaussian Splatting (3DGS) and various monocular 3D temporal models, have demonstrated a powerful ability to reconstruct the 3D physical world from monocular video sequences alone. These monocular 3D temporal models can infer the camera's relative extrinsic parameter matrix, intrinsic parameter matrix, full-density depth map, and 3D spatial point cloud from a continuous 2D image stream.

[0003] However, in the practice of autonomous driving engineering, how to quantitatively evaluate the performance of these monocular 3D time-series models has become a technical bottleneck that urgently needs to be solved. Summary of the Invention

[0004] This application provides a model evaluation method, apparatus, device, medium, and product that can quantitatively evaluate the performance of a monocular three-dimensional temporal model.

[0005] In a first aspect, embodiments of this application provide a model evaluation method, the method comprising: Acquire multiple image frames and the laser point cloud data corresponding to each image frame; For each image frame, the laser point cloud data corresponding to the image frame is projected onto the two-dimensional image plane where the image frame is located to obtain the projected image corresponding to the image frame, wherein the projected image includes multiple effective pixels and a reference depth corresponding to each effective pixel; Multiple image frames are input into a monocular 3D temporal model to be tested, and the prediction results of the monocular 3D temporal model for depth prediction of each image frame are obtained. The prediction results include the predicted depth of each pixel in the image frame and the depth uncertainty of each predicted depth. The scaling scale is calculated based on the predicted depth corresponding to multiple effective pixels in each image frame, the reference depth corresponding to multiple effective pixels, and the depth uncertainty corresponding to multiple effective pixels. The predicted depths corresponding to multiple effective pixels are corrected according to the scaling scale to obtain the corrected depths corresponding to each effective pixel. The evaluation result of the monocular 3D temporal model is determined based on the correction depth corresponding to the effective pixel and the reference depth corresponding to the effective pixel.

[0006] Secondly, embodiments of this application provide a model evaluation apparatus, the apparatus comprising: The acquisition module is used to acquire multiple image frames and laser point cloud data corresponding to each image frame; The projection module is used to project the laser point cloud data corresponding to each image frame onto the two-dimensional image plane where the image frame is located, so as to obtain the projected image corresponding to the image frame. The projected image includes a plurality of effective pixels and a reference depth corresponding to each effective pixel. The prediction module is used to input multiple image frames into a monocular 3D temporal model to be tested, and obtain the prediction result of the monocular 3D temporal model for depth prediction of each image frame. The prediction result includes the predicted depth of each pixel in the image frame and the depth uncertainty of each predicted depth. The calculation module is used to calculate the scaling scale based on the predicted depth corresponding to multiple effective pixels in each image frame, the reference depth corresponding to multiple effective pixels, and the depth uncertainty corresponding to multiple effective pixels. The correction module is used to correct the predicted depth corresponding to the multiple effective pixels according to the scaling scale, so as to obtain the corrected depth corresponding to each effective pixel. The determination module is used to determine the evaluation result of the monocular three-dimensional temporal model based on the correction depth corresponding to the effective pixel and the reference depth corresponding to the effective pixel.

[0007] Thirdly, embodiments of this application provide an electronic device, including: a processor and a memory storing computer program instructions; the processor executes the computer program instructions to implement the model evaluation method as described in the first aspect.

[0008] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the model evaluation method as described in the first aspect.

[0009] Fifthly, embodiments of this application provide a computer program product in which instructions, when executed by a processor of an electronic device, cause the electronic device to perform the model evaluation method as described in the first aspect.

[0010] In this embodiment, laser point cloud data is projected onto a two-dimensional image plane to provide an accurate reference depth for each effective pixel, thus compensating for the lack of absolute scale in monocular 3D temporal models and providing a reliable benchmark for evaluation. Using the reference depth provided by the laser point cloud data, a scaling scale S is calculated, converting the relative depth output by the monocular 3D temporal model (the predicted depth output by the model is a relative depth) into an absolute physical depth. This gives the output of the monocular 3D temporal model a real physical meaning. Furthermore, various quantitative indicators can be calculated using the correction depth and reference depth corresponding to the effective pixels, providing a clear and quantifiable basis for optimizing the performance of the monocular 3D temporal model and enabling a quantitative evaluation of its performance. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart illustrating the model evaluation method provided in the embodiments of this application; Figure 2 This is a schematic diagram of the structure of the model evaluation device provided in the embodiments of this application; Figure 3 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0013] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0014] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0015] In all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. Additionally, when embodiments of this application require access to sensitive personal information, separate permission or consent from the user is obtained through pop-ups or redirects to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments obtained.

[0016] The model evaluation method provided in the embodiments of this application will be introduced first below.

[0017] Figure 1 A schematic flowchart of a model evaluation method provided in one embodiment of this application is shown. Figure 1 As shown, the model evaluation method provided in this application embodiment includes the following steps 101-106, wherein: Step 101: Acquire multiple image frames and the laser point cloud data corresponding to each image frame.

[0018] Multiple image frames can refer to temporally consecutive image frames within the same sliding window (also known as a temporal window). For example, in autonomous driving, cameras on the vehicle continuously capture images of the environment. A sliding window algorithm is used to segment the continuous video stream into independent temporal evaluation scenarios, each of which includes multiple image frames. Simultaneously with image acquisition, radar on the vehicle also collects laser point cloud data of the same area. After obtaining the image frames and laser point cloud data, strict timestamp synchronization is performed to ensure a one-to-one correspondence between the image frames and the laser point cloud data. Specific implementation methods can be found in relevant technical documentation and are not limited here.

[0019] In addition, after acquiring the image frame, a two-dimensional semantic mask that distinguishes between rigid background and moving target can be generated using an image segmentation convolutional neural network. Based on this two-dimensional semantic mask, it can be determined which pixels in the image frame are static pixels and which pixels are dynamic pixels.

[0020] Step 102: For each image frame, project the laser point cloud data corresponding to the image frame onto the two-dimensional image plane where the image frame is located to obtain the projected image corresponding to the image frame, wherein the projected image includes multiple effective pixels and a reference depth corresponding to each effective pixel.

[0021] For example, the point cloud is initially located in the vehicle's own coordinate system. The pose transformation matrix from the vehicle's coordinate system to the world coordinate system is obtained, along with the extrinsic parameter matrix from the vehicle's coordinate system to the target camera's coordinate system. By multiplying these matrices, a comprehensive transformation matrix is ​​derived to transform the world coordinate system to the target camera's coordinate system. For any three-dimensional physical reference point located in the world coordinate system The spatial rigid body pose transformation process follows the following homogeneous coordinate transformation formula:

[0022] In the rigid body transformation operations described above, the three-dimensional physical reference point is represented by a four-dimensional homogeneous coordinate column vector. By transforming the matrix This can be mapped to the homogeneous three-dimensional physical coordinates of the point in the target camera coordinate system. .

[0023] After completing the rigid body alignment in 3D space, call the intrinsic parameter matrix of the target camera. Performing perspective projection operations reduces the dimensionality of three-dimensional physical coordinate points to horizontal and vertical discrete pixel coordinate values ​​on a two-dimensional image plane. and Its perspective projection physical equation is defined as follows:

[0024] In this perspective projection equation, the horizontal and vertical physical distance components of a 3D point in the target camera coordinate system , and vertical physical depth component Construct the input vector. Call the target camera's intrinsic parameter matrix. Perform matrix multiplication to filter out depth components. These are the invisible data points located behind the camera.

[0025] For details on the above coordinate transformation and projection processes, please refer to the relevant technical records, which will not be elaborated here.

[0026] To address the background penetration error caused by multiple points mapping to the same pixel location (i.e., a pixel), in one embodiment of this application, the laser point cloud data corresponding to the image frame is projected onto the two-dimensional image plane where the image frame is located to obtain the projected image corresponding to the image frame, including: The laser point cloud data corresponding to the image frame is projected onto the two-dimensional image plane where the image frame is located. If at least one point in the laser point cloud data is projected onto the same pixel, the minimum depth corresponding to at least one of the points is taken as the reference depth corresponding to the pixel, and the pixel is taken as the effective pixel of the projected image.

[0027] In other words, when multiple points project onto the same pixel location in the projected image due to perspective, the vertical physical depths of these points in the target camera coordinate system are extracted and sorted in ascending order. Only the minimum depth value at the top is extracted and assigned to that pixel location to obtain the projected image. Simultaneously, a corresponding binary type two-dimensional effective mask matrix is ​​generated. Pixel locations in the projected image that are successfully assigned the minimum depth value are marked as logically true (pixel locations marked as logically true are valid pixels), while the remaining locations are marked as logically false.

[0028] The projected image obtained through the above processing avoids the error of background penetrating the foreground. The projected image can be regarded as a sparse depth reference map. The values ​​of the effective pixels in the reference map can be used as reference depth for subsequent use, for example, to evaluate or calibrate the depth prediction results output by the monocular 3D temporal model.

[0029] Step 103: Input multiple image frames into the monocular 3D temporal model to be tested, and obtain the prediction result of the monocular 3D temporal model for depth prediction of each image frame. The prediction result includes the predicted depth of each pixel in the image frame and the depth uncertainty of each predicted depth.

[0030] Generative artificial intelligence algorithms, represented by 3D Gaussian Splatting (3DGS) and various monocular 3D temporal models, have demonstrated a powerful ability to reconstruct the 3D physical world from monocular video sequences alone. These monocular 3D temporal models can infer the camera's relative extrinsic parameter matrix, intrinsic parameter matrix, full-dense depth map, and 3D spatial point cloud from a continuous 2D image stream.

[0031] In this embodiment, the monocular 3D temporal model can obtain the predicted depth of each pixel in the image frame output by the monocular 3D temporal model, and the depth uncertainty of each predicted depth, based on multiple image frames. The greater the depth uncertainty, the less basis the monocular 3D temporal model has in predicting the depth of that pixel.

[0032] Step 104: Calculate the scaling scale based on the predicted depth corresponding to the multiple effective pixels in each image frame, the reference depth corresponding to the multiple effective pixels, and the depth uncertainty corresponding to the multiple effective pixels.

[0033] Since a monocular camera cannot directly obtain absolute depth, a monocular 3D temporal model can only predict relative depth (e.g., "object A is twice as far as object B," but the exact distance is unknown). However, laser point clouds can provide sparse but accurate absolute depth. Based on these two factors, in this embodiment, the aim is to find an optimal scaling factor (i.e., scaling scale) that makes the relative depth (i.e., predicted depth) output by the monocular 3D temporal model as close as possible to the absolute depth (i.e., reference depth) provided by the laser point cloud. Therefore, in this step, the scaling scale can be treated as an unknown parameter. The scaling scale is multiplied by the predicted depth corresponding to the effective pixel, and the distance between the product and the reference depth corresponding to the effective pixel is calculated. A weighting coefficient is determined based on the depth uncertainty corresponding to the effective pixel. The weighting coefficient is multiplied by the distance to obtain the target value of the effective pixel. The target values ​​of each effective pixel are accumulated, and the scaling scale value that minimizes the accumulated target value is calculated.

[0034] Step 105: Correct the predicted depth corresponding to the multiple effective pixels according to the scaling scale to obtain the corrected depth corresponding to each effective pixel.

[0035] For example, multiplying the scaling factor by the predicted depth corresponding to the effective pixels yields the corrected depth corresponding to the effective pixels.

[0036] Step 106: Determine the evaluation result of the monocular 3D temporal model based on the correction depth corresponding to the effective pixel and the reference depth corresponding to the effective pixel.

[0037] For example, the absolute value of the difference between the correction depth corresponding to the effective pixel and the reference depth corresponding to the effective pixel is calculated, and the sum of the absolute values ​​corresponding to each effective pixel is used as the evaluation result of the monocular three-dimensional temporal model.

[0038] For example, the absolute value of the difference between the corrected depth corresponding to the effective pixel and the reference depth corresponding to the effective pixel is divided by the reference depth corresponding to the effective pixel to obtain the depth error corresponding to the effective pixel. The mean depth error corresponding to each effective pixel is used as the evaluation result of the monocular 3D temporal model.

[0039] The depth error corresponding to the effective pixels obtained through the above processing, expressed as a dimensionless percentage deviation, can objectively reflect the accuracy of the corrected depth. The mean of the depth error corresponding to each effective pixel is used as the evaluation result of the monocular 3D temporal model, achieving a quantitative and interpretable evaluation of the monocular 3D model, facilitating subsequent optimization and adjustment by developers.

[0040] Steps 101-106 above project the laser point cloud data onto a two-dimensional image plane, providing an accurate reference depth for each effective pixel. This compensates for the lack of absolute scale in monocular 3D temporal models, providing a reliable benchmark for evaluation. Using the reference depth provided by the laser point cloud data, the scaling scale S is calculated, converting the relative depth output by the monocular 3D temporal model (the predicted depth output by the model is relative) into an absolute physical depth. This gives the output of the monocular 3D temporal model a real physical meaning. Furthermore, various quantitative indicators can be calculated using the correction depth and reference depth corresponding to the effective pixels, providing a clear and quantifiable basis for optimizing the performance of the monocular 3D temporal model and enabling quantitative evaluation of the monocular 3D temporal model.

[0041] In one embodiment of this application, a scaling scale is calculated based on the predicted depth corresponding to a plurality of effective pixels in each image frame, the reference depth corresponding to a plurality of effective pixels, and the depth uncertainty corresponding to a plurality of effective pixels, including: For each image frame, the static prior weight of each pixel in the image frame is obtained. The static prior weight is used to identify static pixels and dynamic pixels by using different values. For each effective pixel, a weight coefficient is constructed based on the static prior weight and the depth uncertainty corresponding to the effective pixel. The weight coefficient corresponding to the effective pixel is negatively correlated with the depth uncertainty corresponding to the effective pixel. An objective function is constructed based on the weight coefficients corresponding to the effective pixels in each image frame, the prediction depth corresponding to the effective pixels, the reference depth corresponding to the effective pixels, and the scaling scale, wherein the scaling scale in the objective function is an unknown parameter. The scaling factor is obtained by solving the objective function.

[0042] Specifically, if a pixel in an image frame is a static pixel, then the static prior weight of that pixel is 1; if a pixel in an image frame is a dynamic pixel, then the static prior weight of that pixel is 0.

[0043] In the above description, the weight coefficient for each valid pixel is not preset but dynamically determined. This dynamic weight determination allows the calculation of the scaling scale to focus only on the most reliable information while ignoring unreliable interference. For example, setting the static prior weight of dynamic pixels to 0 prevents them from participating in the scaling scale calculation, avoiding interference from moving vehicles, pedestrians, and other dynamic objects. This allows the scaling scale calculation to focus only on static background information. Pixels with high depth uncertainty (such as weak texture regions or occluded regions) have negative exponential mapping terms close to 0, thus having minimal impact on the scaling scale calculation. This allows the scaling scale calculation to refer only to high-confidence depth prediction values, automatically filtering out unreliable prediction results. This improves the accuracy of the scaling scale calculation.

[0044] Based on the static prior weights corresponding to the effective pixels and the depth uncertainty corresponding to the effective pixels, the weight coefficients corresponding to the effective pixels are constructed, including: Based on the depth uncertainty corresponding to the effective pixel, a negative exponential mapping term corresponding to the effective pixel is constructed, and the depth uncertainty corresponding to the effective pixel is negatively correlated with the negative exponential mapping term corresponding to the effective pixel. Multiply the static prior weight corresponding to the effective pixel point and the negative exponential mapping term corresponding to the effective pixel point to obtain the weight coefficient corresponding to the effective pixel point. If the effective pixel point is a static pixel point, the static prior weight corresponding to the effective pixel point is 1; if the effective pixel point is a dynamic pixel point, the static prior weight corresponding to the effective pixel point is 0.

[0045] For example, the negative exponent mapping term corresponding to the effective pixel is ,in, This is the index of the image frame within the time window, with values ​​ranging from 1 to the total number of frames N. These are the coordinates of the effective pixels in the projected image. For image frames The corresponding effective pixels of the projected image The corresponding depth uncertainty. The negative exponential mapping term corresponding to the effective pixels will affect the depth uncertainty output by the monocular 3D temporal model. Convert to confidence error decay weights When the depth of uncertainty is greater, the weight approaches zero, which can prevent abnormal areas with high uncertainty from participating in loss calculation.

[0046] Image Frame Effective pixels of the projected image The corresponding weighting coefficient is ,in, It is an image frame Effective pixels of the projected image The static prior weights, if valid pixels If it is a static pixel, then The value is 1; if the number of valid pixels is 1. For dynamic pixels, then The value is 0.

[0047] By determining the weight coefficients corresponding to valid pixels using the above method, the negative exponential mapping terms for regions with high depth uncertainty (such as weak texture regions and occluded regions) approach 0. Therefore, their impact on scaling calculation is minimal. Thus, when calculating the scaling scale, only high-confidence depth prediction values ​​are referenced, automatically filtering unreliable prediction results, improving the accuracy of scaling scale estimation, especially in complex scenes with weak textures and occlusion. Dynamic pixels are automatically filtered out, avoiding interference from moving vehicles, pedestrians, and other dynamic objects on scaling scale estimation, focusing only on the static background. Furthermore, the automatic filtering of dynamic pixels and pixels with high depth uncertainty eliminates their need to participate in subsequent complex calculations, reducing computational load and improving the real-time performance of data processing.

[0048] Specifically, based on the weight coefficients corresponding to the effective pixels in each image frame, the predicted depth corresponding to the effective pixels, the reference depth corresponding to the effective pixels, and the scaling scale, an objective function is constructed, including: For any valid pixel in each image frame, calculate the sum of squared Euclidean distances between the predicted depth corresponding to the valid pixel (multiplied by the scaling factor) and the reference depth corresponding to the valid pixel; and... Multiply the sum of squared Euclidean distances corresponding to the effective pixels by the weight coefficients corresponding to the effective pixels to obtain the product corresponding to the effective pixels; The objective function is constructed based on the sum of the products corresponding to each effective pixel in each image frame.

[0049] Specifically, the spatiotemporal joint weighted least squares optimization objective function is based on the entire time window. Determined according to the following formula (3): (3) The optimal scaling factor is obtained by iteratively calculating the partial derivatives of the objective function. This makes the image frame The corresponding effective pixels of the projected image Corresponding prediction depth After scaling, with effective pixels Corresponding reference depth The sum of squares of the Euclidean distances between them is minimized. This is the set of valid pixels in the projected image.

[0050] After determining S, the predicted depths corresponding to multiple effective pixels are multiplied by the scaling scale to compensate for the predicted depth output by the monocular 3D temporal model, thus obtaining a corrected depth with real physical distance; the scaling scale is multiplied by the camera translation vector to obtain the absolute physical translation vector, thereby achieving absolute scale compensation for the camera translation vector.

[0051] The objective function described above organically integrates multiple image frames in the temporal dimension and effective pixels in the spatial dimension. Since the objective function covers all image frames within the temporal window, the calculated scaling scale S is applicable to the entire window, avoiding inter-frame scale jumps that may occur in single-frame scale estimation and ensuring scale consistency across image frames. Furthermore, because the objective function integrates information from multiple image frames rather than relying solely on a single frame, it improves the accuracy of scale estimation, especially when single-frame image information is insufficient (such as in weak texture regions or occluded regions). In addition, the objective function is a strictly convex function, which guarantees finding a global optimum and avoids getting trapped in local optima, ensuring that the scaling scale S is globally optimal.

[0052] Figure 2 A structural diagram of the model evaluation apparatus provided in an embodiment of this application is shown. Figure 2 As shown, the model evaluation device 200 includes: The acquisition module 201 is used to acquire multiple image frames and laser point cloud data corresponding to each image frame; The projection module 202 is used to project the laser point cloud data corresponding to each image frame onto the two-dimensional image plane where the image frame is located, so as to obtain the projected image corresponding to the image frame. The projected image includes a plurality of effective pixels and a reference depth corresponding to each effective pixel. The prediction module 203 is used to input multiple image frames into the monocular three-dimensional temporal model to be tested, and obtain the prediction result of the monocular three-dimensional temporal model for depth prediction of each image frame. The prediction result includes the predicted depth of each pixel in the image frame, and the depth uncertainty of each predicted depth. The calculation module 204 is used to calculate the scaling scale based on the predicted depth corresponding to the multiple effective pixels in each image frame, the reference depth corresponding to the multiple effective pixels, and the depth uncertainty corresponding to the multiple effective pixels. The correction module 205 is used to correct the predicted depth corresponding to the multiple effective pixels according to the scaling scale, so as to obtain the corrected depth corresponding to each effective pixel. The determination module 206 is used to determine the evaluation result of the monocular three-dimensional temporal model based on the correction depth corresponding to the effective pixel and the reference depth corresponding to the effective pixel.

[0053] The model evaluation apparatus 200 provided in this application embodiment can implement all the processes implemented in the aforementioned model evaluation method embodiment and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0054] Figure 3 A schematic diagram of the hardware structure of the electronic device provided in an embodiment of this application is shown.

[0055] The electronic device may include a processor 601 and a memory 602 storing computer program instructions.

[0056] Specifically, the processor 601 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0057] Memory 602 may include mass storage for data or instructions. For example, and not limitingly, memory 602 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 602 may include removable or non-removable (or fixed) media. Where appropriate, memory 602 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 602 is non-volatile solid-state memory.

[0058] Memory may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, and electrical, optical, or other physical / tangible memory storage devices. Therefore, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the methods according to the first or second aspect of this disclosure.

[0059] The processor 601 reads and executes computer program instructions stored in the memory 602 to implement any of the model evaluation methods in the above embodiments.

[0060] In one example, the electronic device may also include a communication interface 603 and a bus 610. For example, Figure 3 As shown, the processor 601, memory 602, and communication interface 603 are connected through bus 610 and complete communication with each other.

[0061] The communication interface 603 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0062] Bus 610 includes hardware, software, or both. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 610 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, this application contemplates any suitable bus or interconnect.

[0063] Furthermore, in conjunction with the model evaluation methods in the above embodiments, this application embodiment can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the model evaluation methods in the above embodiments.

[0064] This application provides a computer program product in which the instructions are executed by the processor of an electronic device, causing the electronic device to perform any of the model evaluation methods described in the above embodiments.

[0065] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described as examples. However, the method process of this application is not limited to the specific steps described. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0066] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0067] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0068] The foregoing flowcharts and / or block diagrams of methods, apparatus (systems) according to embodiments of the present disclosure have described various aspects of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to create a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowcharts and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block of the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0069] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A model evaluation method, characterized in that, The method includes: Acquire multiple image frames and the laser point cloud data corresponding to each image frame; For each image frame, the laser point cloud data corresponding to the image frame is projected onto the two-dimensional image plane where the image frame is located to obtain the projected image corresponding to the image frame, wherein the projected image includes multiple effective pixels and a reference depth corresponding to each effective pixel; Multiple image frames are input into a monocular 3D temporal model to be tested, and the prediction results of the monocular 3D temporal model for depth prediction of each image frame are obtained. The prediction results include the predicted depth of each pixel in the image frame and the depth uncertainty of each predicted depth. The scaling scale is calculated based on the predicted depth corresponding to multiple effective pixels in each image frame, the reference depth corresponding to multiple effective pixels, and the depth uncertainty corresponding to multiple effective pixels. The predicted depths corresponding to multiple effective pixels are corrected according to the scaling scale to obtain the corrected depths corresponding to each effective pixel. The evaluation result of the monocular 3D temporal model is determined based on the correction depth corresponding to the effective pixel and the reference depth corresponding to the effective pixel.

2. The model evaluation method according to claim 1, characterized in that, The scaling scale is calculated based on the predicted depth corresponding to multiple effective pixels in each image frame, the reference depth corresponding to multiple effective pixels, and the depth uncertainty corresponding to multiple effective pixels, including: For each image frame, the static prior weight of each pixel in the image frame is obtained. The static prior weight is used to identify static pixels and dynamic pixels by using different values. For each effective pixel, a weight coefficient is constructed based on the static prior weight and the depth uncertainty corresponding to the effective pixel. The weight coefficient corresponding to the effective pixel is negatively correlated with the depth uncertainty corresponding to the effective pixel. An objective function is constructed based on the weight coefficients corresponding to the effective pixels in each image frame, the prediction depth corresponding to the effective pixels, the reference depth corresponding to the effective pixels, and the scaling scale, wherein the scaling scale in the objective function is an unknown parameter. The scaling factor is obtained by solving the objective function.

3. The model evaluation method according to claim 2, characterized in that, Based on the static prior weights corresponding to the effective pixels and the depth uncertainty corresponding to the effective pixels, the weight coefficients corresponding to the effective pixels are constructed, including: Based on the depth uncertainty corresponding to the effective pixel, a negative exponential mapping term corresponding to the effective pixel is constructed, and the depth uncertainty corresponding to the effective pixel is negatively correlated with the negative exponential mapping term corresponding to the effective pixel. Multiply the static prior weight corresponding to the effective pixel point and the negative exponential mapping term corresponding to the effective pixel point to obtain the weight coefficient corresponding to the effective pixel point. If the effective pixel point is a static pixel point, the static prior weight corresponding to the effective pixel point is 1; if the effective pixel point is a dynamic pixel point, the static prior weight corresponding to the effective pixel point is 0.

4. The model evaluation method according to claim 2, characterized in that, Based on the weight coefficients corresponding to the effective pixels in each image frame, the predicted depth corresponding to the effective pixels, the reference depth corresponding to the effective pixels, and the scaling scale, a target function is constructed, including: For any valid pixel in each image frame, calculate the sum of squared Euclidean distances between the predicted depth corresponding to the valid pixel (multiplied by the scaling factor) and the reference depth corresponding to the valid pixel; and... Multiply the sum of squared Euclidean distances corresponding to the effective pixels by the weight coefficients corresponding to the effective pixels to obtain the product corresponding to the effective pixels; The objective function is constructed based on the sum of the products corresponding to each effective pixel in each image frame.

5. The model evaluation method according to claim 1, characterized in that, Projecting the laser point cloud data corresponding to the image frame onto the two-dimensional image plane where the image frame is located, to obtain the projected image corresponding to the image frame, including: The laser point cloud data corresponding to the image frame is projected onto the two-dimensional image plane where the image frame is located. If at least one point in the laser point cloud data is projected onto the same pixel, the minimum depth corresponding to at least one of the points is taken as the reference depth corresponding to the pixel, and the pixel is taken as the effective pixel of the projected image.

6. The model evaluation method according to claim 1, characterized in that, The evaluation result of the monocular 3D temporal model is determined based on the correction depth corresponding to the effective pixels and the reference depth corresponding to the effective pixels, including: The depth error corresponding to the effective pixel is obtained by dividing the absolute value of the difference between the corrected depth corresponding to the effective pixel and the reference depth corresponding to the effective pixel by the reference depth corresponding to the effective pixel. The mean depth error corresponding to each effective pixel is used as the evaluation result of the monocular 3D temporal model.

7. A model evaluation device, characterized in that, The device includes: The acquisition module is used to acquire multiple image frames and laser point cloud data corresponding to each image frame; The projection module is used to project the laser point cloud data corresponding to each image frame onto the two-dimensional image plane where the image frame is located, so as to obtain the projected image corresponding to the image frame. The projected image includes a plurality of effective pixels and a reference depth corresponding to each effective pixel. The prediction module is used to input multiple image frames into a monocular 3D temporal model to be tested, and obtain the prediction result of the monocular 3D temporal model for depth prediction of each image frame. The prediction result includes the predicted depth of each pixel in the image frame and the depth uncertainty of each predicted depth. The calculation module is used to calculate the scaling scale based on the predicted depth corresponding to multiple effective pixels in each image frame, the reference depth corresponding to multiple effective pixels, and the depth uncertainty corresponding to multiple effective pixels. The correction module is used to correct the predicted depth corresponding to the multiple effective pixels according to the scaling scale, so as to obtain the corrected depth corresponding to each effective pixel. The determination module is used to determine the evaluation result of the monocular three-dimensional temporal model based on the correction depth corresponding to the effective pixel and the reference depth corresponding to the effective pixel.

8. An electronic device, characterized in that, include: Processor and memory storing computer program instructions; When the processor executes the computer program instructions, it implements the model evaluation method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions that, when executed by a processor, implement the model evaluation method as described in any one of claims 1-6.

10. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device performs the model evaluation method as described in any one of claims 1-6.

Images