A method, device, equipment and medium for optimizing three-dimensional shape in standardized flow space
By extracting the image data of the target vehicle, calculating the three-dimensional spatial points, using the trained network to predict the symbol distance function value and the target loss function value, and updating the shape encoding, the accuracy and efficiency of traditional three-dimensional shape reconstruction in complex scenes and occlusions is solved. Problem, more efficient three-dimensional shape reconstruction is achieved.
Patent Information
- Application Number
- CN202411205913.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-30
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-08-30
AI Technical Summary
Traditional three-dimensional shape reconstruction technology is difficult to deal with complex scenarios, partial occlusion, and low accuracy and low efficiency.
By obtaining the image data of the target vehicle, extracting feature points, calculating three-dimensional spatial points, using the trained network to predict the symbol distance function value, calculate the target loss function value, including surface loss, rendering depth loss and mask loss, and update the shape encoding until it is The loss function converges, achieving accurate reconstruction of the three-dimensional shape of the target vehicle.
It improves the accuracy and efficiency of three-dimensional reconstruction, enhances the robustness of occlusion and noise, reduces the dependence on hardware, and is suitable for complex scenarios such as autonomous driving.
Smart Images

Figure CN118710822B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of three-dimensional shape reconstruction, and in particular to a method, device, equipment and medium for optimizing three-dimensional shape in a standardized flow space. Background Art
[0002] With the development of computer vision and graphics technology, 3D shape reconstruction has become a key technology in many application fields, including virtual reality, augmented reality, robotics, autonomous driving, and medical imaging. By reconstructing the 3D shape of an object from a 2D image or 3D scan data, detailed geometric information of the object can be obtained, which can be analyzed, modeled, and applied. However, 3D shape reconstruction still faces many challenges, such as complex scenes, partial occlusion, noise interference, data sparsity, and computational resource limitations. In past studies, there are two main methods for 3D shape reconstruction: methods based on 3D scan data and image-based methods. Methods based on 3D scan data usually rely on high-precision laser scanners or structured light technology to directly obtain 3D point cloud data of the object. However, these methods have high equipment costs and cumbersome data acquisition processes. Image-based methods extract feature points or edge information from multi-view images and combine stereo vision or structured light technologies for 3D reconstruction. Although this method is relatively low-cost, it may have difficulties in dealing with complex scenes and occlusion problems.
[0003] In the related art, deep learning technology has made some progress in the field of 3D shape reconstruction. For example, methods such as convolutional neural networks (CNN) and generative adversarial networks (GAN) are used to learn the 3D shape of an object from an image or point cloud. However, these methods usually require a large amount of labeled data for training and may not perform well when dealing with unseen scenes or objects. In the related art, 3D shape reconstruction methods based on deep learning still face challenges in dealing with complex scenes, separation of foreground and background, and high precision requirements.
[0004] In view of this, a 3D shape optimization method is needed to improve the accuracy, efficiency and robustness of 3D shape reconstruction. Summary of the invention
[0005] In view of this, the present invention provides a standardized flow space three-dimensional shape optimization method, device, equipment and medium to solve the problem that traditional three-dimensional shape reconstruction technology is difficult to handle complex scenes, partial occlusions, and has low accuracy and efficiency.
[0006] In the first aspect, the present invention provides a standardized flow space three-dimensional shape optimization method, the method comprising: acquiring image data of a target vehicle; based on the image data, extracting feature points of the target vehicle, and calculating the three-dimensional space points of the feature points; processing the shape codes corresponding to the three-dimensional space points through a trained network, and predicting the signed distance function values of the three-dimensional space points; based on the signed distance function values, calculating the target loss function values corresponding to the three-dimensional space points; the target loss function comprises surface loss, rendering depth loss and mask loss; the surface loss is used to indicate the size of the signed distance function values of each three-dimensional space point; the rendering depth loss is the error of the depth distance of the three-dimensional space point; the mask loss is used to indicate the feature points of the three-dimensional shape that is obscured; the shape code is updated based on the target loss function value until the loss function converges, thereby realizing the reconstruction of the three-dimensional shape of the target vehicle.
[0007] In the second aspect, the present invention provides a standardized flow space three-dimensional shape optimization device, which includes: a data acquisition module, used to acquire image data of a target vehicle; a space point calculation module, used to extract feature points of the target vehicle based on the image data, and calculate the three-dimensional space points of the feature points; a function value prediction module, used to process the shape coding corresponding to the three-dimensional space points through a trained network, and predict the signed distance function value of the three-dimensional space points; a loss calculation module, used to calculate the target loss function value corresponding to the three-dimensional space points based on the signed distance function value; the target loss function includes surface loss, rendering depth loss and mask loss; the surface loss is used to indicate the size of the signed distance function value of each three-dimensional space point; the rendering depth loss is the error of the depth distance of the three-dimensional space point; the mask loss is used to indicate the feature points of the occluded three-dimensional shape; the coding update module is used to update the shape coding based on the target loss function value until the loss function converges, thereby realizing the reconstruction of the three-dimensional shape of the target vehicle.
[0008] In a third aspect, the present invention provides a computer device comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the standardized flow space three-dimensional shape optimization method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.
[0009] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the standardized flow space three-dimensional shape optimization method of the first aspect or any corresponding embodiment thereof.
[0010] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions for causing a computer to execute the standardized flow space three-dimensional shape optimization method of the first aspect or any corresponding embodiment thereof.
[0011] The technical solution provided by one or more embodiments of the present invention obtains image data of the target vehicle, extracts feature points based on these data, and calculates the three-dimensional space points corresponding to these feature points. Then, the trained network is used to process the corresponding shape coding, and the signed distance function value of each three-dimensional point is predicted. Through these values, the target loss function including surface loss, rendering depth loss and mask loss is calculated, wherein the surface loss measures the magnitude of the signed distance function value, the rendering depth loss reflects the error of the depth distance, and the mask loss indicates the occluded three-dimensional shape feature points. Based on the value of the target loss function, the shape coding is continuously updated until the loss function converges to achieve accurate three-dimensional shape reconstruction, which is suitable for target vehicles in complex scenes such as autonomous driving. This process not only improves the accuracy and efficiency of three-dimensional reconstruction, but also enhances the robustness to occlusion and noise, while reducing dependence on hardware. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0013] Figure 1 is a schematic flow chart of a method for optimizing a three-dimensional shape in a standardized flow space according to an embodiment of the present invention;
[0014] Figure 2 is a structural block diagram of a normalized flow space three-dimensional shape optimization device according to an embodiment of the present invention;
[0015] Figure 3 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0016] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.
[0017] The embodiment of the present invention provides a normalized flow space three-dimensional shape optimization method, which achieves the effect of improving the accuracy, efficiency and robustness of three-dimensional shape reconstruction by introducing foreground mask constraints and multivariate loss functions.
[0018] According to an embodiment of the present invention, an embodiment of a standardized flow space three-dimensional shape optimization method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0019] In this embodiment, a standardized flow space three-dimensional shape optimization method is provided. Figure 1 is a flow chart of a normalized flow space three-dimensional shape optimization method according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:
[0020] Step S101, acquiring image data of a target vehicle.
[0021] In the field of autonomous driving, in order to obtain the driving status of the target vehicle in front, the self-vehicle usually needs to obtain the two-dimensional image data of the target vehicle, and then perform three-dimensional reconstruction of the target vehicle to generate a three-dimensional model of the target vehicle in the surrounding environment. Specifically, on the target vehicle, the image data of the target vehicle in front can be collected by the image acquisition device set at the front of the self-vehicle.
[0022] Acquiring image data of the target vehicle is the basis for 3D shape reconstruction. Specifically, a camera or other image acquisition device can be used to record high-resolution images of the vehicle. The obtained image data will be used for subsequent computer vision algorithm processing to reconstruct the 3D shape of the vehicle from the 2D image.
[0023] Step S102: extracting feature points of the target vehicle based on the image data, and calculating the three-dimensional space points of the feature points.
[0024] Before reconstructing the target vehicle in three dimensions, it is necessary to first obtain the feature points on the two-dimensional image of the target vehicle. Feature points refer to points with unique properties in the image, usually easily distinguishable areas in the image. Image data is analyzed to identify significant and distinguishable points on the vehicle, such as corners, edges or other characteristic points. These feature points are extracted from the image using computer vision algorithms, such as the ORB-SLAM2 algorithm. The calculated three-dimensional point coordinates are converted from the camera coordinate system to a unified three-dimensional space coordinate system for subsequent three-dimensional modeling and analysis.
[0025] Step S103, processing the shape code corresponding to the three-dimensional space point through the trained network to predict the signed distance function value of the three-dimensional space point.
[0026] In the above steps, the three-dimensional point coordinates generated by directly using a computer vision algorithm such as the ORB-SLAM2 algorithm based on the two-dimensional image can actually be considered as the basic three-dimensional model of the target vehicle, but its accuracy is low and it is easily disturbed by the environment. Therefore, in the embodiment of the present application, it is first necessary to use a trained network to process the potential shape encoding of the standardized flow space corresponding to each three-dimensional space point. Standardized flow space is a mathematical and computational concept used to represent and process three-dimensional shapes in three-dimensional shape reconstruction and computer vision tasks. Shape encoding is a numerical representation that encodes the characteristics of a three-dimensional shape. Predict the signed distance function (SDF) value of each three-dimensional space point. The signed distance function value is a function defined in space that assigns a value to each point, indicating the distance from the point to the nearest surface, and has a sign (positive or negative) according to the position of the point (inside or outside the surface).
[0027] In the embodiment of the present application, the trained network may be a neural network that has been trained with samples and annotations, such as a convolutional neural network CNN. The neural network may use the shape encoding of the three-dimensional points of the vehicle as samples and the signed distance function value of each three-dimensional point as an annotation for training, so that the trained neural network can process the shape encoding of the three-dimensional space point, thereby directly obtaining the signed distance function value of the three-dimensional space point.
[0028] Step S104, based on the signed distance function value, calculate the target loss function value corresponding to the three-dimensional space point; the target loss function includes surface loss, rendering depth loss and mask loss; the surface loss is used to indicate the size of the signed distance function value of each three-dimensional space point; the rendering depth loss is the error of the depth distance of the three-dimensional space point; the mask loss is used to indicate the feature points of the obscured three-dimensional shape.
[0029] In this step, the signed distance function value is used to characterize the distance and direction of each point in space to the nearest surface. Based on the signed distance function value, the target loss function is designed to quantify the consistency between the reconstructed shape and the observed data, and improve the 3D reconstruction result by optimizing the loss. The following is an explanation of the components of the target loss function:
[0030] The surface loss focuses on the signed distance function value of each 3D space point to ensure that the predicted signed distance function value can accurately reflect the actual distance from the point to the surface. If the signed distance function value is zero, it means that the point is exactly on the surface; if the signed distance function value is positive or negative, it means that the point is outside the surface, and the sign indicates whether the point is inside or outside the surface.
[0031] The rendering depth loss characterizes the difference between the predicted depth and the actual observed depth distance. By comparing the predicted depth of each spatial point with the true depth estimated from the image data, this loss helps the optimization algorithm correct the prediction error and improve the accuracy of the depth estimation.
[0032] The mask loss is used to deal with occlusion problems in the scene, that is, when some feature points of the 3D shape are occluded by other objects and are therefore not visible in the observed data. The loss analyzes whether the feature points are occluded and adjusts the shape encoding accordingly so that the reconstruction result can correctly reflect the shape of the occluded area. The mask loss ensures that even when some information is missing, a structurally reasonable and visually realistic 3D reconstruction can be obtained.
[0033] Step S105, updating the shape encoding based on the target loss function value until the loss function converges, thereby achieving reconstruction of the three-dimensional shape of the target vehicle.
[0034] In this step, the shape encoding is adjusted by an optimization algorithm to minimize the value of the objective loss function. This process is iterative, and the shape encoding is updated at each iteration until the value of the loss function reaches a lower stable state, i.e. convergence.
[0035] For example, during shape reconstruction, we fix the network weights for shape optimization For each image, the vehicle feature points are extracted using the traditional ORB-SLAM2 method, and its 3D point in three-dimensional space is calculated. For each instance, assume Represents the latent shape encoding of the normalized flow space, which is first initialized from a zero-valued vector , then the trained network Able to predict each 3D point The signed distance function (SDF) value of ,Right now ,in Represents the transformation from the camera coordinate system to the object coordinate system.
[0036] Under the supervision of a specific loss function, we try to find the optimal shape encoding and coordinate transformation matrix, thereby recovering the 3D shape that best matches the current observation.
[0037] The standardized flow space 3D shape optimization method provided in this embodiment introduces surface loss, mask loss and rendering depth loss in the loss function, which can effectively improve the accuracy, efficiency and robustness of 3D shape reconstruction, and can effectively handle 3D shape reconstruction tasks in scenes with occlusion and other complex scenes. In addition to being applicable to 3D modeling of autonomous driving vehicles, the method shown in the embodiment of this application is also applicable to other application fields that require high-precision 3D reconstruction.
[0038] In some optional implementations, extracting feature points of the target vehicle based on the image data in step S102 includes the following steps:
[0039] Step a11, obtaining image data of the target vehicle.
[0040] In this step, image data of the target vehicle is collected, usually by taking photos or videos of the vehicle through a camera. These image data form the basis for subsequent processing.
[0041] Step a12, responding to the labeling instruction, completing the labeling of the outline of the target vehicle and obtaining the labeled data.
[0042] In this step, the vehicle’s outline is annotated, which involves drawing a bounding box in the image or using image segmentation techniques to identify and mark the exact boundaries of the vehicle.
[0043] Step a13, export the labeled data as 2D instance segmentation labels.
[0044] In this step, the obtained labeled data is converted into 2D instance segmentation labels. The 2D instance segmentation labels provide classification information of vehicle instances for each pixel in the image, distinguishing between vehicles and other backgrounds.
[0045] Step a14, extracting feature points of the target vehicle based on the image data of the target vehicle and the 2D instance segmentation label.
[0046] In this step, the vehicle image data and the generated 2D instance segmentation labels are combined to use computer vision algorithms to identify and extract the feature points of the vehicle.
[0047] The method provided in this embodiment can accurately extract the feature points of the target vehicle from the image, providing necessary information for subsequent 3D shape reconstruction. This method combines image processing, annotation, instance segmentation and feature extraction technology, which can improve the accuracy and efficiency of feature point extraction, thereby providing a basis for 3D reconstruction.
[0048] In some optional implementations, calculating the target loss function value corresponding to the three-dimensional space point based on the signed distance function value in step S103 includes the following steps:
[0049] Process the signed distance function through the multivariate loss function to obtain the target loss function value;
[0050] The multivariate loss function is:
[0051]
[0052] in, , , , represents the weight, is the surface loss, is the mask loss, To render the depth loss, Encode the shape, Indicates the frame number of the image.
[0053] In this step, the signed distance function value is first input into a multivariate loss function. The target loss function is a weighted combination that comprehensively considers different loss terms to evaluate the consistency between the predicted 3D shape and the actual observed data. Represents the weights of different loss terms. The weights determine the importance of each loss term in the total loss calculation. The target loss function is the weighted sum of each loss, which provides a comprehensive metric to guide the optimization of shape encoding. By minimizing this loss function, the optimal shape encoding can be found, thereby achieving accurate reconstruction of the target vehicle's 3D shape.
[0054] The method provided in this embodiment can effectively reconstruct the three-dimensional shape of the target vehicle from the two-dimensional image data, and can maintain high accuracy and robustness even in the presence of occlusion and complex scenes.
[0055] In some optional implementations, processing the signed distance function through a multivariate loss function to obtain a target loss function value includes:
[0056] Step b11, through the transformation matrix , the three-dimensional space point Convert to the coordinate system of the target vehicle to obtain the point in the coordinate system .
[0057] In this step, each 3D space point is transformed from its original coordinate system to the target vehicle's coordinate system. This is usually done by using a transformation matrix, which may contain transformations such as translation, rotation, and scaling.
[0058] Step b12, the surface loss is obtained based on the following formula: :
[0059]
[0060] in, is the trained network, Encode the shape.
[0061] In this step, the formula is to calculate the difference between the predicted signed distance function value and the actual surface, and the loss is calculated by squaring the difference. If the predicted signed distance function value is accurate, the surface loss will be small; if the prediction is inaccurate, the surface loss will be large, thus guiding the adjustment of the shape encoding during the optimization process.
[0062] In the method provided in this embodiment, surface loss is an important component of the target loss function, which helps to guide the optimization of shape coding to achieve more accurate three-dimensional shape reconstruction.
[0063] In some optional implementations, processing the signed distance function through a multivariate loss function to obtain a target loss function value further includes:
[0064] Step c11, based on the following formula, calculate the The probability of sampling points being empty ;
[0065]
[0066] in, is the preset distance threshold, is the signed distance function value.
[0067] In this step, the probability that each sample point is "empty" (i.e., does not hit any surface) during ray tracing is calculated. Reflects the The probability that a sampling point does not touch the surface.
[0068] Step c12, obtain the hit probability based on the following formula: ;
[0069]
[0070] in, Indicates the number of sampling points; Represents light Hit probability The sampling point at which this function terminates.
[0071] In this step, assume that the light go through sampling points. The ray may hit One of the sampling points may also escape. We use the variable represents the sampling point where the ray terminates with probability, where Indicates that the ray did not hit the surface.
[0072] Step c13, obtain the cost function based on the following formula :
[0073]
[0074] in, Represents light The probability of intersecting a pixel in the object mask, .
[0075] In this step, Represents light intersects with a pixel in the object mask, so should hit the surface; indicates that the ray did not hit the target object. Finally, for the event ( ) Design a cost function , thereby penalizing predictions that are inconsistent with observations.
[0076] Step c14, obtain the mask loss based on the following formula: :
[0077]
[0078] in, Represents the union of the set of pixels that are inside the masked region and the set of pixels that are outside the masked region but inside the 2D bounding box.
[0079] In this step, the mask loss is calculated based on the cost function and the set of pixels inside and outside the mask region.
[0080] The method provided in this embodiment considers ray tracing, sampling point probability, and mask information in the 3D shape reconstruction process to optimize the reconstruction results and improve the ability to handle occlusion and complex scenes. In this way, the algorithm can more accurately predict and adjust the shape encoding to minimize the reconstruction error, especially in the presence of occlusion and complex background.
[0081] In some optional implementations, processing the signed distance function through a multivariate loss function to obtain a target loss function value further includes:
[0082] The image data includes a foreground object and a background. Based on the image data, a mask region is generated to calculate a mask loss in a target loss function value, wherein white or non-zero pixels represent foreground objects and black or zero pixels represent backgrounds.
[0083] Specifically, based on the image data, a mask region is generated, which is a special image region used to indicate which pixels belong to the foreground object and which belong to the background. The mask region is usually a binary image, in which white or non-zero pixels represent the foreground object, that is, the area where the 3D shape we want to reconstruct is located. Black or zero pixels represent the background, that is, the area that does not participate in the 3D shape reconstruction. During the optimization process, the mask loss serves as a feedback signal to guide the algorithm to adjust the shape encoding to minimize the loss and improve the accuracy of the reconstructed shape.
[0084] The method provided in this embodiment can more effectively distinguish the foreground and background during the three-dimensional shape reconstruction process, improve the ability to handle occlusion and complex scenes, and thus achieve higher quality three-dimensional shape reconstruction.
[0085] In some optional implementations, processing the signed distance function through a multivariate loss function to obtain a target loss function value further includes:
[0086] The rendering depth loss is obtained based on the following formula:
[0087]
[0088] in, including points that are on the surface at depth, as well as points that are not on the surface but still within the 2D bounding box, Indicates the depth of the point. Indicates the sampling depth.
[0089] In this step, the depth of the point is estimated by stereo triangulation or bundle adjustment. After that, the rendered depth loss is calculated at these points. The rendered depth loss implicitly aligns the observed depth points with the unoccluded parts of the surface. When ray tracing along each ray, each sampled depth is , the sampling interval step is The method provided in this embodiment considers the depth information of multiple sampling points and the corresponding hit probability, calculates the rendering depth loss, and quantifies the difference between the predicted depth and the actual observed depth, thereby providing an optimization direction for 3D shape reconstruction and optimizing the robustness in 3D shape reconstruction.
[0090] In this embodiment, a standardized flow space three-dimensional shape optimization device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.
[0091] This embodiment provides a standardized flow space three-dimensional shape optimization device, such as Figure 2 As shown, including:
[0092] The data acquisition module 201 is used to acquire image data of the target vehicle.
[0093] The spatial point calculation module 202 is used to extract the characteristic points of the target vehicle based on the image data and calculate the three-dimensional spatial points of the characteristic points.
[0094] The function value prediction module 203 is used to process the shape coding corresponding to the three-dimensional space point through the trained network and predict the signed distance function value of the three-dimensional space point.
[0095] The loss calculation module 204 is used to calculate the target loss function value corresponding to the three-dimensional space point based on the signed distance function value; the target loss function includes surface loss, rendering depth loss and mask loss; the surface loss is used to indicate the size of the signed distance function value of each three-dimensional space point; the rendering depth loss is the error of the depth distance of the three-dimensional space point; the mask loss is used to indicate the feature points of the obscured three-dimensional shape.
[0096] The coding update module 205 is used to update the shape coding based on the target loss function value until the loss function converges, thereby realizing the reconstruction of the three-dimensional shape of the target vehicle.
[0097] In some optional implementations, the above-mentioned spatial point calculation module 202 includes:
[0098] The first space point calculation unit is used to obtain image data of the target vehicle.
[0099] The second space point calculation unit is used to respond to the marking instruction, complete the marking of the outline of the target vehicle, and obtain the marked data.
[0100] The third spatial point calculation unit is used to export the labeled data into 2D instance segmentation labels.
[0101] The fourth spatial point calculation unit is used to extract feature points of the target vehicle based on the image data of the target vehicle and the 2D instance segmentation label.
[0102] In some optional implementations, the loss calculation module 204 includes:
[0103] A multivariate loss calculation unit, used for processing the signed distance function through the multivariate loss function to obtain a target loss function value;
[0104] The multivariate loss function is:
[0105]
[0106] in, , , , represents the weight, is the surface loss, is the mask loss, To render the depth loss, Encode the shape, Indicates the frame number of the image.
[0107] In some optional implementations, the multivariate loss calculation unit includes:
[0108] The first surface loss calculation subunit is used to calculate the surface loss through the transformation matrix , the three-dimensional space point Convert to the coordinate system of the target vehicle to obtain the point in the coordinate system .
[0109] The second surface loss calculation subunit is used to obtain the surface loss based on the following formula :
[0110]
[0111] in, is the trained network, Encode the shape.
[0112] In some optional implementations, the multivariate loss calculation unit includes:
[0113] The first mask loss calculation subunit is used to calculate the first mask loss in the ray tracing process based on the following formula: The probability of sampling points being empty ;
[0114]
[0115] in, is the preset distance threshold, is the signed distance function value.
[0116] The second mask loss calculation subunit is used to obtain the hit probability based on the following formula ;
[0117]
[0118] in, Indicates the number of sampling points; Represents light Hit probability The sampling point at which this function terminates.
[0119] The third mask loss calculation subunit is used to obtain the cost function based on the following formula :
[0120]
[0121] in, Represents light The probability of intersecting a pixel in the object mask, .
[0122] The fourth mask loss calculation subunit is used to obtain the mask loss based on the following formula: :
[0123]
[0124] in, Represents the union of the set of pixels that are inside the masked region and the set of pixels that are outside the masked region but inside the 2D bounding box.
[0125] In some optional implementations, before calculating the target loss function value corresponding to the three-dimensional space point based on the signed distance function value in the loss calculation module 204, the following steps are further included:
[0126] A foreground-background distinguishing unit is used for image data including a foreground object and a background. Based on the image data, a mask area is generated to calculate the mask loss in the target loss function value, wherein white or non-zero pixels represent foreground objects and black or zero pixels represent backgrounds.
[0127] In some optional implementations, the multivariate loss calculation unit includes:
[0128] The rendering loss calculation subunit is used to obtain the rendering depth loss based on the following formula:
[0129]
[0130] in, including points that are on the surface at depth, as well as points that are not on the surface but still within the 2D bounding box, Indicates the depth of the point. Indicates the sampling depth.
[0131] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.
[0132] The standardized flow space three-dimensional shape optimization device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.
[0133] The embodiment of the present invention also provides a computer device having the above Figure 2 The normalized flow space 3D shape optimization setup is shown.
[0134] See also Figure 3 , Figure 3 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 3 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 3 A processor 10 is taken as an example.
[0135] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.
[0136] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.
[0137] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0138] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.
[0139] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 3 The example of connecting through bus is taken in the following.
[0140] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device may be a touch screen.
[0141] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.
[0142] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.
[0143] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.
Claims
1. A normalized flow space three-dimensional shape optimization method, characterized in that: The method comprises: Acquire image data of the target vehicle; Based on the image data, extract the feature points of the target vehicle and calculate the three-dimensional space points of the feature points; Processing the shape code corresponding to the three-dimensional space point through the trained network to predict the signed distance function value of the three-dimensional space point; Based on the signed distance function value, the target loss function value corresponding to the three-dimensional space point is calculated; the target loss function includes surface loss, rendering depth loss and mask loss; the surface loss is used to indicate the size of the signed distance function value of each three-dimensional space point; the rendering depth loss is the error of the depth distance of the three-dimensional space point; the mask loss is used to indicate the feature point of the three-dimensional shape that is blocked; The signed distance function value is processed by a multivariate loss function to obtain the target loss function value; the multivariate loss function is: in, , , , represents the weight, is the surface loss, is the mask loss, To render the depth loss, Encode the shape, Indicates the frame number of the image; Based on the following formula, the first The probability of sampling points being empty ; in, is the preset distance threshold, is the signed distance function value; The hit probability is obtained based on the following formula : in, Indicates the number of sampling points; Represents light Hit probability The sampling point at which the signal is terminated; The cost function is obtained based on the following formula : in, Represents light The probability of intersecting a pixel in the object mask, ; The mask loss is obtained based on the following formula : in, represents the union of the set of pixels within the mask region and the set of pixels outside the mask region but within the 2D bounding box; The shape encoding is updated based on the target loss function value until the loss function converges, thereby achieving reconstruction of the three-dimensional shape of the target vehicle.
2. The method according to claim 1, characterized in that The extracting the feature points of the target vehicle based on the image data comprises: Acquire image data of the target vehicle; Responding to the labeling instruction, completing the labeling of the target vehicle's outline and obtaining the labeled data; Exporting the labeled data as 2D instance segmentation labels; Feature points of the target vehicle are extracted based on the image data of the target vehicle and the 2D instance segmentation label.
3. The method according to claim 1, characterized in that The step of processing the signed distance function through a multivariate loss function to obtain the target loss function value includes: By transforming the matrix , the three-dimensional space point Convert to the coordinate system of the target vehicle to obtain the point in the coordinate system ; The surface loss is obtained based on the following formula : in, is the trained network, The shape is encoded.
4. The method according to claim 1, characterized in that: Before calculating the target loss function value corresponding to the three-dimensional space point based on the signed distance function value, the method further includes: The image data includes a foreground object and a background. Based on the image data, the mask area is generated to calculate the mask loss in the target loss function value, wherein white or non-zero pixels represent the foreground object and black or zero pixels represent the background.
5. The method according to claim 1, characterized in that The step of processing the signed distance function through a multivariate loss function to obtain the target loss function value includes: The rendering depth loss is obtained based on the following formula: in, including points that are on the surface at depth, as well as points that are not on the surface but still within the 2D bounding box, Indicates the depth of the point. Indicates the sampling depth.
6. A standardized flow space three-dimensional shape optimization device, characterized in that: The device comprises: A data acquisition module, used to acquire image data of a target vehicle; A spatial point calculation module, used to extract the characteristic points of the target vehicle based on the image data, and calculate the three-dimensional spatial points of the characteristic points; A function value prediction module, used to process the shape code corresponding to the three-dimensional space point through a trained network, and predict the signed distance function value of the three-dimensional space point; A loss calculation module, used to calculate the target loss function value corresponding to the three-dimensional space point based on the signed distance function value; the target loss function includes surface loss, rendering depth loss and mask loss; the surface loss is used to indicate the size of the signed distance function value of each three-dimensional space point; the rendering depth loss is the error of the depth distance of the three-dimensional space point; the mask loss is used to indicate the feature point of the three-dimensional shape that is blocked; The signed distance function value is processed by a multivariate loss function to obtain the target loss function value; the multivariate loss function is: in, , , , represents the weight, is the surface loss, is the mask loss, To render the depth loss, Encode the shape, Indicates the frame number of the image; Based on the following formula, the first The probability of sampling points being empty ; in, is the preset distance threshold, is the signed distance function value; The hit probability is obtained based on the following formula : in, Indicates the number of sampling points; Represents light Hit probability The sampling point at which the signal is terminated; The cost function is obtained based on the following formula : in, Represents light The probability of intersecting a pixel in the object mask, ; The mask loss is obtained based on the following formula : in, represents the union of the set of pixels within the mask region and the set of pixels outside the mask region but within the 2D bounding box; The coding update module is used to update the shape coding based on the target loss function value until the loss function converges, thereby realizing reconstruction of the three-dimensional shape of the target vehicle.
7. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method according to any one of claims 1 to 5 by executing the computer instructions.
8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 5.
