Target object reconstruction method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of reconstruction, and discloses a target object reconstruction method and device, electronic equipment and a storage medium, and the method comprises the steps: controlling a camera to carry out the surrounding shooting of a target object and a calibration plate beside the target object in the same scene, and obtaining a surrounding video containing the target object and the calibration plate; decomposing the surround video into multiple frames of images, and arranging the multiple frames of images according to the sequence of shooting time to obtain an image sequence; performing inversion operation on the calibration plate pose matrix corresponding to each frame of image to obtain a camera pose matrix corresponding to each frame of image, and processing the camera pose matrix corresponding to each frame of image and the image sequence through a modeling tool to obtain a sparse point cloud containing the target object and the calibration plate; and processing the final point cloud data and the surround video through a Gaussian splash model to obtain a reconstruction model conforming to the physical size of the target object. According to the method, the reconstruction model conforming to the physical size of the target object can be obtained.

Description

Technical Field

[0001] The present application relates to the field of reconstruction technology, and in particular to a target object reconstruction method, device, electronic device and storage medium. Background Art

[0002] The reconstructed model of the target object fundamentally breaks through the expression bottleneck of two-dimensional data in the spatial dimension by converting the physical entity into a computable three-dimensional digital asset, providing structured, multi-level three-dimensional information support for downstream tasks.

[0003] However, although the reconstructed model of the target object can generate a three-dimensional geometric structure, there is a deviation between the displayed size of the reconstructed model of the target object and the physical size of the target object. This scale inconsistency will significantly affect the practical application of downstream tasks. Therefore, the reconstructed model of the target object needs to be manually adjusted. The manual adjustment method consumes a lot of human resources and time resources and is easily affected by human intervention. Therefore, how to obtain a reconstructed model that conforms to the physical size of the target object is an urgent problem that needs to be solved. Summary of the Invention

[0004] The embodiments of the present application provide a target object reconstruction method, device, electronic device, and storage medium to solve the above-mentioned technical problem of how to obtain a reconstructed model that meets the physical dimensions of the target object.

[0005] In a first aspect, an embodiment of the present application provides a target object reconstruction method, which is applied to an electronic device. The target object reconstruction method includes: Control the camera to capture a surround video of the target object and the calibration plate next to the target object in the same scene, and obtain a surround video containing the target object and the calibration plate; Decompose the surround video into multiple frames of images, and arrange the multiple frames of images in the order of shooting time to obtain an image sequence; Obtain the calibration plate pose matrix corresponding to each frame of multiple images, perform inverse operation on the calibration plate pose matrix corresponding to each frame of image, and obtain the camera pose matrix corresponding to each frame of image. Use modeling tools to process the camera pose matrix and image sequence corresponding to each frame of image to obtain a sparse point cloud containing the target object and the calibration plate. The calibration plate pose matrix is ​​the coordinate transformation matrix from the calibration plate's coordinate system to the camera's coordinate system, and the camera pose matrix is ​​the coordinate transformation matrix from the camera's coordinate system to the calibration plate's coordinate system. Align the coordinate system of the sparse point cloud with the coordinate system of the calibration plate to obtain the aligned sparse point cloud, obtain the current scale of the calibration plate from the aligned sparse point cloud, obtain the true scale of the calibration plate from the preset text, and divide the true scale by the current scale to obtain the scale coefficient of the calibration plate; The original point cloud data of the calibration plate is deleted from the aligned sparse point cloud to obtain the original point cloud data of the target object. The coordinate value of each point in the original point cloud data of the target object is multiplied by the scale coefficient to obtain the final point cloud data of the target object. The final point cloud data and the surround video are input into the Gaussian splash model. The final point cloud data and the surround video are processed by the Gaussian splash model to obtain a reconstructed model that conforms to the physical size of the target object.

[0006] In a possible implementation of the first aspect, controlling the camera to perform surround shooting of a target object and a calibration plate next to the target object in the same scene to obtain a surround video including the target object and the calibration plate includes: Obtain a path that simultaneously surrounds the target object and the calibration plate next to the target object, and select the path that simultaneously surrounds the target object and the calibration plate next to the target object as the surrounding path; A target object and a calibration plate next to the target object in the same scene are selected as the shooting objects, and the camera is controlled to surround the shooting object along the surround path to obtain a surround video containing the target object and the calibration plate.

[0007] In a possible implementation of the first aspect, aligning the coordinate system of the sparse point cloud with the coordinate system of the calibration plate to obtain the aligned sparse point cloud, obtaining the current scale of the calibration plate from the aligned sparse point cloud, obtaining the true scale of the calibration plate from preset text, and dividing the true scale by the current scale to obtain a scale coefficient of the calibration plate includes: Use the alignment module of the modeling tool to align the coordinate system of the sparse point cloud with the coordinate system of the calibration plate to obtain the aligned sparse point cloud; The current scale of the calibration plate is obtained from the aligned sparse point cloud, the true scale of the calibration plate is obtained from the preset text, and the true scale is divided by the current scale to obtain the scale coefficient of the calibration plate.

[0008] In a possible implementation of the first aspect, the method includes deleting the original point cloud data of the calibration plate from the aligned sparse point cloud to obtain the original point cloud data of the target object, multiplying the coordinate value of each point in the original point cloud data of the target object by a scale factor to obtain the final point cloud data of the target object, inputting the final point cloud data and the surround video into a Gaussian splash model, and processing the final point cloud data and the surround video using the Gaussian splash model to obtain a reconstructed model that conforms to the physical size of the target object, including: The original point cloud data of the calibration plate is deleted from the aligned sparse point cloud to obtain the original point cloud data of the target object. The coordinate value of each point in the original point cloud data of the target object is multiplied by the scale coefficient to obtain each adjusted point. Each adjusted point is combined into the final point cloud data of the target object. Access the preset file, load the Gaussian splash model from the preset file, input the final point cloud data and surround video into the Gaussian splash model, and process the final point cloud data and surround video through the Gaussian splash model to obtain a reconstructed model that meets the physical size of the target object.

[0009] In a possible implementation of the first aspect, the original point cloud data of the calibration plate is deleted from the aligned sparse point cloud to obtain the original point cloud data of the target object, the coordinate value of each point in the original point cloud data of the target object is multiplied by a scale factor to obtain the final point cloud data of the target object, the final point cloud data and the surround video are input into a Gaussian splash model, and the final point cloud data and the surround video are processed by the Gaussian splash model to obtain a reconstructed model that conforms to the physical size of the target object. The target object reconstruction method includes: A display page is created to display the reconstructed model that conforms to the physical size of the target object.

[0010] In a possible implementation of the first aspect, the original point cloud data of the calibration plate is deleted from the aligned sparse point cloud to obtain the original point cloud data of the target object, the coordinate value of each point in the original point cloud data of the target object is multiplied by a scale factor to obtain the final point cloud data of the target object, the final point cloud data and the surround video are input into a Gaussian splash model, and the final point cloud data and the surround video are processed by the Gaussian splash model to obtain a reconstructed model that conforms to the physical size of the target object. The target object reconstruction method includes: Connect to the preset server and upload the reconstructed model that meets the physical size of the target object to the server.

[0011] In a possible implementation manner of the first aspect, the modeling tool includes one of a COLMAP tool and a Blender tool, or a combination thereof.

[0012] In a second aspect, an embodiment of the present application provides a target object reconstruction device, which is applied to an electronic device, including: The shooting module is used to control the camera to surround the target object and the calibration plate next to the target object in the same scene to obtain a surround video containing the target object and the calibration plate; An arrangement module is used to decompose the surround video into multiple frames of images and arrange the multiple frames of images in the order of shooting time to obtain an image sequence; The first acquisition module is used to obtain the calibration plate pose matrix corresponding to each frame of the multiple images, perform an inverse operation on the calibration plate pose matrix corresponding to each frame of the image, and obtain the camera pose matrix corresponding to each frame of the image. The camera pose matrix and the image sequence corresponding to each frame of the image are processed by a modeling tool to obtain a sparse point cloud containing the target object and the calibration plate. The calibration plate pose matrix is ​​a coordinate transformation matrix from the coordinate system of the calibration plate to the coordinate system of the camera, and the camera pose matrix is ​​a coordinate transformation matrix from the coordinate system of the camera to the coordinate system of the calibration plate. The second acquisition module is used to align the coordinate system of the sparse point cloud with the coordinate system of the calibration plate to obtain the aligned sparse point cloud, obtain the current scale of the calibration plate from the aligned sparse point cloud, obtain the true scale of the calibration plate from the preset text, and divide the true scale by the current scale to obtain the scale coefficient of the calibration plate; The reconstruction module is used to delete the original point cloud data of the calibration plate in the aligned sparse point cloud to obtain the original point cloud data of the target object, multiply the coordinate value of each point in the original point cloud data of the target object by the scale coefficient to obtain the final point cloud data of the target object, input the final point cloud data and the surround video into the Gaussian splash model, and process the final point cloud data and the surround video through the Gaussian splash model to obtain a reconstructed model that conforms to the physical size of the target object.

[0013] In a third aspect, an embodiment of the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the target object reconstruction method of any one of the first aspects described above is implemented.

[0014] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the target object reconstruction method according to any one of the first aspects is implemented.

[0015] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed on an electronic device, enables the electronic device to execute the target object reconstruction method described in any one of the above-mentioned first aspects.

[0016] The beneficial effects of the embodiments of the present application lie in two aspects. On the one hand, the original point cloud data of the calibration plate is deleted from the aligned sparse point cloud to obtain the original point cloud data of the target object. The coordinate value of each point in the original point cloud data of the target object is multiplied by the scale coefficient to obtain the final point cloud data of the target object. The final point cloud data and the surround video are input into the Gaussian splash model. The final point cloud data and the surround video are processed by the Gaussian splash model to obtain a reconstructed model that meets the physical size of the target object, which solves the technical problem of how to obtain a reconstructed model that meets the physical size of the target object. On the other hand, since the reconstructed model that meets the physical size of the target object is automatically obtained, it will not be affected by human intervention, which is conducive to improving the reliability of the reconstructed model that meets the physical size of the target object. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 This is a diagram of an application scenario of the target object reconstruction method provided in an embodiment of the present application; Figure 2 is a flowchart of a target object reconstruction method provided in an embodiment of the present application; Figure 3 A flowchart showing the results of list processing provided in an embodiment of the present application; Figure 4 A schematic block diagram of a target object reconstruction device provided in an embodiment of the present application; Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application; Figure 6 This is a sample image of the original point cloud data provided in the embodiments of the present application; Figure 7 This is a sample diagram of the reconstructed model provided in the embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of this application more clear, the present application is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0020] The target object reconstruction method provided in the embodiments of the present application can be applied to electronic devices such as mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptops, personal computers, netbooks, personal digital assistants, etc. The embodiments of the present application do not impose any restrictions on the specific type of electronic devices.

[0021] See also Figure 1 , Figure 1 The application scenario diagram of the target object reconstruction method provided in the embodiment of the present application is detailed as follows: Figure 1 The system includes electronic equipment and a camera. The electronic equipment controls the camera to capture a surround image of a target object and a calibration plate next to the target object in the same scene, thereby obtaining a surround video including the target object and the calibration plate.

[0022] In an embodiment of the present application, the electronic device controls the camera to perform surround shooting of the target object and the calibration plate next to the target object in the same scene, and can quickly obtain a surround video including the target object and the calibration plate.

[0023] See also Figure 2 , Figure 2 3 is a flow chart of a target object reconstruction method provided in an embodiment of the present application, which can be applied to electronic devices.

[0024] like Figure 2 As shown, the target object reconstruction method provided in the embodiment of the present application includes the following steps, which are detailed as follows: S201, controlling the camera to perform surround shooting of a target object and a calibration plate next to the target object in the same scene, to obtain a surround video including the target object and the calibration plate; Target objects include but are not limited to cups, shoes, vases, cylindrical containers, chairs, tables, desk lamps, and sofas.

[0025] The calibration plate is a flat plate with known precise size and pattern, which is used to provide a visual reference benchmark.

[0026] The same scene refers to the same room or living room. The target object and the calibration plate next to the target object share the same viewing angle and lighting conditions.

[0027] For ease of explanation, the following examples are given: For example: a target object and a calibration plate next to the target object in the same room.

[0028] For example: a target object and a calibration plate next to the target object in the same living room.

[0029] The control camera performs surround shooting of the target object and the calibration plate next to the target object in the same scene to obtain a surround video containing the target object and the calibration plate, including: Obtain a path that simultaneously surrounds the target object and the calibration plate next to the target object, and select the path that simultaneously surrounds the target object and the calibration plate next to the target object as the surrounding path; A target object and a calibration plate next to the target object in the same scene are selected as the shooting objects, and the camera is controlled to surround the shooting object along the surround path to obtain a surround video containing the target object and the calibration plate.

[0030] Optionally, a target object and a calibration plate next to the target object in the same scene are selected as the shooting objects, and the camera is controlled to surround the shooting object along a surround path to obtain a surround video containing the target object and the calibration plate, including: The target object and the calibration plate next to the target object in the same scene are selected as the shooting objects. By holding the camera, the camera is controlled to surround the shooting object along the surround path to obtain a surround video containing the target object and the calibration plate.

[0031] Optionally, a target object and a calibration plate next to the target object in the same scene are selected as the shooting objects, and the camera is controlled to surround the shooting object along a surround path to obtain a surround video containing the target object and the calibration plate, including: A target object and a calibration plate next to the target object in the same scene are selected as the shooting objects. The robot is connected wirelessly and a shooting command is sent to the robot. The robot's camera is controlled by the shooting command to surround the shooting object along a surround path to obtain a surround video containing the target object and the calibration plate.

[0032] S202, decomposing the surround video into multiple frames of images, and arranging the multiple frames of images in the order of shooting time to obtain an image sequence; S203, obtaining a calibration plate pose matrix corresponding to each frame of the multiple images, performing an inverse operation on the calibration plate pose matrix corresponding to each frame of the image, obtaining a camera pose matrix corresponding to each frame of the image, processing the camera pose matrix corresponding to each frame of the image and the image sequence using a modeling tool, and obtaining a sparse point cloud containing the target object and the calibration plate, wherein the calibration plate pose matrix is ​​a coordinate transformation matrix from the coordinate system of the calibration plate to the coordinate system of the camera, and the camera pose matrix is ​​a coordinate transformation matrix from the coordinate system of the camera to the coordinate system of the calibration plate; Among them, the calibration plate pose matrix corresponding to each frame of the multiple images is obtained, and the calibration plate pose matrix corresponding to each frame of the image is inverted to obtain the camera pose matrix corresponding to each frame of the image. The camera pose matrix and image sequence corresponding to each frame of the image are processed by the modeling tool to obtain a sparse point cloud containing the target object and the calibration plate, including: The feature points of the calibration plate of each frame image are detected to obtain the calibration plate pose matrix corresponding to each frame image, and the calibration plate pose matrix corresponding to each frame image is inverted to obtain the camera pose matrix corresponding to each frame image. The camera pose matrix and image sequence corresponding to each frame image are input into the modeling tool. The camera pose matrix and image sequence corresponding to each frame image are processed by the modeling tool to obtain a sparse point cloud containing the target object and the calibration plate. The calibration plate pose matrix is ​​the coordinate transformation matrix from the coordinate system of the calibration plate to the coordinate system of the camera, and the camera pose matrix is ​​the coordinate transformation matrix from the coordinate system of the camera to the coordinate system of the calibration plate.

[0033] In the task of 3D reconstruction, obtaining the camera pose matrix corresponding to each frame of image can anchor discrete observation data to a unified spatiotemporal reference. Specifically, the camera pose matrix corresponding to each frame of image solves the spatial discretization problem caused by camera motion in multi-view data by converting local 3D information from the camera's perspective to a global reference frame.

[0034] Among them, by combining the camera pose matrix corresponding to each frame of image and the image sequence, a sparse point cloud containing the target object and the calibration plate can be constructed. This process provides key spatial structure and scale information for three-dimensional scene understanding.

[0035] S204: aligning the coordinate system of the sparse point cloud with the coordinate system of the calibration plate to obtain an aligned sparse point cloud, obtaining the current scale of the calibration plate from the aligned sparse point cloud, obtaining the true scale of the calibration plate from a preset text, and dividing the true scale by the current scale to obtain a scale factor of the calibration plate; Among them, the aligned sparse point cloud can optimize data storage and computing efficiency, and avoid the additional overhead caused by overlapping or redundant point clouds.

[0036] The steps of aligning the coordinate system of the sparse point cloud with the coordinate system of the calibration plate to obtain the aligned sparse point cloud, obtaining the current scale of the calibration plate from the aligned sparse point cloud, obtaining the true scale of the calibration plate from the preset text, and dividing the true scale by the current scale to obtain the scale factor of the calibration plate include: Use the alignment module of the modeling tool to align the coordinate system of the sparse point cloud with the coordinate system of the calibration plate to obtain the aligned sparse point cloud; The current scale of the calibration plate is obtained from the aligned sparse point cloud, the true scale of the calibration plate is obtained from the preset text, and the true scale is divided by the current scale to obtain the scale coefficient of the calibration plate.

[0037] For ease of explanation, the following examples are given: For example, the current length of the calibration plate in the aligned sparse point cloud is 10 mm, the actual length of the calibration plate is 100 mm, and the scale factor is 10.

[0038] For example, the current length of the calibration plate in the aligned sparse point cloud is 20 mm, the actual length of the calibration plate is 100 mm, and the scale factor is 5.

[0039] The modeling tool includes one of a COLMAP tool and a Blender tool or a combination thereof.

[0040] Preferably, the modeling tool adopts COLMAP tool.

[0041] In the field of 3D modeling, COLMAP and Blender are both open-source 3D modeling tools. The full name of COLMAP is "3D reconstruction tool based on structure from motion and multi-view stereo vision."

[0042] S205, delete the original point cloud data of the calibration plate from the aligned sparse point cloud to obtain the original point cloud data of the target object, multiply the coordinate value of each point in the original point cloud data of the target object by the scale coefficient to obtain the final point cloud data of the target object, input the final point cloud data and the surround video into the Gaussian splash model, process the final point cloud data and the surround video through the Gaussian splash model to obtain a reconstructed model that meets the physical size of the target object.

[0043] The Gaussian splash model adopts a three-dimensional Gaussian splash model or a two-dimensional Gaussian splash model.

[0044] Among them, the full English name of the three-dimensional Gaussian splatting model is: 3D Gaussian Splatting, and the abbreviation of the full English name of the three-dimensional Gaussian splatting model is: 3DGS.

[0045] Among them, the full English name of the two-dimensional Gaussian splatting model is: 2D Gaussian Splatting, and the abbreviation of the full English name of the two-dimensional Gaussian splatting model is: 2DGS.

[0046] Among them, the scale coefficient plays a key role in scale normalization in the reconstruction process of the target object. Since the reconstruction process of the target object can usually only obtain the geometric shape of the target object but cannot directly determine the physical size of the target object, the calibration plate is used as a reference object, and the scale coefficient of the calibration plate is applied to the original point cloud data of the target object. By multiplying the coordinate value of each point in the original point cloud data of the target object by the scale coefficient, the final point cloud data of the target object is obtained. Since the final point cloud data can reflect the true geometric characteristics of the target object, the final point cloud data lays a precise foundation for the modeling of the target object.

[0047] In addition, the final point cloud data and surround video are input into the Gaussian splash model, and the final point cloud data and surround video are processed by the Gaussian splash model to obtain a reconstructed model that conforms to the physical dimensions of the target object. The reconstructed model that conforms to the physical dimensions of the target object can be directly used in application scenarios that require precise dimensions, such as engineering inspection and reverse design.

[0048] The method includes deleting the original point cloud data of the calibration plate from the aligned sparse point cloud to obtain the original point cloud data of the target object, multiplying the coordinate value of each point in the original point cloud data of the target object by a scale factor to obtain the final point cloud data of the target object, inputting the final point cloud data and the surround video into a Gaussian splash model, and processing the final point cloud data and the surround video through the Gaussian splash model to obtain a reconstructed model that conforms to the physical size of the target object, including: The original point cloud data of the calibration plate is deleted from the aligned sparse point cloud to obtain the original point cloud data of the target object. The coordinate value of each point in the original point cloud data of the target object is multiplied by the scale coefficient to obtain each adjusted point. Each adjusted point is combined into the final point cloud data of the target object. Access the preset file, load the Gaussian splash model from the preset file, input the final point cloud data and surround video into the Gaussian splash model, and process the final point cloud data and surround video through the Gaussian splash model to obtain a reconstructed model that meets the physical size of the target object.

[0049] The target object reconstruction method comprises the following steps: deleting the original point cloud data of the calibration plate from the aligned sparse point cloud to obtain the original point cloud data of the target object; multiplying the coordinate value of each point in the original point cloud data of the target object by a scale factor to obtain the final point cloud data of the target object; inputting the final point cloud data and the surround video into a Gaussian splash model; processing the final point cloud data and the surround video using the Gaussian splash model to obtain a reconstructed model that conforms to the physical size of the target object. Step A: creating a display page, and displaying the reconstructed model that meets the physical size of the target object through the display page.

[0050] The target object reconstruction method comprises the following steps: deleting the original point cloud data of the calibration plate from the aligned sparse point cloud to obtain the original point cloud data of the target object; multiplying the coordinate value of each point in the original point cloud data of the target object by a scale factor to obtain the final point cloud data of the target object; inputting the final point cloud data and the surround video into a Gaussian splash model; processing the final point cloud data and the surround video using the Gaussian splash model to obtain a reconstructed model that conforms to the physical size of the target object. Step B: Connect to a preset server and upload a reconstructed model that matches the physical size of the target object to the server.

[0051] Among them, a reconstructed model that meets the physical dimensions of the target object is uploaded to the server. Even if data is lost or damaged on the local device, the reconstructed model can be obtained from the server, thereby ensuring the security and integrity of the reconstructed model.

[0052] Among them, step A and step B can be executed simultaneously or at different times, and the specific execution order is not limited here.

[0053] The beneficial effects of the embodiments of the present application lie in two aspects. On the one hand, the original point cloud data of the calibration plate is deleted from the aligned sparse point cloud to obtain the original point cloud data of the target object. The coordinate value of each point in the original point cloud data of the target object is multiplied by the scale coefficient to obtain the final point cloud data of the target object. The final point cloud data and the surround video are input into the Gaussian splash model. The final point cloud data and the surround video are processed by the Gaussian splash model to obtain a reconstructed model that meets the physical size of the target object, which solves the technical problem of how to obtain a reconstructed model that meets the physical size of the target object. On the other hand, since the reconstructed model that meets the physical size of the target object is automatically obtained, it will not be affected by human intervention, which is conducive to improving the reliability of the reconstructed model that meets the physical size of the target object.

[0054] See also Figure 3 , Figure 3 The flowchart of the display list processing results provided in the embodiment of the present application is detailed as follows: S301, deleting the original point cloud data of the calibration plate from the aligned sparse point cloud to obtain the original point cloud data of the target object, multiplying the coordinate value of each point in the original point cloud data of the target object by a scale factor to obtain each adjusted point, and combining each adjusted point into the final point cloud data of the target object; S302, access the preset file, load the Gaussian splash model from the preset file, input the final point cloud data and surround video into the Gaussian splash model, process the final point cloud data and surround video through the Gaussian splash model, and obtain a reconstructed model that meets the physical size of the target object.

[0055] Among them, each point of the original point cloud data of the target object is different.

[0056] For ease of explanation, the following examples are given: For example, the target object is a cylindrical container for potato chips.

[0057] refer to Figure 6 , Figure 6 This is a sample image of the original point cloud data provided in the embodiments of this application.

[0058] Figure 6 shows the original point cloud data of a cylindrical container, Figure 6 The arcs in the figure can guide the viewer's sight and help understand the original point cloud data of the cylindrical container.

[0059] refer to Figure 7 , Figure 7 This is a sample diagram of the reconstructed model provided in the embodiment of the present application.

[0060] in, Figure 7 shows a reconstructed model that conforms to the physical dimensions of the cylindrical container. Figure 7 Also shown are the X-axis, Y-axis, and Z-axis; The color of the X axis is set to red; the color of the Y axis is set to green, and the color of the Z axis is set to blue.

[0061] To facilitate the display of the reconstructed model that conforms to the physical dimensions of the cylindrical container, Figure 7 Adaptive adjustments were made. Figure 7 The arcs in the figure can guide the viewer's eyes and help understand the reconstruction model that conforms to the physical dimensions of the cylindrical container.

[0062] In the embodiment of the present application, since the reconstructed model that meets the physical size of the target object is automatically acquired and will not be affected by human intervention, it is beneficial to improve the reliability of the reconstructed model that meets the physical size of the target object.

[0063] For the target object reconstruction method described in the above embodiment, please refer to Figure 4 , Figure 4 This is a schematic block diagram of a target object reconstruction device provided in an embodiment of the present application. Figure 4 The target object reconstruction apparatus 400 shown can be applied to Figure 1 The electronic device in the application scenario diagram shown below takes the electronic device as an example. Figure 4 The target object reconstruction device 400 shown in FIG. 4 is described in detail. The target object reconstruction device 400 may include a shooting module 401 , an arrangement module 402 , a first acquisition module 403 , a second acquisition module 404 , and a reconstruction module 405 .

[0064] The shooting module 401 is used to control the camera to perform surround shooting of the target object and the calibration plate next to the target object in the same scene, thereby obtaining a surround video including the target object and the calibration plate; an arrangement module 402 for decomposing the surround video into multiple frames of images and arranging the multiple frames of images in the order of shooting time to obtain an image sequence; The first acquisition module 403 is used to obtain the calibration plate pose matrix corresponding to each frame of the multiple images, perform an inverse operation on the calibration plate pose matrix corresponding to each frame of the image, obtain the camera pose matrix corresponding to each frame of the image, and process the camera pose matrix and the image sequence corresponding to each frame of the image through a modeling tool to obtain a sparse point cloud containing the target object and the calibration plate. The calibration plate pose matrix is ​​a coordinate transformation matrix from the coordinate system of the calibration plate to the coordinate system of the camera, and the camera pose matrix is ​​a coordinate transformation matrix from the coordinate system of the camera to the coordinate system of the calibration plate. The second acquisition module 404 is configured to align the coordinate system of the sparse point cloud with the coordinate system of the calibration plate to obtain an aligned sparse point cloud, obtain the current scale of the calibration plate from the aligned sparse point cloud, obtain the true scale of the calibration plate from a preset text, and divide the true scale by the current scale to obtain a scale factor of the calibration plate; The reconstruction module 405 is used to delete the original point cloud data of the calibration plate from the aligned sparse point cloud to obtain the original point cloud data of the target object, multiply the coordinate value of each point in the original point cloud data of the target object by the scale coefficient to obtain the final point cloud data of the target object, input the final point cloud data and the surround video into the Gaussian splash model, and process the final point cloud data and the surround video through the Gaussian splash model to obtain a reconstructed model that conforms to the physical size of the target object.

[0065] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referenced to each other.

[0066] The beneficial effects of the embodiments of the present application lie in two aspects. On the one hand, the original point cloud data of the calibration plate is deleted from the aligned sparse point cloud to obtain the original point cloud data of the target object. The coordinate value of each point in the original point cloud data of the target object is multiplied by the scale coefficient to obtain the final point cloud data of the target object. The final point cloud data and the surround video are input into the Gaussian splash model. The final point cloud data and the surround video are processed by the Gaussian splash model to obtain a reconstructed model that meets the physical size of the target object, which solves the technical problem of how to obtain a reconstructed model that meets the physical size of the target object. On the other hand, since the reconstructed model that meets the physical size of the target object is automatically obtained, it will not be affected by human intervention, which is conducive to improving the reliability of the reconstructed model that meets the physical size of the target object.

[0067] See also Figure 5 , Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0068] like Figure 5 As shown, Figure 5 The electronic device 2 includes: at least one processor 20, a memory 21, and a computer program 22 stored in the memory 21 and executable on the at least one processor 20, wherein the processor 20 implements the steps of any of the above-mentioned method embodiments when executing the computer program 22.

[0069] The electronic device 2 may include, but is not limited to, a processor 20 and a memory 21. Those skilled in the art will appreciate that Figure 5 This is merely an example of the electronic device 2 and does not constitute a limitation on the electronic device 2 . The electronic device 2 may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device 2 may also include input and output devices, network access devices, etc.

[0070] The processor 20 is configured to run a computer program 22 stored in the memory 21 and implement the following steps when executing the computer program 22: Control the camera to capture a surround video of the target object and the calibration plate next to the target object in the same scene, and obtain a surround video containing the target object and the calibration plate; Decompose the surround video into multiple frames of images, and arrange the multiple frames of images in the order of shooting time to obtain an image sequence; Obtain the calibration plate pose matrix corresponding to each frame of multiple images, perform inverse operation on the calibration plate pose matrix corresponding to each frame of image, and obtain the camera pose matrix corresponding to each frame of image. Use modeling tools to process the camera pose matrix and image sequence corresponding to each frame of image to obtain a sparse point cloud containing the target object and the calibration plate. The calibration plate pose matrix is ​​the coordinate transformation matrix from the calibration plate's coordinate system to the camera's coordinate system, and the camera pose matrix is ​​the coordinate transformation matrix from the camera's coordinate system to the calibration plate's coordinate system. Align the coordinate system of the sparse point cloud with the coordinate system of the calibration plate to obtain the aligned sparse point cloud, obtain the current scale of the calibration plate from the aligned sparse point cloud, obtain the true scale of the calibration plate from the preset text, and divide the true scale by the current scale to obtain the scale coefficient of the calibration plate; The original point cloud data of the calibration plate is deleted from the aligned sparse point cloud to obtain the original point cloud data of the target object. The coordinate value of each point in the original point cloud data of the target object is multiplied by the scale coefficient to obtain the final point cloud data of the target object. The final point cloud data and the surround video are input into the Gaussian splash model. The final point cloud data and the surround video are processed by the Gaussian splash model to obtain a reconstructed model that conforms to the physical size of the target object.

[0071] In some embodiments, the processor 20 is configured to implement: Obtain a path that simultaneously surrounds the target object and the calibration plate next to the target object, and select the path that simultaneously surrounds the target object and the calibration plate next to the target object as the surrounding path; A target object and a calibration plate next to the target object in the same scene are selected as the shooting objects, and the camera is controlled to surround the shooting object along the surround path to obtain a surround video containing the target object and the calibration plate.

[0072] In some embodiments, the processor 20 is configured to implement: Use the alignment module of the modeling tool to align the coordinate system of the sparse point cloud with the coordinate system of the calibration plate to obtain the aligned sparse point cloud; The current scale of the calibration plate is obtained from the aligned sparse point cloud, the true scale of the calibration plate is obtained from the preset text, and the true scale is divided by the current scale to obtain the scale coefficient of the calibration plate.

[0073] In some embodiments, the processor 20 is configured to implement: The original point cloud data of the calibration plate is deleted from the aligned sparse point cloud to obtain the original point cloud data of the target object. The coordinate value of each point in the original point cloud data of the target object is multiplied by the scale coefficient to obtain each adjusted point. Each adjusted point is combined into the final point cloud data of the target object. Access the preset file, load the Gaussian splash model from the preset file, input the final point cloud data and surround video into the Gaussian splash model, and process the final point cloud data and surround video through the Gaussian splash model to obtain a reconstructed model that meets the physical size of the target object.

[0074] In some embodiments, the processor 20 is configured to implement: A display page is created to display the reconstructed model that conforms to the physical size of the target object.

[0075] In some embodiments, the processor 20 is configured to implement: Connect to the preset server and upload the reconstructed model that meets the physical size of the target object to the server.

[0076] The processor 20 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors, application-specific integrated circuits, field programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor.

[0077] In some embodiments, the memory 21 may be an internal storage unit of the electronic device 2, such as a hard disk or memory of the electronic device 2. In other embodiments, the memory 21 may also be an external storage device of the electronic device 2, such as a plug-in hard disk, a smart memory card, a Secure Digital (SD) card, a flash card, etc. equipped on the electronic device 2. Furthermore, the memory 21 may include both an internal storage unit of the electronic device 2 and an external storage device. The memory 21 is used to store an operating system, application programs, a boot loader, data, and other programs, such as the program code of the computer program. The memory 21 may also be used to temporarily store data that has been output or is about to be output.

[0078] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of this application. Their specific functions and technical effects can be found in the method embodiment section and will not be repeated here.

[0079] An embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0080] The computer-readable storage medium stores program codes, which can be called by a processor to execute the target object reconstruction method described in the above method embodiment.

[0081] The computer-readable storage medium has a storage space for program codes.

[0082] The program code includes the code of any step in the target object reconstruction method described in the above method embodiment.

[0083] The specific implementation of the above operations can be found in the previous embodiments and will not be repeated here.

[0084] Since the computer program stored in the computer-readable storage medium can execute any target object reconstruction method provided in the embodiments of the present application, the computer-readable storage medium can achieve the beneficial effects that can be achieved by any target object reconstruction method provided in the embodiments of the present application. Please see the previous embodiments for details and will not be repeated here.

[0085] An embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the above-mentioned target object reconstruction method.

[0086] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0087] The above are only preferred embodiments of the present application and do not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A target object reconstruction method, characterized in that: Applied to electronic equipment, the target object reconstruction method includes: Control the camera to capture a surround video of the target object and the calibration plate next to the target object in the same scene, and obtain a surround video containing the target object and the calibration plate; Decompose the surround video into multiple frames of images, and arrange the multiple frames of images in the order of shooting time to obtain an image sequence; Obtain the calibration plate pose matrix corresponding to each frame of multiple images, perform inverse operation on the calibration plate pose matrix corresponding to each frame of image, and obtain the camera pose matrix corresponding to each frame of image. Use modeling tools to process the camera pose matrix and image sequence corresponding to each frame of image to obtain a sparse point cloud containing the target object and the calibration plate. The calibration plate pose matrix is ​​the coordinate transformation matrix from the calibration plate's coordinate system to the camera's coordinate system, and the camera pose matrix is ​​the coordinate transformation matrix from the camera's coordinate system to the calibration plate's coordinate system. Align the coordinate system of the sparse point cloud with the coordinate system of the calibration plate to obtain the aligned sparse point cloud, obtain the current scale of the calibration plate from the aligned sparse point cloud, obtain the true scale of the calibration plate from the preset text, and divide the true scale by the current scale to obtain the scale coefficient of the calibration plate; The original point cloud data of the calibration plate is deleted from the aligned sparse point cloud to obtain the original point cloud data of the target object. The coordinate value of each point in the original point cloud data of the target object is multiplied by the scale coefficient to obtain the final point cloud data of the target object. The final point cloud data and the surround video are input into the Gaussian splash model. The final point cloud data and the surround video are processed by the Gaussian splash model to obtain a reconstructed model that conforms to the physical size of the target object.

2. The target object reconstruction method according to claim 1, characterized in that: The control camera performs surround shooting of the target object and the calibration plate next to the target object in the same scene, and obtains a surround video including the target object and the calibration plate, including: Obtain a path that simultaneously surrounds the target object and the calibration plate next to the target object, and select the path that simultaneously surrounds the target object and the calibration plate next to the target object as the surrounding path; A target object and a calibration plate next to the target object in the same scene are selected as the shooting objects, and the camera is controlled to surround the shooting object along the surround path to obtain a surround video containing the target object and the calibration plate.

3. The target object reconstruction method according to claim 1, characterized in that: The method aligns the coordinate system of the sparse point cloud with the coordinate system of the calibration plate to obtain an aligned sparse point cloud, obtains the current scale of the calibration plate from the aligned sparse point cloud, obtains the true scale of the calibration plate from a preset text, and divides the true scale by the current scale to obtain a scale factor of the calibration plate, including: Use the alignment module of the modeling tool to align the coordinate system of the sparse point cloud with the coordinate system of the calibration plate to obtain the aligned sparse point cloud; The current scale of the calibration plate is obtained from the aligned sparse point cloud, the true scale of the calibration plate is obtained from the preset text, and the true scale is divided by the current scale to obtain the scale coefficient of the calibration plate.

4. The target object reconstruction method according to claim 1, characterized in that: The method includes deleting the original point cloud data of the calibration plate from the aligned sparse point cloud to obtain the original point cloud data of the target object, multiplying the coordinate value of each point in the original point cloud data of the target object by a scale factor to obtain the final point cloud data of the target object, inputting the final point cloud data and the surround video into a Gaussian splash model, and processing the final point cloud data and the surround video through the Gaussian splash model to obtain a reconstructed model that conforms to the physical size of the target object, including: The original point cloud data of the calibration plate is deleted from the aligned sparse point cloud to obtain the original point cloud data of the target object. The coordinate value of each point in the original point cloud data of the target object is multiplied by the scale coefficient to obtain each adjusted point. Each adjusted point is combined into the final point cloud data of the target object. Access the preset file, load the Gaussian splash model from the preset file, input the final point cloud data and surround video into the Gaussian splash model, and process the final point cloud data and surround video through the Gaussian splash model to obtain a reconstructed model that meets the physical size of the target object.

5. The target object reconstruction method according to claim 1, characterized in that: The target object reconstruction method comprises: deleting the original point cloud data of the calibration plate from the aligned sparse point cloud to obtain the original point cloud data of the target object; multiplying the coordinate value of each point in the original point cloud data of the target object by a scale factor to obtain the final point cloud data of the target object; inputting the final point cloud data and the surround video into a Gaussian splash model; processing the final point cloud data and the surround video using the Gaussian splash model to obtain a reconstructed model that conforms to the physical size of the target object. A display page is created to display the reconstructed model that conforms to the physical size of the target object.

6. The target object reconstruction method according to claim 1, characterized in that: The target object reconstruction method comprises: deleting the original point cloud data of the calibration plate from the aligned sparse point cloud to obtain the original point cloud data of the target object; multiplying the coordinate value of each point in the original point cloud data of the target object by a scale factor to obtain the final point cloud data of the target object; inputting the final point cloud data and the surround video into a Gaussian splash model; processing the final point cloud data and the surround video using the Gaussian splash model to obtain a reconstructed model that conforms to the physical size of the target object. Connect to the preset server and upload the reconstructed model that meets the physical size of the target object to the server.

7. The target object reconstruction method according to any one of claims 1 to 6, characterized in that: The modeling tool includes one of a COLMAP tool and a Blender tool or a combination thereof.

8. A target object reconstruction device, characterized in that: Used in electronic equipment, including: The shooting module is used to control the camera to surround the target object and the calibration plate next to the target object in the same scene to obtain a surround video containing the target object and the calibration plate; An arrangement module is used to decompose the surround video into multiple frames of images and arrange the multiple frames of images in the order of shooting time to obtain an image sequence; The first acquisition module is used to obtain the calibration plate pose matrix corresponding to each frame of the multiple images, perform an inverse operation on the calibration plate pose matrix corresponding to each frame of the image, and obtain the camera pose matrix corresponding to each frame of the image. The camera pose matrix and the image sequence corresponding to each frame of the image are processed by a modeling tool to obtain a sparse point cloud containing the target object and the calibration plate. The calibration plate pose matrix is ​​a coordinate transformation matrix from the coordinate system of the calibration plate to the coordinate system of the camera, and the camera pose matrix is ​​a coordinate transformation matrix from the coordinate system of the camera to the coordinate system of the calibration plate. The second acquisition module is used to align the coordinate system of the sparse point cloud with the coordinate system of the calibration plate to obtain the aligned sparse point cloud, obtain the current scale of the calibration plate from the aligned sparse point cloud, obtain the true scale of the calibration plate from the preset text, and divide the true scale by the current scale to obtain the scale coefficient of the calibration plate; The reconstruction module is used to delete the original point cloud data of the calibration plate in the aligned sparse point cloud to obtain the original point cloud data of the target object, multiply the coordinate value of each point in the original point cloud data of the target object by the scale coefficient to obtain the final point cloud data of the target object, input the final point cloud data and the surround video into the Gaussian splash model, and process the final point cloud data and the surround video through the Gaussian splash model to obtain a reconstructed model that conforms to the physical size of the target object.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the target object reconstruction method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the target object reconstruction method according to any one of claims 1 to 7 is implemented.

Images