Three-dimensional object detection method, electronic device, and computer-readable storage medium

Abstract

The application discloses a three-dimensional object detection method, an electronic device and a computer readable storage medium. The geometric model information of a target object is obtained by collecting multi-view pictures of the target object, and then the initial frame target object pose is estimated based on the geometric model information using a brute force matching method. Finally, the target object is continuously tracked and the pose is estimated using an energy function defined based on a region color histogram, and the motion tracking model is used to correct and optimize the pose estimation result. Specifically, the target object geometry information is extracted, and then the object detection and tracking are performed according to the extracted geometry information. In the detection process, the initial pose estimation process is added to avoid using third-party software to provide the initial pose. In the tracking process, the motion tracking model is used to correct and optimize the target object pose, so that the object tracking is more stable. Therefore, the application has the advantages of convenient use, fast running speed and stable tracking.

Description

Technical Field

[0001] The embodiments of the present invention relate to, but are not limited to, the field of augmented reality technology, and in particular to a three-dimensional object detection method, an electronic device, and a computer-readable storage medium. Background Technology

[0002] In the field of augmented reality, virtual interaction with real-world objects is crucial, such as dynamic instruction manuals and virtual creation. This involves 3D object detection technology. Currently, existing 3D object detection technologies can be mainly divided into three types: 3D object detection using deep learning, object detection using traditional point cloud matching, and object detection based on object geometry information. The first method uses deep learning to predict the depth map corresponding to the acquired 2D image, then performs 3D reconstruction to estimate the pose of the target object in 3D space. This method requires high-quality training of the deep learning network, necessitating a large-scale dataset with specific data distribution requirements. The second method pre-collects point cloud data of the target object, then associates the target point cloud with a 2D image containing the target object to estimate the pose of the target object in 3D space. The main drawback of this method is that high-quality point cloud acquisition requires sophisticated equipment, and the point cloud matching accuracy is relatively low. The third method requires a 3D model of the target object and its initial pose, then uses edge distance fields or region color histograms for 3D object tracking. The tracking performance of this type of method depends on the quality of the provided object model and the accuracy of the initial pose. Summary of the Invention

[0003] The following is an overview of the subject matter described in detail herein. This overview is not intended to limit the scope of the claims.

[0004] This invention provides a three-dimensional object detection method, an electronic device, and a computer-readable storage medium, which have the advantages of being easy to use, running fast, and more stable in tracking compared to existing three-dimensional object detection and attitude tracking methods.

[0005] In a first aspect, embodiments of the present invention provide a three-dimensional object detection method, comprising: Collect multi-view images of the target object to obtain the geometric model information of the target object; Based on the geometric model information, the estimated pose of the target object in the initial frame is calculated using a brute-force matching algorithm; The estimated pose of the target object is continuously tracked based on the energy function defined by the local color histogram. The pose estimation result of the target object is corrected using a motion tracking model, wherein the pose estimation result indicates the estimated pose of the target object in three-dimensional space.

[0006] In a second aspect, embodiments of the present invention provide a three-dimensional object detection device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the three-dimensional object detection method described in the first aspect above.

[0007] Thirdly, embodiments of the present invention provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the three-dimensional object detection method described in the first aspect above.

[0008] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer-executable program for causing a computer to perform the three-dimensional object detection method described in the first aspect above.

[0009] This invention includes: acquiring multi-view images of a target object to obtain its geometric model information; calculating the estimated initial pose of the target object in the initial frame using a brute-force matching algorithm based on the geometric model information; continuously tracking the estimated pose of the target object using an energy function defined by a local color histogram; and correcting the pose estimation result of the target object using a motion tracking model, wherein the pose estimation result indicates the estimated pose of the target object in three-dimensional space. This invention achieves three-dimensional object detection and tracking through three-dimensional object geometric model construction, three-dimensional object initial pose estimation, and three-dimensional object tracking. The purpose of three-dimensional object reconstruction is to provide geometric model information of the target object for subsequent initial pose estimation and object tracking. Initial pose estimation is used for object tracking initialization, while three-dimensional object tracking can obtain the spatial position information of the target object in real time. Based on this, multi-view images of the target object are acquired to obtain its geometric model information. Then, a brute-force matching method is used to estimate the target object's pose in the initial frame based on this geometric model information. Finally, an energy function defined by a local color histogram is used to continuously track and estimate the pose of the target object, and a motion tracking model is used to correct and optimize the pose estimation result. Specifically, the geometric information of the target object is extracted, and then object detection and tracking are performed based on this extracted geometric information. During the detection process, an initial pose estimation procedure is added to avoid using third-party software to provide the initial pose. During tracking, a motion model is used to correct and optimize the target object's pose, resulting in more stable object tracking. Therefore, this invention has the advantages of ease of use, fast operation, and more stable tracking.

[0010] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the description, claims, and drawings. Attached Figure Description

[0011] The accompanying drawings are provided to further understand the technical solutions of the present invention and constitute a part of the specification. They are used together with the embodiments of the present invention to explain the technical solutions of the present invention, and do not constitute a limitation on the technical solutions of the present invention.

[0012] Figure 1 This is a main flowchart of a three-dimensional object detection method provided in one embodiment of the present invention; Figure 2 This is a schematic diagram of a three-dimensional object detection device according to an embodiment of the present invention; Figure 3 This is a sub-flowchart of a three-dimensional object detection method provided in one embodiment of the present invention; Figure 4 This is a sub-flowchart of a three-dimensional object detection method provided in one embodiment of the present invention; Figure 5 This is a sub-flowchart of a three-dimensional object detection method provided in one embodiment of the present invention; Figure 6 This is a schematic diagram of the structure of a three-dimensional object detection device provided in one embodiment of the present invention; Figure 7 This is a schematic diagram of an electronic device structure provided in one embodiment of the present invention. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0014] It should be understood that in the description of the embodiments of the present invention, "multiple" (or "amounts") means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. If "first," "second," etc., are used in the description, they are only for the purpose of distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0015] In the field of augmented reality, virtual interaction with real-world objects is crucial, such as dynamic instruction manuals and virtual creation. This involves 3D object detection technology. Currently, existing 3D object detection technologies can be mainly divided into three types: 3D object detection using deep learning, object detection using traditional point cloud matching, and object detection based on object geometry information. The first method uses deep learning to predict the depth map corresponding to the acquired 2D image, then performs 3D reconstruction to estimate the pose of the target object in 3D space. This method requires high-quality training of the deep learning network, necessitating a large-scale dataset with specific data distribution requirements. The second method pre-collects point cloud data of the target object, then associates the target point cloud with a 2D image containing the target object to estimate the pose of the target object in 3D space. The main drawback of this method is that high-quality point cloud acquisition requires sophisticated equipment, and the point cloud matching accuracy is relatively low. The third method requires a 3D model of the target object and its initial pose, then uses edge distance fields or region color histograms for 3D object tracking. The tracking performance of this type of method depends on the quality of the provided object model and the accuracy of the initial pose.

[0016] To address the problems existing in the prior art, this invention provides a 3D object detection method, electronic device, and computer-readable storage medium. The method involves acquiring multi-view images of a target object to obtain its geometric model information; calculating the estimated pose of the target object in the initial frame using a brute-force matching algorithm based on the geometric model information; continuously tracking the estimated pose of the target object based on an energy function defined by a local color histogram; and correcting the pose estimation result of the target object using a motion tracking model. The pose estimation result indicates the estimated pose of the target object in 3D space. This invention achieves 3D object detection and tracking through 3D object geometric model construction, initial 3D object pose estimation, and 3D object tracking. The purpose of 3D object reconstruction is to provide geometric model information of the target object for subsequent initial pose estimation and object tracking. Initial pose estimation is used for object tracking initialization, while 3D object tracking can obtain the spatial position information of the target object in real time. Based on this, the geometric model information of the target object is obtained by acquiring multi-view images of the target object. Then, based on this geometric model information, a brute-force matching method is used to estimate the target object's pose in the initial frame. Finally, an energy function defined based on the region color histogram is used for continuous tracking and pose estimation of the target object, and a motion tracking model is used to correct and optimize the pose estimation results. Specifically, the geometric information of the target object is extracted, and then object detection and tracking are performed based on the extracted geometric information. During the detection process, an initial pose estimation process is added to avoid using third-party software to provide the initial pose. During the tracking process, a motion tracking model is used to correct and optimize the target object's pose, making object tracking more stable. Therefore, this invention has the advantages of being easy to use, running quickly, and providing more stable tracking.

[0017] like Figure 1 As shown, Figure 1 This is a flowchart illustrating a three-dimensional object detection method according to an embodiment of the present invention. The three-dimensional object detection method includes, but is not limited to, the following steps: Step 101: Collect multi-view images of the target object to obtain the geometric model information of the target object; Step 102: Calculate the estimated pose of the target object in the initial frame using a brute-force matching algorithm based on the geometric model information; Step 103: Continuously track the estimated pose of the target object based on the energy function defined by the local color histogram; Step 104: Correct the pose estimation result of the target object using a motion tracking model, wherein the pose estimation result indicates the estimated pose of the target object in three-dimensional space.

[0018] Understandably, this method can solve the problem of detection and trajectory tracking for three-dimensional rigid objects. Specifically, it first acquires the target three-dimensional object model and a two-dimensional temporal image containing the target object, and then estimates the pose of the target object in three-dimensional space based on the target object's three-dimensional model in the two-dimensional image, so as to track the three-dimensional object in the real world in real time and obtain the target object's pose in three-dimensional space.

[0019] It is understood that 3D object detection and tracking are achieved through 3D object geometric model construction, 3D object initial pose estimation, and 3D object tracking. Specifically, 3D object reconstruction aims to provide geometric model information of the target object for subsequent initial pose estimation and object tracking. Initial pose estimation is used for object tracking initialization, while 3D object tracking can obtain the spatial position information of the target object in real time. Based on this, the geometric model information of the target object is obtained by acquiring multi-view images. Then, based on this geometric model information, a brute-force matching method is used to estimate the target object's pose in the initial frame. Finally, an energy function defined based on the region color histogram is used for continuous tracking and pose estimation of the target object, and a motion tracking model is used to correct and optimize the pose estimation results. Specifically, the geometric information of the target object is extracted, and then object detection and tracking are performed based on the extracted geometric information. During the detection process, an initial pose estimation process is added to avoid using third-party software to provide the initial pose. During the tracking process, a motion tracking model is used to correct and optimize the target object's pose, making object tracking more stable. Therefore, this invention has the advantages of ease of use, fast operation speed, and more stable tracking.

[0020] Understandably, constructing a 3D object's geometric model mainly involves three parts: image acquisition, reconstruction, and subtraction. During image acquisition, the target object is placed on a horizontal plane, with the entire object occupying 70% of the image during capture. Shooting angles are set at 22.5° intervals horizontally and vertically, taking pictures directly at the target object. Only one image is taken when the vertical angle is 90°, for a total of 49 shooting positions. During reconstruction, SFM is used for sparse reconstruction, MVS for dense reconstruction, and finally, surface reconstruction is performed to obtain the target object's mesh model. Finally, geometric information is extracted from the reconstructed model, and surface smoothing and surface reduction algorithms are used to reduce redundant content.

[0021] Understandably, for the initial pose estimation of a 3D object, the geometric model of the target object can be loaded, the initial frame captured by a monocular camera can be input, the target object can be located using a deep learning network to obtain candidate boxes containing the target object, and then the pose of the target object in the current frame can be roughly estimated by brute-force template matching based on the image data in the candidate boxes.

[0022] It is understandable that the execution of this method can be achieved using a 3D object detection device, such as... Figure 2 As shown, the 3D object detection device includes a monocular camera, a computing device, and a memory. The monocular camera is mainly used to acquire 2D image data containing the target object, the computing device is used to detect the initial pose of the target object in the 2D image and track it in real time in subsequent image sequences, and the memory is used to store the 3D model of the target object and provide memory for the calculation process. It should be noted that using a monocular camera is less costly than existing 3D object detection methods that use depth cameras or binocular cameras.

[0023] Understandably, when taking multi-view images of a target object, if the target object has rich texture, it is taken directly; if the target object has weak texture or no texture, a layer of rich texture is tightly wrapped around the surface of the target object before taking the picture.

[0024] like Figure 3 As shown, step 101 may include, but is not limited to, the following sub-steps: Step 1011: Acquire multi-view images of the target object using a monocular camera; Step 1012: Preprocess the multi-view images of the target object using an image enhancement algorithm; Step 1013: Perform sparse reconstruction on the target object to generate a sparse point cloud and depth map of the target object. Step 1014: Fuse the sparse point cloud and the depth map to achieve dense point cloud reconstruction; Step 1015: Reconstruct the three-dimensional model of the target object using the radial basis implicit surface reconstruction algorithm; Step 1016: Extract geometric model information from the 3D model.

[0025] Understandably, using a monocular camera to capture multi-view images of a target object is less costly compared to existing 3D object detection methods that use depth cameras or binocular cameras. If the target object has rich texture, it is captured directly; if the target object has weak texture or no texture, a richly textured pattern is tightly wrapped around the object's surface before capturing the image. Image enhancement algorithms are used to preprocess the acquired multi-view images of the target object; then, SFM (Structure from Motion) software is used to perform sparse reconstruction of the target object, generating a sparse point cloud and depth map; then, MVS (Machine Vision Studio) software is used to fuse the sparse point cloud and depth map to achieve dense point cloud reconstruction; finally, the 3D model of the target object is reconstructed using a radial basis function implicit surface reconstruction algorithm.

[0026] Following step 1016, the following sub-steps may be included, but are not limited to: Noise is removed using a mesh smoothing algorithm, and the vertices of the target object's geometric model are reduced using a polygon reduction algorithm.

[0027] It is understandable that extracting geometric information from a 3D model, using a mesh smoothing algorithm to remove noise, and using a polygon reduction algorithm to reduce the vertices of the geometric model can reduce the computational cost in subsequent target object pose estimation and tracking.

[0028] like Figure 4 As shown, step 102 may include, but is not limited to, the following sub-steps: Step 1021: Load the geometric model information of the target object; Step 1022: Obtain the initial frame of the target object using a monocular camera; Step 1023: Input the initial frame into the deep learning network to perform preliminary localization of the target object and obtain the candidate box of the target object in the image to obtain the image data within the candidate box; Step 1024: Perform brute-force template matching on the image data within the candidate bounding box to estimate the initial pose of the target object.

[0029] Understandably, this involves loading the geometric model of the target object and using a monocular camera to capture a single frame of a 2D image containing the target object. The subscript 0 indicates that the sequence number of the current frame is 0, which is the initial frame. The obtained two-dimensional image data... The data is input into a deep learning network to perform initial localization of the target object, obtaining candidate bounding boxes of the target object in the image. The image data within each candidate bounding box is denoted as... '. Image data within the obtained candidate boxes. 'A brute-force template matching process is performed to roughly estimate the target object's pose. Since this step uses image data within the candidate bounding boxes for brute-force template matching, it avoids the brute-force template matching required by segmenting and traversing the entire image, thus reducing the number of matching pairs. Furthermore, because the distance between the target object's geometric model and the monocular camera can be roughly estimated based on the candidate bounding box size, it avoids setting templates at different distances during the brute-force matching process, thereby reducing the number of templates to be matched.'

[0030] Using deep learning methods to obtain the bounding box of the target object significantly reduces the number of brute-force matches compared to existing template matching methods, resulting in faster initial pose estimation. During tracking, a motion model is used to correct and optimize the target object's pose, leading to more stable object tracking.

[0031] like Figure 5 As shown, step 103 may include, but is not limited to, the following sub-steps: Step 1031: Based on the initial pose or the pose of the target object in the previous frame, calculate the local color histograms corresponding to the background and foreground of the two-dimensional projection contour of the 3D model, respectively. The area outside the two-dimensional projection contour is the background, and the area inside the two-dimensional projection contour is the foreground. Step 1032: Estimate the pose of the target object in the current frame using the energy function defined by the local color histogram.

[0032] It is understandable that a monocular camera is used to acquire two-dimensional image data containing the target object. Here, t represents the sequence number of the current frame. Based on the initial pose of the target object or the pose of the previous frame, the 2D projection contour of the obtained 3D model is calculated, with the area outside the 2D projection contour as the background and the area inside as the foreground. Taking the 2D points mapped from the 3D vertices of the 3D model onto the projection contour as the centers, local color histograms of the foreground and background with radius r are calculated respectively. Since a simplified geometric model is used here, the number of vertices on the model is reduced, and the number of local color histograms that need to be calculated is also reduced, thereby reducing computational and storage costs. The energy function defined by the local color histogram is used to optimize the target object in the current frame. The lowered position.

[0033] Step 104 may include, but is not limited to, the following sub-steps: When the energy function value is less than or equal to the first threshold, it is determined that the target object has been successfully located, the motion tracking model is updated, and the process moves to the next frame pose estimation of the target object. If the energy function value is greater than the first threshold, it is determined that the tracking of the target object has been lost.

[0034] Understandably, the energy function value, calculated based on the energy function defined by the local color histogram, can be used to determine whether the target object has been tracked. If the energy function value is less than or equal to a first threshold, the target object is considered successfully localized, the motion tracking model is updated, and the process moves to the next frame's pose estimation for the target object. If the energy function value is greater than the first threshold, the target object is considered lost.

[0035] Step 104 may also include the following sub-steps: If the energy function value is greater than the first threshold, the energy function value of the pose is determined by estimating the pose of the target object based on the motion tracking model. If the energy function value is less than or equal to the second threshold, the target object is determined to be successfully located, and the process proceeds to the next frame pose estimation of the target object. If the energy function value is greater than the second threshold, it is determined that the tracking of the target object has been lost. The first threshold is less than the second threshold.

[0036] Understandably, when the energy function value is greater than the first threshold, the pose of the target object can be estimated based on the motion tracking model to determine the energy function value of that pose. When the energy function value is less than or equal to the second threshold, it is determined that the target object has been successfully located, and the process proceeds to the next frame pose estimation for the target object; when the energy function value is greater than the second threshold, it is determined that the tracking of the target object has been lost; wherein, the first threshold is less than the second threshold.

[0037] Based on this, this invention uses a monocular camera for target object detection. It acquires geometric model information of the target object by collecting multi-view images, then uses a brute-force matching method to estimate the target object's pose in the initial frame based on this geometric model information. Finally, it uses a region color histogram-based method for continuous tracking and pose estimation, and utilizes a motion tracking model to correct and optimize the pose estimation results. Therefore, this invention mainly consists of three steps: constructing a geometric model, initial target object pose estimation, and object tracking and pose estimation. Since this invention uses only a monocular camera, its implementation cost is lower compared to existing 3D object detection methods using depth cameras and binocular cameras. This invention automatically extracts the geometric information of the target object and then performs object detection and tracking based on this extracted information. It does not require third-party modeling software to create a 3D model of the target object, thus offering the advantage of ease of use. This invention uses deep learning methods to obtain the target object's bounding box, which significantly reduces the number of brute-force matches compared to existing template matching methods, resulting in faster initial pose estimation. During tracking, a motion model is used to correct and optimize the target object's pose, making object tracking more stable, with the advantages of fast operation and more stable tracking. In addition, after 3D modeling, the number of vertices in the geometric model is reduced by mesh smoothing algorithm and polygon reduction algorithm, thereby reducing the computational cost in pose estimation.

[0038] The three-dimensional object detection method provided by the present invention will be further described below with reference to specific embodiments.

[0039] This embodiment uses the following three-dimensional object detection device, such as... Figure 2 As shown, the 3D object detection device includes a monocular camera, a computing device, and a memory. The monocular camera is mainly used to acquire 2D image data containing the target object, the computing device is used to detect the initial pose of the target object in the 2D image and to track it in real time in subsequent image sequences, and the memory is used to store the 3D model of the target object and provide memory for the calculation process.

[0040] The specific implementation method of this invention is as follows: (1) Constructing geometric model information: S101. Use the monocular camera in this device to photograph the target object. If the target object has rich texture, photograph it directly; if the target object has weak texture or no texture, tightly wrap a layer of rich texture pattern around the surface of the object before photographing. Place the target object on a horizontal surface, with the entire target object occupying 70% of the image during photographing. Set the shooting angle to 22.5° intervals in the horizontal direction and 22.5° intervals in the vertical direction, photographing directly at the target object. When the vertical angle is 90°, only one image is taken, for a total of 49 shooting positions.

[0041] S102. First, use image enhancement algorithms to preprocess the multi-view images of the target object acquired in step S101; then use SFM software (such as openMVG) to perform sparse reconstruction of the target object, generating sparse point cloud and depth map of the target object; then use MVS software (such as openMVS) to fuse the sparse point cloud and depth map to achieve dense point cloud reconstruction; finally, reconstruct the three-dimensional model of the target object through radial basis implicit surface.

[0042] S103. Extract the geometric information from the 3D model in S102, remove noise using a mesh smoothing algorithm (such as Laplacian smoothing algorithm), and reduce the vertices of the geometric model using a polygon reduction algorithm to reduce the computational and storage costs in subsequent target object pose estimation and tracking.

[0043] (2) Initial pose estimation of the target object: S201. Load the geometric model of the target object constructed in the first part.

[0044] S202. Use the monocular camera in this device to acquire a one-frame two-dimensional image containing the target object. The data is input to the computer section of this device, where the subscript 0 indicates that the sequence number of the current frame is 0, which is the initial frame.

[0045] S203, The two-dimensional image data obtained in S202 The data is input into a deep learning network to perform initial localization of the target object, obtaining candidate bounding boxes of the target object in the image. The image data within each candidate bounding box is denoted as... '.

[0046] S204. The image data within the candidate boxes obtained in step S203. Perform brute-force template matching. Assume the 3D model is placed inside the center of an icosahedron. Using the 12 vertices of the icosahedron as points in a plane, construct 12 viewpoints perpendicular to the vertices and the center of the icosahedron. Each viewpoint is rotated by 4 angles (0, 90, 180, 270) in the plane, resulting in 48 basic templates. Calculate the color histogram and corresponding energy function of the target object under these 48 basic templates. Roughly estimate the target object's pose based on the magnitude of the energy function.

[0047] (3) Object tracking and pose estimation: S301. Use the monocular camera in this device to acquire two-dimensional image data containing the target object. , which is input to the computer part of this device, where t represents the current frame sequence number.

[0048] S302. Based on the target object pose calculated in the previous frame or S204, calculate the two-dimensional projection contour of the simplified 3D model obtained in S103. Let the area outside the projection contour be the background and the area inside be the foreground. Taking the two-dimensional point mapped from the 3D vertices on the model onto the projection contour as the center, calculate the local color histograms of the foreground and background with a radius of r.

[0049] S303. Optimize the target object in the current frame using an energy function defined by a local color histogram. The pose is determined. If the energy function exceeds the threshold h1, it is determined to be lost, and the process proceeds to step S304; otherwise, the localization is successful, the motion tracking model is updated, and the process proceeds to step S301 to perform pose estimation for the next frame.

[0050] S304. Estimate the pose based on the motion tracking model and calculate the energy function under that pose. If the energy function exceeds the threshold h2, it is determined that the target tracking is lost or the target object is not in the field of view, and proceed to S203 for re-localization; otherwise, localization is successful, and proceed to S301 for pose estimation of the next frame.

[0051] Based on this, before constructing the geometric model of the target object, if the target object is a weakly textured or textureless object, a textured pattern needs to be applied to its surface before multi-view image acquisition. After conventional 3D object reconstruction, a mesh smoothing algorithm and a polygon reduction algorithm are used to reduce the vertices of the geometric model while preserving the overall model information, thereby reducing the computational and storage costs in the object pose estimation and tracking process. In the initial pose estimation of the target object, a deep learning network is used to perform preliminary localization of the target object, and brute-force template matching is performed on the image data in the candidate boxes, thus avoiding traversing and matching the entire image and reducing the computational cost of initial pose estimation. A motion model is used during object tracking to enhance the stability of pose estimation. In the detection process, an initial pose estimation process is added to avoid using third-party software to provide the initial pose.

[0052] like Figure 6 As shown in the figure, this embodiment of the invention also provides a three-dimensional object detection device.

[0053] Specifically, the three-dimensional object detection device includes: one or more processors and a memory. Figure 6 Let's take a processor and memory as an example. The processor and memory can be connected via a bus or other means. Figure 6 Taking the example of a connection between China and Israel via a bus.

[0054] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs, such as the three-dimensional object detection method in the above embodiments of the present invention. The processor implements the three-dimensional object detection method in the above embodiments of the present invention by running the non-transitory software program and the program stored in the memory.

[0055] The memory may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data required to execute the three-dimensional object detection method described in the embodiments of the present invention. Furthermore, the memory may include high-speed random access memory and non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the three-dimensional object detection device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0056] The non-transient software program required to implement the three-dimensional object detection method in the above embodiments of the present invention, and the program stored in memory, are executed by one or more processors to perform the three-dimensional object detection method described above, for example, the method described above. Figure 1 Steps 101 to 104 in the method are as follows. Figure 3 Steps 1011 to 1016 in the method are as follows. Figure 4 Steps 1021 to 1024 in the method are as follows. Figure 5Steps 1031 to 1032 of the method involve acquiring multi-view images of the target object to obtain its geometric model information, then using a brute-force matching method to estimate the target object's pose in the initial frame based on this geometric model information. Finally, an energy function defined based on the region color histogram is used to continuously track and estimate the target object's pose, and a motion tracking model is used to correct and optimize the pose estimation results. Specifically, the geometric information of the target object is extracted, and then object detection and tracking are performed based on the extracted geometric information. During the detection process, an initial pose estimation process is added to avoid using third-party software to provide the initial pose. During the tracking process, a motion tracking model is used to correct and optimize the target object's pose, making object tracking more stable. Therefore, this invention has the advantages of being easy to use, running quickly, and providing more stable tracking.

[0057] like Figure 7 As shown, an embodiment of the present invention also provides an electronic device.

[0058] Specifically, the electronic device includes: one or more processors and memory. Figure 7 Let's take a processor and memory as an example. The processor and memory can be connected via a bus or other means. Figure 7 Taking the example of a connection between China and Israel via a bus.

[0059] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs, such as the three-dimensional object detection method in the above embodiments of the present invention. The processor implements the three-dimensional object detection method in the above embodiments of the present invention by running the non-transitory software program and the program stored in the memory.

[0060] The memory may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data required to execute the three-dimensional object detection method described in the embodiments of the present invention. Furthermore, the memory may include high-speed random access memory and non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the three-dimensional object detection device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0061] The non-transient software program required to implement the three-dimensional object detection method in the above embodiments of the present invention, and the program stored in memory, are executed by one or more processors to perform the three-dimensional object detection method described above, for example, the method described above. Figure 1Steps 101 to 104 in the method are as follows. Figure 3 Steps 1011 to 1016 in the method are as follows. Figure 4 Steps 1021 to 1024 in the method are as follows. Figure 5 Steps 1031 to 1032 of the method involve acquiring multi-view images of the target object to obtain its geometric model information, then using a brute-force matching method to estimate the target object's pose in the initial frame based on this geometric model information. Finally, an energy function defined based on the region color histogram is used to continuously track and estimate the target object's pose, and a motion tracking model is used to correct and optimize the pose estimation results. Specifically, the geometric information of the target object is extracted, and then object detection and tracking are performed based on the extracted geometric information. During the detection process, an initial pose estimation process is added to avoid using third-party software to provide the initial pose. During the tracking process, a motion tracking model is used to correct and optimize the target object's pose, making object tracking more stable. Therefore, this invention has the advantages of being easy to use, running quickly, and providing more stable tracking.

[0062] Furthermore, embodiments of the present invention also provide a computer-readable storage medium storing a computer-executable program, which is executed by one or more control processors, for example, by... Figure 6 One or more processors can be executed, causing them to perform the three-dimensional object detection method described in the above embodiments of the present invention, for example, to perform the above-described... Figure 1 Steps 101 to 104 in the method are as follows. Figure 3 Steps 1011 to 1016 in the method are as follows. Figure 4 Steps 1021 to 1024 in the method are as follows. Figure 5 Steps 1031 to 1032 of the method involve acquiring multi-view images of the target object to obtain its geometric model information, then using a brute-force matching method to estimate the target object's pose in the initial frame based on this geometric model information. Finally, an energy function defined based on the region color histogram is used to continuously track and estimate the target object's pose, and a motion tracking model is used to correct and optimize the pose estimation results. Specifically, the geometric information of the target object is extracted, and then object detection and tracking are performed based on the extracted geometric information. During the detection process, an initial pose estimation process is added to avoid using third-party software to provide the initial pose. During the tracking process, a motion tracking model is used to correct and optimize the target object's pose, making object tracking more stable. Therefore, this invention has the advantages of being easy to use, running quickly, and providing more stable tracking.

[0063] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable programs, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable programs, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0064] The above provides a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of the present invention.

Claims

1. A method for detecting three-dimensional objects, characterized in that, include: Collect multi-view images of the target object to obtain the geometric model information of the three-dimensional model of the target object; Based on the geometric model information, the estimated pose of the target object in the initial frame is calculated using a brute-force matching algorithm; The estimated pose of the target object is continuously tracked based on the energy function defined by the local color histogram. The pose estimation result of the target object is corrected using a motion tracking model, wherein the pose estimation result indicates the estimated pose of the target object in three-dimensional space; The continuous tracking of the estimated pose of the target object using the energy function defined based on the local color histogram includes: Based on the initial pose or the pose of the previous frame of the target object, calculate the local color histograms corresponding to the background and foreground of the two-dimensional projection contour of the three-dimensional model, respectively, wherein the area outside the two-dimensional projection contour is the background and the area inside the two-dimensional projection contour is the foreground. The pose of the target object in the current frame is estimated using the energy function defined by the local color histogram; The step of correcting the pose estimation result of the target object using a motion tracking model includes: When the energy function value is less than or equal to the first threshold, it is determined that the target object has been successfully located, the motion tracking model is updated, and the process moves to the next frame pose estimation of the target object. If the energy function value is greater than the first threshold, it is determined that the tracking of the target object has been lost. The method of correcting the pose estimation result of the target object using a motion tracking model further includes: If the energy function value is greater than the first threshold, the energy function value of the pose is determined by estimating the pose of the target object based on the motion tracking model. When the energy function value is less than or equal to the second threshold, it is determined that the target object has been successfully located, and the process proceeds to the next frame pose estimation of the target object. If the energy function value is greater than the second threshold, it is determined that the tracking of the target object has been lost. Wherein, the first threshold is less than the second threshold.

2. The method according to claim 1, characterized in that, The acquisition of multi-view images of the target object to obtain geometric model information of the target object's 3D model includes: Multi-view images of the target object are acquired using a monocular camera; The multi-view images of the target object are preprocessed using image enhancement algorithms; The target object is sparsely reconstructed to generate a sparse point cloud and a depth map of the target object. The sparse point cloud and the depth map are fused to achieve dense point cloud reconstruction; The three-dimensional model of the target object is reconstructed using a radial basis implicit surface reconstruction algorithm; Extract the geometric model information from the three-dimensional model.

3. The method according to claim 2, characterized in that, After extracting the geometric model information from the three-dimensional model, the method further includes: Noise is removed using a mesh smoothing algorithm, and the vertices of the geometric model of the target object are reduced using a polygon reduction algorithm.

4. The method according to claim 2, characterized in that, The step of calculating the estimated pose of the target object in the initial frame using a brute-force matching algorithm based on the geometric model information includes: Load the geometric model information of the target object; The initial frame of the target object is acquired using a monocular camera; The initial frame is input into a deep learning network to perform preliminary localization of the target object, obtain candidate boxes of the target object in the image, and obtain image data within the candidate boxes; Brute-force template matching is performed on the image data within the candidate bounding box to estimate the initial pose of the target object.

5. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the computer program, it implements the three-dimensional object detection method as described in any one of claims 1 to 4.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer-executable program for causing a computer to perform the three-dimensional object detection method as described in any one of claims 1 to 4.

Images