Object Determination Method, Device, Equipment, Storage Medium, and Program Product

Abstract

The present disclosure provides an object determination method, apparatus, device, storage medium, and program product, which relate to the technical field of image processing, and particularly to the technical field of computer vision. The specific implementation solution is as follows: obtaining an overall object image and a plurality of partial object images, wherein any one of the partial object images overlaps with at least one other partial object image among the plurality of partial object images; performing object detection on the overall object image to obtain target object data; performing matching on the plurality of partial object images to obtain matching data; performing three-dimensional reconstruction on the partial object images according to the matching data to obtain a three-dimensional target object image; and determining the size data of the target object according to the mapping of the target object data relative to the three-dimensional target object image.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technologies, particularly to the field of computer vision technologies, and specifically to an object determination method, apparatus, device, storage medium, and program product. Background Art

[0002] With the development of computer technologies, all aspects of production and life are gradually moving towards automation. How to improve the operation quality in harsh environments is an urgent problem to be solved. Summary of the Invention

[0003] The present disclosure provides an object determination method, apparatus, device, storage medium, and program product.

[0004] According to one aspect of the present disclosure, there is provided an object determination method, including: obtaining an overall object image and a plurality of local object images, wherein any one of the local object images overlaps with at least one other local object image among the plurality of local object images; performing target detection on the overall object image to obtain target object data; performing matching on the plurality of local object images to obtain matching data; performing three-dimensional reconstruction on the local object images according to the matching data to obtain a three-dimensional target object image; and determining the size data of the target object according to the mapping of the target object data relative to the three-dimensional target object image.

[0005] According to another aspect of the present disclosure, there is provided an object determination apparatus, including: an image acquisition module, a target object data determination module, a matching module, a three-dimensional reconstruction module, and a size data determination module. The image acquisition module is configured to obtain an overall object image and a plurality of local object images, wherein any one of the local object images overlaps with at least one other local object image among the plurality of local object images; the target object data determination module is configured to perform target detection on the overall object image to obtain target object data; the matching module is configured to perform matching on the plurality of local object images to obtain matching data; the three-dimensional reconstruction module is configured to perform three-dimensional reconstruction on the local object images according to the matching data to obtain a three-dimensional target object image; the size data determination module is configured to determine the size data of the target object according to the mapping of the target object data relative to the three-dimensional target object image.

[0006] According to another aspect of the present disclosure, there is provided an electronic device, including: at least one processor and a memory communicatively connected to the at least one processor. Wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method of the embodiments of the present disclosure.

[0007] According to another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the method of the embodiments of the present disclosure.

[0008] According to another aspect of the present disclosure, there is provided a computer program product including a computer program which, when executed by a processor, implements the method of the embodiments of the present disclosure.

[0009] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. Description of the Drawings

[0010] The drawings are used to better understand the solution and do not constitute a limitation to the present disclosure. Among them:

[0011] Figure 1 Schematically shows a system architecture diagram of an object determination method and apparatus according to an embodiment of the present disclosure;

[0012] Figure 2 Schematically shows a flowchart of an object determination method according to an embodiment of the present disclosure;

[0013] Figure 3 Schematically shows a schematic diagram of an object determination method according to an embodiment of the present disclosure;

[0014] Figure 4 Schematically shows a schematic diagram of obtaining matching data according to an embodiment of the present disclosure;

[0015] Figure 5 Schematically shows a schematic diagram of obtaining a three-dimensional target object image according to an embodiment of the present disclosure;

[0016] Figure 6 Schematically shows a schematic diagram of an object determination method according to another embodiment of the present disclosure;

[0017] Figure 7 Schematically shows a block diagram of an object determination apparatus according to an embodiment of the present disclosure; and

[0018] Figure 8 Schematically shows a block diagram of an electronic device that can implement the object determination method of the embodiments of the present disclosure. Detailed Embodiments

[0019] The following describes exemplary embodiments of the present disclosure with reference to the accompanying drawings. Various details of the embodiments of the present disclosure are included to facilitate understanding, and they should be considered merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, descriptions of well-known functions and structures are omitted below for clarity and conciseness.

[0020] The terms used herein are merely for describing specific embodiments and are not intended to limit the present disclosure. The terms "including", "comprising", etc. used herein indicate the presence of the described features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0021] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those of ordinary skill in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0022] In cases where expressions similar to "at least one of A, B, and C, etc." are used, generally, it should be interpreted according to the meaning commonly understood by those of ordinary skill in the art (for example, "a system having at least one of A, B, and C" should include, but not be limited to, a system having only A, only B, only C, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.).

[0023] Currently, with the development of computer technology, all aspects of production and life are gradually becoming automated. However, operations in some harsh environments still need to be performed manually, which have defects such as high labor costs and low operation efficiency.

[0024] For example, the construction process of tunnel surrounding rock involves procedures such as drilling, charging, blasting, cleaning, and foundation laying. Among them, how to drill and how to determine the charge amount depend on the surrounding rock grade, and the surrounding rock grade is related to factors such as the size of surrounding rock fissures. In the harsh environment of the surrounding rock construction site, how to efficiently determine the surrounding rock grade has become an urgent problem to be solved.

[0025] In some embodiments, the surrounding rock grade is obtained by manually visually measuring the size of surrounding rock fissures and calculating. It has defects such as high labor costs and strong subjectivity.

[0026] In some embodiments, a crane is used to carry people to measure at a high place of the surrounding rock to determine the size of surrounding rock fissures. Although the accuracy can be improved, there are safety hazards and it will also affect the construction progress.

[0027] In some embodiments, by placing a length calibrator on the surrounding rock and taking a picture including the length calibrator, estimating the ratio of the number of pixels in the picture to the length of the length calibrator, obtaining a scale factor, and determining the fracture length according to the scale factor. This method for determining the fracture length needs to ensure that the plane of the surrounding rock is parallel to the plane of the camera. In the actual measurement process, large measurement errors may be caused by factors such as the viewing angle and the standing position.

[0028] The following will take the determination of the fracture length of the surrounding rock as an example to illustrate the object determination method of the embodiments of the present disclosure. It should be noted that the object determination method of the embodiments of the present disclosure can also be applied to determining the object size in other scenarios, which is not specifically limited herein.

[0029] Figure 1 Schematically shows the system architecture of the object determination method and device according to an embodiment of the present disclosure. It should be noted that Figure 1 The shown is only an example of the system architecture to which the embodiments of the present disclosure can be applied, to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0030] As Figure 1 shown, the system architecture 100 according to this embodiment may include an imaging device 101, a client 102, a network 103, and a server 104. The network 103 is used to provide a medium for communication links between the imaging device 101, the client 102, and the server 104. The network 103 may include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0031] The user can use the imaging device 101 to take an image and transmit the image to the client 102. The client 102 can interact with the server 104 through the network 103 to receive or send messages, etc. Various communication client applications can be installed on the client 102, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0032] The imaging device 101 may be a device such as a camera.

[0033] The client 102 may be various electronic devices with a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, and desktop computers, etc. The client 102 of the embodiments of the present disclosure can, for example, run application programs.

[0034] Server 104 may be a server that provides various services, such as a background management server (for example only) that supports the websites browsed by the user using the client 102. The background management server may analyze and process data such as user requests received, and feedback the processing results (such as web pages, information, or data obtained or generated according to user requests) to the client. Additionally, server 104 may also be a cloud server, that is, server 104 has cloud computing capabilities.

[0035] It should be noted that the object determination method provided by the embodiments of the present disclosure may be executed by server 104. Correspondingly, the object determination device provided by the embodiments of the present disclosure may be provided in server 104. The object determination method provided by the embodiments of the present disclosure may also be executed by a server or a server cluster different from server 104 and capable of communicating with client 102 and / or server 104. Correspondingly, the object determination device provided by the embodiments of the present disclosure may also be provided in a server or a server cluster different from server 104 and capable of communicating with client 102 and / or server 104.

[0036] In one example, the user may use the imaging device 101 to obtain an overall object image and a plurality of local object images, and transmit the overall object image and the plurality of local object images to the client 102 through the network 103. Server 104 may obtain the overall object image and the plurality of local object images from the client 102 through the network 103.

[0037] It should be understood that Figure 1 the numbers of the imaging device, the client, the network, and the server in

[0038] are merely illustrative. According to the implementation requirements, there may be any number of imaging devices, clients, networks, and servers.

[0039] It should be noted that in the technical solution of the present disclosure, the processing of collection, storage, use, processing, transmission, provision, and disclosure of the user's personal information involved all comply with the provisions of relevant laws and regulations and do not violate public order and good customs.

[0040] The embodiments of the present disclosure provide an object determination method. The following combines Figure 1 with the system architecture of Figures 2 to 6 to describe the object determination method according to the exemplary embodiments of the present disclosure. The object determination method of the embodiments of the present disclosure may be executed, for example, by Figure 1 the server 104 shown.

[0041] Figure 2The flowchart of the object determination method according to an embodiment of the present disclosure is schematically shown.

[0042] As Figure 2 shown, the object determination method 200 of the embodiment of the present disclosure may include, for example, operation S210 to operation S250.

[0043] In operation S210, an overall object image and a plurality of local object images are acquired.

[0044] Hereinafter, the object determination method of the embodiment of the present disclosure will still be described by taking the example of determining the size data of the target object of surrounding rock fractures. The overall object image may be, for example, an image including the overall surrounding rock, and the local object image may be, for example, an image including a local part of the surrounding rock.

[0045] Exemplarily, after obtaining the overall object image by using an imaging device, the overall object image may be divided into a plurality of regions, and each region may be photographed by using the imaging device to obtain a plurality of local object images. Any one local object image overlaps with at least one other local object image among the plurality of local object images.

[0046] Both the overall object image and the plurality of local object images are two-dimensional planar images. The local object image is clearer than the overall exclusive image and can also display more details. The local object image and the overall object image may correspond to different perspectives.

[0047] Exemplarily, three overall object images may be acquired, and the three overall object images may respectively correspond to the perspectives of 45 degrees to the left, 45 degrees to the right, and 90 degrees forward of the target object.

[0048] In operation S220, target detection is performed on the overall object image to obtain target object data.

[0049] The target object data may be understood as relevant data for indirectly calculating the size data of the target object determined according to the two-dimensional overall object image.

[0050] In operation S230, matching is performed on the plurality of local object images to obtain matching data.

[0051] It can be understood that the plurality of local object images are independent of each other, and any one local object image overlaps with at least one other local object image among the plurality of local object images. The overlapping regions may be used as the basis for matching, and matching is performed on the plurality of local object images to obtain matching data.

[0052] In operation S240, three-dimensional reconstruction is performed on the local object image according to the matching data to obtain a three-dimensional target object image.

[0053] 3D reconstruction can be understood as the process of reconstructing 3D information based on single-view or multi-view images. By performing 3D reconstruction on the local object images, the obtained 3D target object image can represent the overall 3D data information of the target object.

[0054] In operation S250, according to the mapping of the target object data relative to the 3D target object image, the size data of the target object is determined.

[0055] It can be understood that the overall object image is the 2D image of the target object, and the 3D target object image is the 3D image of the target object.

[0056] In the object determination method of the embodiments of the present disclosure, by detecting the overall object image, the obtained target object data is used to indirectly determine the size data of the target object. By matching the local object images with overlapping regions, the obtained matching data can be used for accurate 3D reconstruction. The 3D target object image obtained through 3D reconstruction can also reflect the depth of the target object. Therefore, the size data of the target object can be accurately determined through the mapping of the 2D target object data relative to the 3D target object image.

[0057] The object determination method of the embodiments of the present disclosure can automatically determine the object size data and is applicable to application scenarios such as high-precision measurement and harsh working environments. For example, in the application scenario of determining the length of surrounding rock cracks, it has a higher object determination efficiency.

[0058] It should be noted that operations S210 to S250 only represent different operations and do not represent the order of execution. For example, after performing operation S210, the execution order of operation S220 and operation S230 can be swapped, or operation S230 and operation S240 can be executed before operation S220.

[0059] Exemplarily, the overall object image and multiple local object images can also be matched to improve the matching accuracy.

[0060] Figure 3 Schematically shows a schematic diagram of the object determination method 300 according to the embodiments of the present disclosure.

[0061] As Figure 3 shown, the object determination method 300 according to the embodiments of the present disclosure includes operations S310 to S350.

[0062] In operation S310, an overall object image 301 and multiple local object images 302 are obtained. In operation S320, object detection is performed on the overall object image 301 to obtain object data 303. In operation S330, matching is performed on the multiple local object images 302 to obtain matching data 304. In operation S340, based on the matching data 304, three-dimensional reconstruction is performed on the local object images 302 to obtain a three-dimensional object image 305. In operation S350, based on the mapping of the object data 303 relative to the three-dimensional object image 305, the dimension data 306 of the object is determined.

[0063] Figure 4 Schematically shows a schematic diagram of obtaining matching data according to an embodiment of the present disclosure.

[0064] As Figure 4 shown, according to an embodiment of the present disclosure, the operation of performing matching on multiple local object images in operation S430 to obtain matching data may include: operations S431 to S432.

[0065] In operation S431, feature detection is performed on the local object image 401 to obtain feature points 402.

[0066] Feature points can be understood as points where the image gray value changes drastically or points with a large curvature at the image edge. Feature points can reflect the essential features of the image and can also identify the target object in the image.

[0067] Exemplarily, a feature descriptor can be used to characterize the feature points. A feature descriptor can be understood as a feature point represented by a vector, and this vector can describe the information of the pixels around the feature point in a manner specified by humans.

[0068] Exemplarily, the Scale-Invariant Feature Transform (SIFT) algorithm can be used to perform feature detection on the local object image to obtain SIFT feature descriptors. The feature descriptors obtained by the SIFT algorithm have the advantages of high stability, geometric measurement invariance, and high tolerance to light, noise, and micro-viewpoint changes, which can improve the accuracy of feature detection.

[0069] In operation S432, based on the feature points 402, matching is performed on the multiple local object images 401 to obtain initial matching data, and the initial matching data may include initial image matching pairs 403 and the overlapping regions 404 between the images of the initial image matching pairs.

[0070] Since a feature descriptor is a vector, the matching degree between two corresponding feature points can be reflected by the numerical size of the distance between the two feature descriptors.

[0071] "Performing matching on multiple local object images 401" in operation S432 can be understood as: By the above-mentioned feature detection method, at least one feature point of each local object image is detected, and at least one feature point of each local object image is used as a feature point set. Feature point matching is performed on the feature point sets of any two local object images respectively.

[0072] An initial image matching pair can be understood as two local object images that have been successfully matched after matching. The overlapping region between the images of the initial image matching pair can be understood as the region where the successfully matched feature descriptors of the initial image matching pair are located.

[0073] Exemplarily, matching can be performed on multiple local object images 401 according to one of the following matching strategies: brute-force matching method, cross-matching, KNN matching, and RANSAC matching (RANSAC stands for Random Sample Consensus, random sample consensus).

[0074] Exemplarily, matching can be performed on multiple local object images according to the RANSAC matching strategy to obtain accurate and highly robust initial matching data.

[0075] The object determination method of the embodiments of the present disclosure can automatically associate local object images with matching relationships through feature detection and performing matching on multiple local object images, facilitating subsequent operations such as accurate 3D reconstruction.

[0076] As Figure 4 shown, performing matching on multiple local object images in operation S430 to obtain matching data can also include: operation S433.

[0077] In operation S433, geometric verification is performed on the initial image matching pair 403 according to the overlapping region 404 and feature points 402 of the initial image matching pair to obtain target matching data.

[0078] The target matching data can include target image matching pairs 405.

[0079] Exemplarily, it can be done by: determining the essential matrix of the camera according to the initial image matching pair - through the way of the trifocal tensor including two camera foci and a spatial target point, performing overall geometric verification on the initial image matching pair. Among them, the essential matrix can reflect the relationship of a spatial point in the camera coordinate systems of different perspectives.

[0080] Exemplarily, a ratio R can be defined: R = n1 / n2, where, for a certain initial image matching pair C, n1 represents the number of feature points that are matched after geometric verification for the initial image matching pair C, and n2 represents the number of feature points that are matched before geometric verification for the initial image matching pair C. When R is greater than a certain threshold Th, it can be determined that the initial image matching pair C passes the geometric verification, and the initial image matching pair C is determined to be a target image matching pair.

[0081] The initial matching data is obtained by performing matching on the local object image based on feature points, and the initial matching data is only related to the visual feature points. The local object image is a two-dimensional planar image, and it is easy to have matching errors when performing matching only based on visual feature points.

[0082] The object determination method of the embodiments of the present disclosure can verify the matching relationship of the initial image matching pair by performing geometric verification on the initial image matching pair, thereby improving the matching accuracy.

[0083] Figure 5 Schematically shows a schematic diagram of obtaining a three-dimensional target object image according to an embodiment of the present disclosure.

[0084] As Figure 5 shown, performing three-dimensional reconstruction on the local object image according to the matching data in operation S540 to obtain the three-dimensional target object image may include: operations S541 to S544.

[0085] In operation S541, according to the target image matching pair 501, determine the reconstructed initial image matching pair 502.

[0086] There may be multiple target image matching pairs, and one target image matching pair can be determined from the multiple target image matching pairs as the reconstructed initial image matching pair.

[0087] In operation S542, perform image registration on the reconstructed initial image matching pair 502, the local object image 503, and the overall object image 504 to obtain the to-be-registered image 505 and the registered image 506.

[0088] Image registration can be understood as a process of matching and superimposing two or more images acquired at different times, by different imaging devices (imaging devices), or under different conditions.

[0089] Image registration may include: determining image-space coordinate transformation parameters through the matched feature points, and performing registration according to the image-space coordinate transformation parameters.

[0090] Exemplarily, taking the image registration of the reconstructed initial image matching pair for the local object image as an example for illustration. For example, the local object image includes I 1to I 9 The reconstructed initial image matching pair includes I 1 and I 2 All local object images except the reconstructed initial image matching pair can be registered by registering one local object image each time, for a total of 7 registrations. For example, in the first registration, the local object image I 3 is registered to the reconstructed initial image matching pair. At this time, I 1 and I 2 and I 3 are registered images, and I 4 to I9 are images to be registered. It can be understood that when the 7 registrations are completed, the final registered images are obtained, and there are no images to be registered at this time.

[0091] In operation S543, triangulation processing is performed on the image to be registered and the registered image to obtain triangulation feature points 507.

[0092] Triangulation can be understood as: the process of determining spatial points through the triangulation principle.

[0093] The triangulation principle can be understood as follows: Given the camera matrices P and P' of two images in the same world coordinate system, m and m' are a point correspondence of the two images, and m and m' satisfy the epipolar geometric constraint m′Fm = 0. Now, the spatial point corresponding to the point correspondence needs to be calculated based on P and P'. Since the back-projection ray lm of m and the back-projection ray lm' of m' determine a plane f passing through the optical centers of the two cameras, that is, an epipolar plane, the two back-projection rays lm and lm' must intersect at a spatial point. That is to say, the back-projection rays of the corresponding points and the baseline of the two cameras form a triangle, and the vertices of this triangle are the optical centers of the two cameras and the intersection point of the two back-projection rays, and this intersection point is the spatial point X we want to determine.

[0094] The triangulation processing can include: extracting matching point pairs from the image to be registered and the registered image - calculating the essential matrix based on the matching points, and then calculating the relative pose of the image to be registered and the registered image through singular value decomposition - determining the spatial 3D point coordinates corresponding to the matching point pairs according to the relative pose. The spatial 3D point coordinates here are the triangulation feature points.

[0095] In operation S544, based on the triangulation feature points 507 and the three-dimensional reconstruction model 509, three-dimensional reconstruction is performed on the registered image 506 to obtain a three-dimensional target object image 510.

[0096] Exemplarily, the three-dimensional reconstruction model can include an MVS network model.

[0097] It can be understood that the to-be-registered image and the registered image have point correspondences, and the newly registered image can expand the coverage of the three-dimensional target object image for subsequent three-dimensional reconstruction. The object determination method of the embodiments of the present disclosure can continuously expand the three-dimensional model through triangulation processing to obtain a complete three-dimensional target object image with a larger coverage range, and improve the expression ability of the three-dimensional target object image through redundancy.

[0098] It should be noted that the initial reconstruction image matching pairs may correspond to the middle region of the overall target image or the edge region of the overall target image. When the initial reconstruction image matching pairs correspond to the middle region of the overall target image, there will be relatively more local object images overlapping with the initial reconstruction image matching pairs. Therefore, the initial reconstruction image matching pairs corresponding to the middle region of the overall target image have more redundancy and can generate a three-dimensional target object image with higher robustness and accuracy. When the initial reconstruction image matching pairs correspond to the edge region of the overall target image, the three-dimensional reconstruction process is faster.

[0099] As Figure 5 shown, according to the embodiments of the present disclosure, the three-dimensional reconstruction of the registered image based on the triangulated feature points and the three-dimensional reconstruction model in operation S540 to obtain the three-dimensional target object image may include: operation S545 and operation S546.

[0100] In operation S545, bundle adjustment processing is performed on the triangulated feature points 507 to obtain target feature points 508.

[0101] In operation S546, based on the target feature points 508 and the three-dimensional reconstruction model 509, the registered image 506 is three-dimensionally reconstructed to obtain a three-dimensional target object image.

[0102] The triangulated feature points obtained by the above operations are the three-dimensional coordinates of the feature points. In fact, due to the existence of noise, there may be errors in the triangulated feature points and other parameters such as the camera pose that can be obtained by the above operations. Bundle adjustment processing is used to perform adjustment and optimization based on known observed values, so that the parameters such as the triangulated feature points obtained by the above operations are as close as possible to the true values.

[0103] The object determination method of the embodiments of the present disclosure can optimize the numerical values of parameters such as triangulated feature points through bundle adjustment processing, and an accurate three-dimensional target object image can be obtained. In the above embodiments, when the initial reconstruction image matching pairs correspond to the edge region of the overall target image, the three-dimensional reconstruction process is faster, but at the same time the image is sparser. Through bundle adjustment processing, the decrease in the accuracy of three-dimensional reconstruction caused by image sparsity can be avoided.

[0104] Figure 6 Schematically shows an object determination method 600 according to another embodiment of the present disclosure.

[0105] As Figure 6 shown, for the object determination method according to an embodiment of the present disclosure, the target detection of the overall object image in operation S620 to obtain target object data may include: operation S621.

[0106] In operation S621, a target segmentation model 602 is used to perform target detection on the overall object image 601 to obtain target object point data 603 and target object curve data 604.

[0107] Exemplarily, the target segmentation model may include: an HR-Net model (HR-Net, that is, High Resolution Net, high-resolution network).

[0108] The object determination method according to an embodiment of the present disclosure can automatically determine target object point data and target object curve data according to the target segmentation model, improving the object determination efficiency.

[0109] It should be noted that in some cases, the target object point data can also be obtained by performing feature detection on a partial object image in the above operation. That is, after obtaining feature points, at least some of the feature points can be determined as target object feature points, or alternatively, after obtaining feature points, target object curve data can be determined according to at least some of the feature points. In this case, the operation of performing target detection on the overall object image described above may not be executed.

[0110] As Figure 6 shown, for the object determination method 600 according to an embodiment of the present disclosure, the determination of the size data of the target object according to the mapping of the target object data relative to the three-dimensional target object image in operation S650 may include: operations S651 to S652.

[0111] In operation S651, according to the mapping of the target object point data 603 and the target object curve data 604 relative to the three-dimensional target object image 605, three-dimensional target object point data 606 and three-dimensional target object curve data 604 are obtained.

[0112] It can be understood that the overall object image is a two-dimensional image, and the three-dimensional target object image is a three-dimensional image corresponding to the overall object image. The two-dimensional target object point data and target object curve data detected through the overall object image can be mapped to the three-dimensional target object image, thereby obtaining three-dimensional target object point data and three-dimensional target object curve data.

[0113] In operation S652, three-dimensional coordinate calculations are performed on the three-dimensional target object point data 606 and the three-dimensional target object curve data 607 to obtain the size data 608 of the target object.

[0114] The object determination method of the embodiments of the present disclosure can avoid the influence of reasons such as visual deviation and lack of depth through three-dimensional target object point data and three-dimensional target object curve data, and obtain accurate size data of the target object.

[0115] Exemplarily, taking the determination of the length size data of a target object, such as surrounding rock fractures, as an example, the three-dimensional target object point data corresponding to the two end points of the surrounding rock fractures can be determined, and the length size data of the surrounding rock fractures can be determined through a distance calculation formula.

[0116] Figure 7 The block diagram of an object determination device according to an embodiment of the present disclosure is schematically shown.

[0117] As Figure 7 shown, the object determination device 700 of the embodiments of the present disclosure includes, for example, an image acquisition module 710, a target object data determination module 720, a matching module 730, a three-dimensional reconstruction module 740, and a size data determination module 750.

[0118] The image acquisition module 710 is configured to acquire an overall object image and a plurality of local object images. Among them, any one local object image overlaps with at least one other local object image among the plurality of local object images.

[0119] The target object data determination module 720 is configured to perform target detection on the overall object image to obtain target object data.

[0120] The matching module 730 is configured to perform matching on the plurality of local object images to obtain matching data.

[0121] The three-dimensional reconstruction module 740 is configured to perform three-dimensional reconstruction on the local object images according to the matching data to obtain a three-dimensional target object image.

[0122] The size data determination module 750 is configured to determine the size data of the target object according to the mapping of the target object data relative to the three-dimensional target object image.

[0123] According to an embodiment of the present disclosure, the matching module may include: a feature detection sub-module and a matching sub-module.

[0124] The feature detection sub-module may be configured to perform feature detection on the local object images to obtain feature points.

[0125] The matching sub-module may be configured to perform matching on the plurality of local object images according to the feature points to obtain initial matching data, and the initial matching data includes initial image matching pairs and overlapping regions between the images of the initial image matching pairs.

[0126] According to an embodiment of the present disclosure, the matching module may further include: a geometric verification sub-module.

[0127] A geometric verification sub-module can be used to geometrically verify the initial image matching pairs based on the overlapping regions and feature points of the initial image matching pairs, so as to obtain target matching data. Among them, the target matching data includes target image matching pairs.

[0128] According to an embodiment of the present disclosure, the 3D reconstruction module may include: a reconstruction initial image matching pair determination sub-module, an image registration sub-module, a triangulation processing sub-module, and a 3D reconstruction sub-module.

[0129] The reconstruction initial image matching pair determination sub-module can be used to determine the reconstruction initial image matching pairs according to the target image matching pairs.

[0130] The image registration sub-module can be used to perform image registration on the reconstruction initial image matching pairs, local object images, and overall object images to obtain the images to be registered and the registered images.

[0131] The triangulation processing sub-module can be used to perform triangulation processing on the images to be registered and the registered images to obtain triangulated feature points.

[0132] The 3D reconstruction sub-module can be used to perform 3D reconstruction on the registered images according to the triangulated feature points and the 3D reconstruction model to obtain a 3D target object image.

[0133] According to an embodiment of the present disclosure, the 3D reconstruction sub-module may include: a bundle adjustment processing unit and a 3D reconstruction unit.

[0134] The bundle adjustment processing unit can be used to perform bundle adjustment processing on the triangulated feature points to obtain target feature points.

[0135] The 3D reconstruction unit can be used to perform 3D reconstruction on the registered images according to the target feature points and the 3D reconstruction model to obtain a 3D target object image.

[0136] According to an embodiment of the present disclosure, the target object data determination module may include: a target object data determination sub-module.

[0137] The target object data determination sub-module can be used to perform target detection on the overall object image by using a target segmentation model to obtain target object point data and target object curve data.

[0138] According to an embodiment of the present disclosure, the dimension data determination module may include: a data determination sub-module and a calculation sub-module.

[0139] The data determination sub-module can be used to obtain 3D target object point data and 3D target object curve data according to the mapping of the target object point data and the target object curve data relative to the 3D target object image.

[0140] A calculation sub-module, which can be used to perform three-dimensional coordinate calculations on three-dimensional target object point data and three-dimensional target object curve data to obtain the size data of the target object.

[0141] It should be understood that the embodiments of the apparatus part of the present disclosure correspond to the same or similar embodiments of the method part of the present disclosure, and the technical problems solved and the technical effects achieved are also the same or similar. The present disclosure will not be elaborated herein.

[0142] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0143] Figure 8 A schematic block diagram of an exemplary electronic device 800 that can be used to implement the embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as, for example, a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, for example, a personal digital processor, a cellular phone, a smart phone, a wearable device, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely exemplary and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0144] As Figure 8 shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0145] A plurality of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a magnetic disk, an optical disc, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0146] The computing unit 801 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 executes the various methods and processes described above, such as the object determination method. For example, in some embodiments, the object determination method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the object determination method described above can be executed. Alternatively, in other embodiments, the computing unit 801 can be configured to execute the object determination method by any other suitable means (e.g., by means of firmware).

[0147] The various embodiments of the systems and techniques described above in this document can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGA), application-specific integrated circuits (ASIC), application-specific standard products (ASSP), system-on-a-chip systems (SOC), complex programmable logic devices (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a dedicated or general-purpose programmable processor, and can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0148] The program code for implementing the methods of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the program codes are executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program codes can be executed entirely on the machine, partially on the machine, executed partially on the machine and partially on a remote machine as an independent software package, or executed entirely on a remote machine or server.

[0149] In the context of this disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0150] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can also be used to provide for interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic, speech, or tactile input).

[0151] The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or a computing system that includes a middleware component (e.g., an application server), or a computing system that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described here), or a computing system that includes any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0152] A computer system can include clients and servers. The clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0153] It should be understood that the various forms of processes shown above can be used, with steps reordered, added or deleted. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and no limitation is imposed herein.

[0154] The above specific embodiments do not constitute a limitation on the protection scope of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub - combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the protection scope of this disclosure.

Claims

1. An object determination method, comprising: obtaining an overall object image and a plurality of local object images, wherein any one of the local object images overlaps with at least one other local object image among the plurality of local object images; performing object detection on the overall object image to obtain target object data; performing matching on the plurality of local object images to obtain matching data; performing three-dimensional reconstruction on the local object images according to the matching data to obtain a three-dimensional target object image; and determining size data of the target object according to the mapping of the target object data relative to the three-dimensional target object image; wherein the matching data includes target image matching pairs, and the performing three-dimensional reconstruction on the local object images according to the matching data to obtain a three-dimensional target object image includes: determining an initial reconstruction image matching pair according to the target image matching pairs; performing image registration on the initial reconstruction image matching pair, the local object images, and the overall object image to obtain an image to be registered and a registered image; performing triangulation processing on the image to be registered and the registered image to obtain triangulated feature points; and performing three-dimensional reconstruction on the registered image according to the triangulated feature points and a three-dimensional reconstruction model to obtain the three-dimensional target object image.

2. The method according to claim 1, wherein the performing matching on the plurality of local object images to obtain matching data includes: performing feature detection on the local object images to obtain feature points; and performing matching on the plurality of local object images according to the feature points to obtain initial matching data, the initial matching data including initial image matching pairs and overlapping regions between the images of the initial image matching pairs.

3. The method according to claim 2, wherein the performing matching on the plurality of local object images to obtain matching data further includes: performing geometric verification on the initial image matching pairs according to the overlapping regions of the initial image matching pairs and the feature points to obtain target matching data, wherein the target matching data includes target image matching pairs.

4. The method according to claim 1, wherein the performing three-dimensional reconstruction on the registered image according to the triangulated feature points and a three-dimensional reconstruction model to obtain the three-dimensional target object image includes: performing bundle adjustment processing on the triangulated feature points to obtain target feature points; and performing three-dimensional reconstruction on the registered image according to the target feature points and the three-dimensional reconstruction model to obtain the three-dimensional target object image.

5. The method according to any one of claims 1 to 4, wherein the performing object detection on the overall object image to obtain target object data includes: using a target segmentation model to perform object detection on the overall object image to obtain target object point data and target object curve data.

6. The method according to claim 5, wherein the determining size data of the target object according to the mapping of the target object data relative to the three-dimensional target object image includes: Obtain three-dimensional target object point data and three-dimensional target object curve data based on the mapping of the target object point data and the target object curve data relative to the three-dimensional target object image; and Perform three-dimensional coordinate calculation on the three-dimensional target object point data and the three-dimensional target object curve data to obtain the size data of the target object.

7. An object determination device,[[]] Comprising: An image acquisition module, configured to acquire an overall object image and a plurality of local object images, wherein any one of the local object images overlaps with at least one other local object image among the plurality of local object images; A target object data determination module, configured to perform target detection on the overall object image to obtain target object data; A matching module, configured to perform matching on the plurality of local object images to obtain matching data; A three-dimensional reconstruction module, configured to perform three-dimensional reconstruction on the local object images according to the matching data to obtain a three-dimensional target object image; and A size data determination module, configured to determine the size data of the target object according to the mapping of the target object data relative to the three-dimensional target object image; Wherein, the matching data includes target image matching pairs, and the three-dimensional reconstruction module includes: A reconstruction initial image matching pair determination sub-module, configured to determine a reconstruction initial image matching pair according to the target image matching pairs; An image registration sub-module, configured to perform image registration on the reconstruction initial image matching pairs, the local object images, and the overall object image to obtain an image to be registered and a registered image; A triangulation processing sub-module, configured to perform triangulation processing on the image to be registered and the registered image to obtain triangulated feature points; and A three-dimensional reconstruction sub-module, configured to perform three-dimensional reconstruction on the registered image according to the triangulated feature points and a three-dimensional reconstruction model to obtain the three-dimensional target object image.

8. The device according to claim 7,[[]] Wherein, The matching module includes: A feature detection sub-module, configured to perform feature detection on the local object images to obtain feature points; and A matching sub-module, configured to perform matching on the plurality of local object images according to the feature points to obtain initial matching data, where the initial matching data includes initial image matching pairs and overlapping regions between the images of the initial image matching pairs.

9. The device according to claim 8,[[]] Wherein, The matching module further includes: A geometric verification sub-module, configured to perform geometric verification on the initial image matching pairs according to the overlapping regions of the initial image matching pairs and the feature points to obtain target matching data, where the target matching data includes target image matching pairs.

10. The device according to claim 7,[[]] Wherein, The three-dimensional reconstruction sub-module includes: A bundle adjustment processing unit, configured to perform bundle adjustment processing on the triangulated feature points to obtain target feature points; and A three-dimensional reconstruction unit, configured to perform three-dimensional reconstruction on the registered image according to the target feature points and the three-dimensional reconstruction model to obtain the three-dimensional target object image.

11. The device according to any one of claims 7 to 10,[[]] Wherein, The target object data determination module includes: A target object data determination sub-module, configured to perform target detection on the overall object image by using a target segmentation model to obtain target object point data and target object curve data.

12. The apparatus according to claim 11, wherein, The size data determination module includes: A data determination sub-module, configured to obtain three-dimensional target object point data and three-dimensional target object curve data according to the mapping of the target object point data and the target object curve data relative to the three-dimensional target object image; and A calculation sub-module, configured to perform three-dimensional coordinate calculation on the three-dimensional target object point data and the three-dimensional target object curve data to obtain the size data of the target object.

13. An electronic device, including: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method according to any one of claims 1-6.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to execute the method according to any one of claims 1-6.

15. A computer program product, including a computer program, where the computer program, when executed by a processor, implements the method according to any one of claims 1-6.

Images