Loopback detection method and apparatus, electronic device, and medium

Abstract

Embodiments of the present disclosure relate to a loop detection method and device, electronic equipment and medium, wherein the method comprises: obtaining a frame image set of a scanned object; each frame image in the frame image set has a plurality of scanning marker points, and an initial framework of the scanned object is generated according to the frame image set; a candidate framework point set of each initial framework point in the initial framework is obtained based on coordinate position information of each initial framework point in the initial framework, a target framework point in the candidate framework point set of each initial framework point that satisfies a preset loop detection condition is identified, a link relationship between each initial framework point and the corresponding target framework point is established, and a marker point global optimization process is performed on the initial framework based on the link relationship between the initial framework point and the corresponding target framework point, to obtain a first target framework. By using the above technical solution, the accuracy and reliability of loop detection can be improved, so that the model generated by final three-dimensional reconstruction is more accurate and reliable.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of scanning, in particular to a loop detection method and device, electronic equipment and medium. BACKGROUND

[0002] At present, scanning technology is developing, and a marker point is the most effective means for assisting image stitching. In order to solve the problem of marker point layering error caused by cumulative error, a radius search is usually used to detect a loop.

[0003] However, when scanning a large scene (such as a 20 square meter room, a car, etc.), there is a large cumulative error in the marker point, and there is a serious layering error when looping, so that the model section generated by the final three-dimensional reconstruction looks like a spiral, or different marker points are easily marked as the same marker point, so that the final three-dimensional reconstruction model has an error result. SUMMARY

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a loop detection method and device, electronic equipment and medium.

[0005] The loop detection method provided by the present disclosure comprises:

[0006] Obtaining a frame image set of a scanned object; each frame image in the frame image set has a plurality of scanning marker points, and one scanning marker point corresponds to one marker point of the scanned object;

[0007] Generating an initial framework of the scanned object according to the frame image set; wherein the initial framework includes a plurality of initial framework points, each initial framework point is determined based on the plurality of scanning marker points, and one initial framework point corresponds to a scanning marker point in the plurality of frame images;

[0008] Obtaining a candidate framework point set of each initial framework point in the initial framework based on the coordinate position information of each initial framework point;

[0009] Identifying a target framework point in the candidate framework point set of each initial framework point that meets a preset loop detection condition, and establishing a link relationship between the each initial framework point and the corresponding target framework point;

[0010] Performing marker point global optimization processing on the initial framework based on the link relationship between the initial framework point and the corresponding target framework point, to obtain a first target framework.

[0011] The loop detection device provided by the present disclosure comprises:

[0012] The first obtaining module is configured to obtain a frame image set of a scanned object; each frame image in the frame image set has a plurality of scanning mark points, and one scanning mark point corresponds to one mark point of the scanned object;

[0013] The generating module is configured to generate an initial framework of the scanned object according to the frame image set; the initial framework includes a plurality of initial framework points, each initial framework point is determined based on the plurality of scanning mark points, and one initial framework point corresponds to a scanning mark point in the plurality of frame images;

[0014] The second obtaining module is configured to obtain a candidate framework point set of each initial framework point in the initial framework based on coordinate position information of each initial framework point;

[0015] The identifying module is configured to identify a target framework point in the candidate framework point set of each initial framework point that satisfies a preset loop detection condition;

[0016] The link establishing module is configured to establish a link relationship between each initial framework point and the corresponding target framework point;

[0017] The processing module is configured to perform mark point global optimization processing on the initial framework based on the link relationship between the initial framework point and the corresponding target framework point, to obtain a first target framework. The embodiment of the present disclosure also provides an electronic device, which includes a processor, a memory for storing executable instructions of the processor, and the processor is configured to read the executable instructions from the memory and execute the instructions to implement the loop detection method provided by the embodiment of the present disclosure.

[0018] The embodiment of the present disclosure also provides a computer readable storage medium, which stores a computer program, and the computer program is used to execute the loop detection method provided by the embodiment of the present disclosure.

[0019] The technical scheme provided by the embodiments of the present disclosure has the following advantages compared with the prior art: the loop detection scheme provided by the embodiments of the present disclosure acquires a frame image set of a scanned object; each frame image in the frame image set has a plurality of scanning mark points, one scanning mark point corresponds to one mark point of the scanned object, and an initial framework of the scanned object is generated according to the frame image set; wherein the initial framework includes a plurality of initial framework points, each initial framework point is determined based on the plurality of scanning mark points, one initial framework point corresponds to a scanning mark point in the plurality of frame images, a candidate framework point set of each initial framework point in the initial framework is acquired based on coordinate position information of each initial framework point, a target framework point in the candidate framework point set of each initial framework point that satisfies a preset loop detection condition is identified, and a link relationship between each initial framework point and the corresponding target framework point is established, the initial framework is subjected to mark point global optimization processing based on the link relationship between the initial framework point and the corresponding target framework point, and a first target framework is obtained. By using the above technical scheme, loop detection in any scanning scene can be processed, the loop detection effect is guaranteed, the accuracy and reliability of loop detection are improved, and the detection efficiency and effect under loop detection are further improved, so that the model generated by final three-dimensional reconstruction is more accurate and reliable. BRIEF DESCRIPTION OF DRAWINGS

[0020] The above and other features, advantages, and aspects of the present disclosure will become more apparent by describing in detail the embodiments thereof with reference to the attached drawings. Throughout the drawings, the same or similar reference numerals refer to the same or similar elements. It should be understood that the drawings are schematic, and the original and elements are not necessarily drawn according to the scale.

[0021] Figure 1 A flowchart of a loop detection method provided by the embodiments of the present disclosure;

[0022] Figure 2 A flowchart of another loop detection method provided by the embodiments of the present disclosure;

[0023] Figure 3 A structure diagram of a loop detection device provided by the embodiments of the present disclosure;

[0024] Figure 4 A structure diagram of an electronic device provided by the embodiments of the present disclosure. DETAILED DESCRIPTION

[0025] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms and should not be interpreted in a limited sense as set forth in the embodiments set forth herein, but rather, the embodiments are provided to more thoroughly and completely understand the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0026] It should be understood that each of the steps recited in the method embodiments of the present disclosure can be performed in different orders and / or in parallel. In addition, the method embodiments can include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0027] The term "comprising" and variations thereof as used herein are open-ended, that is "including, but not limited to". The term "based on" is "based, at least in part, on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Related terms are defined in the following description.

[0028] It should be noted that the "first", "second", and the like concepts mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not intended to limit the order or interdependence of the functions performed by these devices, modules or units.

[0029] It should be noted that the modification of "one" or "multiple" mentioned in the present disclosure is illustrative and not limiting, and those skilled in the art should understand that unless otherwise explicitly indicated in the context, it should be understood as "one or more".

[0030] The names of the messages or information exchanged between the devices in the embodiments of the present disclosure are only for illustrative purposes, and are not intended to limit the scope of the messages or information.

[0031] In actual application, when scanning a large scene (such as a 20 square meter room, a car, etc.), there is a large cumulative error of the marker points, and there is a serious layering error when looping back.

[0032] For example, when scanning a super large cylinder, theoretically, one turn can obtain a model with a circular cross section. However, due to the cumulative error, the cross section of the final model is not circular, the starting point and the ending point, which should be the same point, become two points with a certain distance, and the cross section of the entire model looks like a spiral.

[0033] It can also be understood that some landmarks will be considered as two or more landmarks due to accumulated errors, the distance between the landmarks that should be the same landmark is large, which exceeds the distance threshold of the existing loop detection scheme, that is, there is a large layer error.

[0034] To solve the above problems, the present disclosure provides a loop detection method, which comprises the following steps: obtaining a frame image set of a scanned object, each frame image in the frame image set has a plurality of scanning landmarks, and one scanning landmark corresponds to one landmark of the scanned object; generating an initial framework of the scanned object according to the frame image set, wherein the initial framework includes a plurality of initial framework points, each initial framework point is determined based on the plurality of scanning landmarks, and one initial framework point corresponds to a scanning landmark in the plurality of frame images; obtaining a candidate framework point set of each initial framework point in the initial framework based on the coordinate position information of each initial framework point; identifying a target framework point in the candidate framework point set of each initial framework point that meets a preset loop detection condition; establishing a link relationship between each initial framework point and the corresponding target framework point; and performing landmark global optimization processing on the initial framework based on the link relationship between the initial framework point and the corresponding target framework point to obtain a first target framework.

[0035] Therefore, the loop detection in any scanning scene can be processed, the loop detection effect is guaranteed, the accuracy and reliability of the loop detection are improved, the model generated by subsequent three-dimensional reconstruction based on the loop detection result obtained by the present scheme is more accurate and reliable, the scanning requirements of various scenes are met, and the user experience is further improved.

[0036] Specifically, Figure 1 A flowchart of a loop detection method provided by the embodiment of the present disclosure is shown in the figure, the method can be executed by a loop detection device, wherein the device can be realized by software and / or hardware, and can be integrated in an electronic device. As shown in the figure, the method comprises the following steps. Figure 1 The method comprises the following steps.

[0037] Step 101, obtaining a frame image set of a scanned object; each frame image in the frame image set has a plurality of scanning landmarks, and one scanning landmark corresponds to one landmark of the scanned object.

[0038] Step 102, generating an initial framework of the scanned object according to the frame image set; wherein the initial framework includes a plurality of initial framework points, each initial framework point is determined based on the plurality of scanning landmarks, and one initial framework point corresponds to a scanning landmark in the plurality of frame images.

[0039] The frame image set refers to a plurality of frame images obtained when the scanned object is scanned, wherein the scanned object is selected according to an application scenario, such as a plurality of frame images obtained by scanning a car through a monocular camera, or a plurality of frame images obtained by scanning a room through a binocular camera.

[0040] The scanned object has a plurality of actually existing marker points, which can be stickers attached to the scanned object or inherent features of the scanned object.

[0041] Each frame image refers to a frame image of the scanned object obtained at a certain time, which includes a plurality of scanning marker points based on the actually existing marker points and the camera pose (such as the position, orientation and upward direction of the camera) when the frame image is obtained. One scanning marker point corresponds to one marker point of the scanned object.

[0042] The initial frame includes a plurality of initial frame points, each of which is determined based on a plurality of scanning marker points. One initial frame point corresponds to a scanning marker point in a plurality of frame images. It can be understood that one initial frame point is used to represent a certain marker point of the actually existing scanned object. Since a certain marker point of the actually existing scanned object is captured by a plurality of frame images, one initial frame point corresponds to a scanning marker point in a plurality of frame images.

[0043] In the embodiments of the present disclosure, there are many ways to generate the initial frame of the scanned object according to the frame image set.

[0044] In one embodiment, a reference frame is obtained as empty, a first frame image is obtained from the frame image set, a frame point is determined based on all scanning marker points corresponding to the first frame image and added to the reference frame, a second frame image is sequentially obtained from the frame image set, a scanning marker point corresponding to the second frame image is obtained from the reference frame, and it is determined whether the scanning marker point corresponding to the second frame image and the matched to-be-processed frame point are the same marker point based on the scanning marker point feature information. That is, it is determined whether the two represent the same marker point of the scanned object.

[0045] When the scanning marker point corresponding to the second frame image and the matched to-be-processed frame point are the same marker point, it is indicated that the to-be-processed frame point corresponds to the scanning marker point, and the feature information of the to-be-processed frame point needs to be updated, that is, the normal angle, marker point radius and other feature information in the scanning marker point corresponding to the second frame image also need to be added to the feature information of the to-be-processed frame point.

[0046] When the scanning marker point corresponding to the second frame image and the matched to-be-processed frame point are not the same marker point, a new frame point needs to be added, and a frame point is determined based on the scanning marker point corresponding to the second frame image and added to the reference frame.

[0047] After judging frame by frame, the initial frame is obtained.

[0048] In another embodiment, the reference frame is obtained as empty, all scanning marker points corresponding to the frame image set are directly obtained, and then the frame points are added to the reference frame based on the position coordinate information of all scanning marker points after the de-duplication processing to obtain the initial frame.

[0049] Each scanning marker point has corresponding coordinate position information, and the coordinate position information refers to the position coordinate value of the scanning marker point in the world coordinate system. The image coordinate value of each scanning marker point in each frame image is extracted in advance, and the position coordinate information of each scanning marker point in each frame image is obtained based on the image coordinate value of the scanning marker point and the camera internal storage matrix.

[0050] Specifically, after the frame image set is obtained by scanning the scanned object, the initial frame of the scanned object is generated based on the frame image set.

[0051] In step 103, a candidate frame point set of each initial frame point in the initial frame is obtained based on the coordinate position information of each initial frame point.

[0052] The coordinate position information corresponding to the initial frame point can uniquely identify the position of the initial frame point. In the embodiments of the present disclosure, there are many ways to obtain the candidate frame point set of each initial frame point in the initial frame based on the coordinate position information of each initial frame point. In some embodiments, the position coordinate value of each initial frame point is determined based on the coordinate position information of each initial frame point, the candidate position range is determined based on the position coordinate value and the preset radius distance, and the frame points in the candidate position range are obtained as the candidate frame point set of each initial frame point.

[0053] In other embodiments, the position coordinate value of each initial frame point is obtained based on the coordinate position information of each initial frame point, the position distance between two initial frame points is calculated based on the position coordinate value of each initial frame point, and the candidate frame point set of each initial frame point is determined based on the position distance and the distance threshold. The above two ways are only examples of obtaining the candidate frame point set of each initial frame point in the initial frame based on the coordinate position information of each initial frame point. The present disclosure does not make specific restrictions on the way of obtaining the candidate frame point set of each initial frame point in the initial frame based on the coordinate position information of each initial frame point.

[0054] Further, the candidate frame point set of each initial frame point in the initial frame is obtained based on the coordinate position information of each initial frame point.

[0055] Step 104, identifying a target framework point in the candidate framework point set of each initial framework point that satisfies a preset loop detection condition, and establishing a link relationship between each initial framework point and the corresponding target framework point.

[0056] The loop detection condition is set in advance, such as connectivity characteristics, feature matching, and the like, and is specifically selected and set according to an application scenario.

[0057] In the embodiments of the present disclosure, there are many ways to identify the target framework point in the candidate framework point set of each initial framework point that satisfies the preset loop detection condition. In some embodiments, it is determined whether each initial framework point and the candidate framework point in the candidate framework point set belong to the same frame image. When each initial framework point and the candidate framework point do not belong to the same frame image, it is determined whether the feature information of each initial framework point and the candidate framework point matches. When the feature information of each initial framework point and the candidate framework point matches, the candidate framework point is determined as the target framework point.

[0058] It should be noted that if the initial framework point and the candidate framework point belong to the same frame image, it means that the initial framework point and the candidate framework point cannot be the same landmark point, and the next step of judgment can be omitted. Or, in the case where the feature information of the initial framework point and the candidate framework point does not match, it means that the initial framework point and the candidate framework point are not the same landmark point, and the link relationship does not need to be established.

[0059] In other embodiments, it is directly determined whether the feature information of each initial framework point and the candidate framework point in the candidate framework point set matches. In the case where the feature information of each initial framework point and the candidate framework point matches, the candidate framework point is determined as the target framework point.

[0060] The link relationship means that the two framework points can be the same landmark point, that is, each initial framework point and the corresponding target framework point can represent the same landmark point on the scanned object. In the embodiments of the present disclosure, the link relationship can be stored in a separate data table, which records the link relationship between all framework points. One link relationship record the identification (such as the serial number, which uniquely identifies a landmark point) of two framework points in the initial framework.

[0061] Further, the target framework point in the candidate framework point set of each initial framework point that satisfies the preset loop detection condition is identified, and the link relationship between each initial framework point and the corresponding target framework point is established.

[0062] Step 105, based on the link relationship between the initial framework point and the corresponding target framework point, performing landmark global optimization processing on the initial framework to obtain a first target framework.

[0063] In the embodiments of the present disclosure, after the initial frame and the link relationship are acquired, a landmark point global optimization process is needed to further improve the accuracy of loop detection. The landmark point global optimization process is performed based on the initial frame and the link relationship to obtain a first target frame.

[0064] In the embodiments of the present disclosure, there are many ways to perform the landmark point global optimization process on the initial frame based on the link relationship between the initial frame points and the corresponding target frame points to obtain the first target frame. In some embodiments, the updated position coordinates of the initial frame points are obtained by calculating the mapping relationship between the scanning landmark points in each frame of image and the initial frame points in the initial frame, the coordinate position information of each initial frame point, and the link relationship, and the minimum distance between the scanning landmark points in each frame of image and the initial frame points in the initial frame, and the minimum distance between the two initial frame points having the link relationship in the initial frame. The first target frame is obtained by updating the position of the initial frame points based on the updated position coordinates.

[0065] In other embodiments, the updated position coordinates of the first target frame points are obtained by calculating the mapping relationship between the scanning landmark points in each frame of image and the initial frame points in the initial frame, the coordinate position information of each initial frame point, and the link relationship, and the preset global target based on a preset calculation formula or algorithm. Finally, the first target frame is obtained by updating the position of the initial frame points based on the updated position coordinates of the first target frame points.

[0066] The first target frame includes the frame points after the landmark point global optimization process, that is, the link relationship between each first target frame point is more accurate. Therefore, the loop detection result is determined based on the first target frame, the wrong layer problem is further avoided, and the accuracy of loop detection is improved.

[0067] To sum up, the loop detection method of the embodiment of the present disclosure, by acquiring a frame image set of a scanned object; each frame image in the frame image set has a plurality of scanning marker points, one scanning marker point corresponds to a marker point of a scanned object, and an initial framework of the scanned object is generated according to the frame image set; wherein the initial framework includes a plurality of initial framework points, each initial framework point is determined based on a plurality of scanning marker points, one initial framework point corresponds to a scanning marker point in a plurality of frame images, based on the coordinate position information of each initial framework point, a candidate framework point set of each initial framework point in the initial framework is acquired, a target framework point in the candidate framework point set of each initial framework point that meets a preset loop detection condition is identified, and a linking relationship between each initial framework point and the corresponding target framework point is established, and based on the linking relationship between the initial framework point and the corresponding target framework point, a marker point global optimization processing is performed on the initial framework to obtain a first target framework. By adopting the above technical solution, the loop detection in any scanning scene can be processed, the loop detection effect is guaranteed, the accuracy and reliability of the loop detection are improved, and the detection efficiency and effect under the loop detection are further improved.

[0068] Based on the above embodiment, if the linking relationship is incorrect, an incorrect result will be obtained when the loop detection marks different framework points as one framework point. To solve the above problem, specifically, Figure 2 The flowchart of another loop detection method provided by the embodiment of the present disclosure is based on the above embodiment, and the loop detection method is further optimized. As shown in the figure, Figure 2 The method comprises the following steps:

[0069] Step 201, acquiring a reference framework that is empty, acquiring a first frame image from the frame image set, determining a framework point based on all scanning marker points corresponding to the first frame image and adding the framework point to the reference framework, sequentially acquiring a second frame image from the frame image set, and acquiring a to-be-processed framework point from the reference framework that matches the scanning marker point corresponding to the second frame image.

[0070] Step 202, judging whether the scanning marker point corresponding to the second frame image and the matched to-be-processed framework point are the same marker point based on the scanning marker point feature information.

[0071] Step 203, when the scanning marker point corresponding to the second frame image and the matched to-be-processed framework point are the same marker point, updating the feature information of the to-be-processed framework point, and when the scanning marker point corresponding to the second frame image and the matched to-be-processed framework point are not the same marker point, then determining a framework point based on the scanning marker point corresponding to the second frame image and adding the framework point to the reference framework. After frame-by-frame judgment on the frame image set, an initial framework is obtained.

[0072] Each initial frame point in the initial frame and each scan marker point of each frame image contains feature information, which includes but is not limited to: marker point radius, marker point connectivity to other marker points and distance and direction, marker point normal vector, etc.

[0073] Specifically, the reference frame is initialized as empty, and the scan marker points corresponding to each frame image in the frame image set are fused into the reference frame in sequence. More specifically, given an empty reference frame, all scan marker points of the first frame image in the frame image set are used to determine frame points to be added to the reference frame, and the scan marker points corresponding to the remaining frame images are sequentially searched for frame points in the reference frame that meet the conditions as candidate frame points, and whether the scan marker points and the candidate frame points are the same marker point is determined by the feature information. If yes, the feature information of the corresponding candidate frame point in the reference frame is updated (the normal vector of the scan marker point is used as the feature information of the candidate frame point by weighted average, and the radius information is not updated); if no or no frame point meeting the condition is found, a frame point is added to the reference frame based on the scan marker point, and the feature information of the scan marker point is assigned to the added frame point in the reference frame, so as to obtain the initial frame.

[0074] Step 204, the position coordinate value of each initial frame point is determined based on the coordinate position information of each initial frame point, and the candidate position range is determined based on the position coordinate value and the preset radius distance, and the marker points in the candidate position range are obtained as the candidate frame point set of each initial frame point.

[0075] Step 205, whether each initial frame point and the candidate frame points in the candidate frame point set belong to the same frame image is determined.

[0076] Step 206, when each initial frame point and the candidate frame points do not belong to the same frame image, whether the feature information of each initial frame point and the candidate frame points matches is determined.

[0077] Step 207, when the feature information of each initial frame point and the candidate frame points matches, the candidate frame point is determined as the target frame point.

[0078] In the embodiment of the present disclosure, for each initial frame point in the initial frame, other initial frame points (in the initial frame) near the initial frame point are searched as candidate frame points with a preset radius.

[0079] Whether the initial frame point and the corresponding candidate frame point are connected is determined. If they are connected, the determination fails and is terminated. If they are not connected, it is further determined whether the feature information matches. If the feature information does not match, the determination fails and is terminated. If the feature information matches, the determination succeeds and the link relationship is added.

[0080] Among these, connectivity will not expand and needs to be judged one by one.

[0081] Specifically, when the scanning markers corresponding to two initial frame points appear simultaneously in the same frame of an image, i.e., are scanned by a certain frame, these two initial frame points cannot represent the same marker on the scanned object in reality. Instead, they must each represent two different markers on the scanned object in reality. In this case, the two initial frame points can be said to be connected, and further confirmation through feature information is needed.

[0082] Furthermore, when the scan marker points corresponding to two initial frame points do not appear simultaneously in the same frame image, these two initial frame points may represent the same marker point on a real-world scanned object, equivalent to the same marker point being divided into two frame points. In this case, these two initial frame points can be considered disconnected. Further confirmation through feature information is needed.

[0083] For example, determine whether the feature information between the initial frame point and the corresponding candidate frame point matches.

[0084] If there is no match, the judgment fails. The initial frame point and the candidate frame point cannot simultaneously refer to a certain marker point on the scanned object in reality.

[0085] If a match is found and the judgment passes, the initial frame point and the candidate frame point may simultaneously represent a certain landmark point that exists in reality. In this case, a link relationship is added between the two frame points.

[0086] Feature information matching can be understood as two frame points having normal, radius, position, and the normal, radius, and relative position information of surrounding connected points. The goal is to determine whether these feature information can be highly matched through rigid body transformation. If they are frame points determined by the same scan marker points, they can be highly matched.

[0087] In this context, a link indicates that two linked frame points may identify the same landmark, but the probability is not 100% and incorrect links may exist. Subsequent links are used to add constraints; after optimization, correct links will have a very small distance between the two linked frame points.

[0088] Step 208: Based on the mapping relationship between the scan marker points in each frame image and the initial frame points in the initial frame, the coordinate position information and link relationship of each initial frame point, and the minimum distance between the scan marker points in each frame image and the initial frame points in the initial frame, as well as the minimum distance between two initial frame points with a link relationship in the initial frame, the updated position coordinates of the initial frame points are obtained.

[0089] Step 209, updating the position of the initial frame points based on the updated position coordinates to obtain a first target frame.

[0090] Specifically, based on the linking relationship between the initial frame points and the corresponding target frame points, the initial frame is subjected to landmark global optimization processing to obtain the optimized frame point information, i.e., the first target frame.

[0091] Specifically, given the mapping relationship of each scanning landmark point in the frame image to the initial frame point, the initial position of the initial frame point in the initial frame, and the linking relationship inside the initial frame, a global energy minimization problem is solved to obtain the updated position coordinates of the optimized frame image pose and the first target frame point. The energy includes: 1) the distance between points in the frame image and the initial frame that are considered to be the same landmark point; 2) the distance between two frame points with a link in the initial frame, and the weight of this energy is reduced by a penalty function to prevent incorrect links from resulting in incorrect results.

[0092] Wherein, each initial frame point in the initial frame is obtained by weighted average, that is, any scanning landmark point in a single frame image can be mapped to an initial frame point in the initial frame.

[0093] Specifically, given a series of unknowns and an energy function (the energy function takes these unknowns as input to obtain an energy value, and the energy value is a scalar), the input that minimizes the energy function result can be obtained, which can be calculated by relevant mathematical solving or algorithm. The unknowns include the pose information corresponding to the frame image set, the position and normal of the initial frame point in the initial frame. The energy function is input to all single frame images, the initial frame, the mapping relationship of the scanning landmark points in all single frame images and the initial frame points in the initial frame, and the linking relationship, to obtain a scalar energy. Here, the single frame image can adjust its position by changing the pose information.

[0094] Step 210, obtaining a candidate target frame point corresponding to the first target frame point in the first target frame, and determining that the first target frame point and the candidate target frame point are the same frame point based on the frame point feature information.

[0095] Step 211, determining a second target frame point based on the first target frame point and the candidate target frame point, and averaging all feature information corresponding to the same frame point as the feature information of the second target frame point to obtain a second target frame.

[0096] Specifically, for each first target frame point in the optimized first target frame, other first target frame points near the landmark point are searched with a smaller radius, and if the normal features and radius features of these first target frame points are the same, the same frame point is marked, and after updating, the relationship between the frame image and the first target frame is updated again to obtain the second target frame.

[0097] In step 212, based on the mapping relationship between the scanning landmark points in each frame image and the second target frame points in the second target frame, the coordinate position information of each second target frame point, and the minimum distance between the scanning landmark points in each frame image and the second target frame points in the second target frame, the updated position coordinates of the second target frame points are obtained through calculation.

[0098] In step 213, based on the updated position coordinates of the second target frame points, the parameter information corresponding to the third target frame and each frame image is obtained.

[0099] Specifically, for the updated frame image and the second target frame (without including the link relationship), the landmark global optimization processing is performed again to obtain the optimized frame image and the pose of the frame image.

[0100] Specifically, given the mapping relationship between each scanning landmark point in the frame image and the second target frame point, and the initial position of the second target frame point in the second target frame, a global energy minimum problem is solved to obtain the updated position coordinates of the second target frame point and the optimized frame image pose. The energy includes: 1) the distance between the points in the frame image and the second target frame which are considered to be the same landmark point.

[0101] Specifically, a series of unknowns and an energy function (the energy function takes the unknowns as input to obtain an energy value, and the energy value is a scalar) are given, and the input with the minimum energy function result can be obtained through relevant mathematical solving or algorithm calculation. The unknowns include the pose of the frame image set, the position and normal of the second target frame point in the second target frame. The energy function is a scalar energy obtained by inputting all single frame images, the second target frame, the mapping relationship between the scanning landmark points of all single frame images and the second target frame points in the second target frame. Here, the single frame image can adjust its position by changing the pose.

[0102] Therefore, the landmark loop detection is stable and reliable, and the landmark global optimization has better robustness.

[0103] Figure 3 A structure diagram of a loop detection device provided by the embodiment of the present disclosure is shown in FIG. 1. The device can be realized by software and / or hardware, and can be integrated in an electronic device. As shown in FIG. 1, the device includes: Figure 3 ​

[0104] The first acquisition module 301 is configured to acquire a frame image set of a scanned object; each frame image in the frame image set has a plurality of scanning mark points, and one scanning mark point corresponds to one mark point of the scanned object;

[0105] The generation module 302 is configured to generate an initial framework of the scanned object according to the frame image set; the initial framework includes a plurality of initial framework points, each initial framework point is determined based on the plurality of scanning mark points, and one initial framework point corresponds to a scanning mark point in the plurality of frame images;

[0106] The second acquisition module 303 is configured to acquire a candidate framework point set of each initial framework point in the initial framework based on coordinate position information of each initial framework point;

[0107] The identification module 304 is configured to identify a target framework point in the candidate framework point set of each initial framework point that meets a preset loop detection condition;

[0108] The link establishment module 305 is configured to establish a link relationship between each initial framework point and the corresponding target framework point;

[0109] The processing module 306 is configured to perform a mark point global optimization process on the initial framework based on the link relationship between the initial framework point and the corresponding target framework point, to obtain a first target framework.

[0110] Optionally, the first acquisition module 301 is specifically configured to:

[0111] acquire a reference framework that is empty, acquire a first frame image from the frame image set, and determine a framework point based on all scanning mark points corresponding to the first frame image and add the framework point to the reference framework;

[0112] acquire a second frame image from the frame image set in sequence, and acquire a to-be-processed framework point that matches a scanning mark point corresponding to the second frame image from the reference framework;

[0113] determine whether the scanning mark point corresponding to the second frame image and the matched to-be-processed framework point are the same mark point based on scanning mark point feature information;

[0114] when the scanning mark point corresponding to the second frame image and the matched to-be-processed framework point are the same mark point, update feature information of the to-be-processed framework point, and when the scanning mark point corresponding to the second frame image and the matched to-be-processed framework point are not the same mark point, determine a framework point based on the scanning mark point corresponding to the second frame image and add the framework point to the reference framework;

[0115] After judging frame by frame on the frame image set, the initial framework is obtained.

[0116] Optionally, the generating module 302 is specifically configured to:

[0117] determine a position coordinate value of each initial framework point based on the coordinate position information of the initial framework point;

[0118] determine a candidate position range based on the position coordinate value and a preset radius distance;

[0119] obtain framework points in the candidate position range as a candidate framework point set of the initial framework point.

[0120] Optionally, the second obtaining module 303 is specifically configured to:

[0121] determine whether the initial framework point and the candidate framework point in the candidate framework point set belong to the same frame image;

[0122] when the initial framework point and the candidate framework point do not belong to the same frame image, determine whether the feature information of the initial framework point and the candidate framework point matches;

[0123] when the feature information of the initial framework point and the candidate framework point matches, determine that the candidate framework point is the target framework point.

[0124] Optionally, the processing module 306 is specifically configured to:

[0125] based on the mapping relationship between the scanning mark point in each frame image and the initial framework point in the initial framework, the coordinate position information of each initial framework point and the linking relationship, and based on the minimum distance between the scanning mark point in each frame image and the initial framework point in the initial framework and the minimum distance between the two initial framework points in the initial framework having the linking relationship, calculate an updated position coordinate of the initial framework point;

[0126] based on the updated position coordinate, update the position of the initial framework point to obtain the first target framework.

[0127] Optionally, the apparatus further includes:

[0128] a third obtaining module configured to obtain a candidate target framework point corresponding to a first target framework point in the first target framework;

[0129] a determining module configured to determine that the first target framework point and the candidate target framework point are the same framework point based on framework point feature information;

[0130] The determining processing module is configured to determine a second target frame point based on the first target frame point and the candidate target frame point, and average all feature information corresponding to the same frame point to obtain feature information of the second target frame point.

[0131] Optionally, the apparatus further comprises:

[0132] The computing module is configured to calculate the updated position coordinates of the second target frame point based on the mapping relationship between the scanning mark point in each frame of image and the second target frame point in the second target frame, the coordinate position information of each second target frame point, and the minimum distance between the scanning mark point in each frame of image and the second target frame point in the second target frame.

[0133] The fourth obtaining module is configured to obtain parameter information corresponding to each frame of image based on the updated position coordinates of the second target frame point.

[0134] The loop detection apparatus provided in the embodiments of the present disclosure can perform the loop detection method provided in any of the embodiments of the present disclosure, and has the function modules and beneficial effects corresponding to the execution method.

[0135] The embodiments of the present disclosure further provide a computer program product, which comprises computer programs / instructions, and the computer programs / instructions are executed by a processor to implement the loop detection method provided in any of the embodiments of the present disclosure.

[0136] Figure 4 A structural schematic diagram of an electronic device provided in the embodiments of the present disclosure is provided. The following specifically refers to Figure 4 which shows a structural schematic diagram of an electronic device 400 suitable for implementing the electronic device in the embodiments of the present disclosure. The electronic device 400 in the embodiments of the present disclosure can include but is not limited to a mobile terminal such as a mobile phone, a notebook computer, a digital broadcast receiver, a PDA (Personal Digital Assistant), a PAD (Tablet Personal Computer), a PMP (Portable Multimedia Player), a vehicle terminal (for example, a vehicle navigation terminal), and the like, and a fixed terminal such as a digital TV, a desktop computer, and the like. Figure 4 The electronic device shown is only an example, and should not bring any limitation to the functions and use range of the embodiments of the present disclosure.

[0137] As Figure 4As shown, the electronic device 400 can include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401 that can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 402 or loaded into a random access memory (RAM) 403 from a storage device 408. Various programs and data required for the operation of the electronic device 400 are also stored in the RAM 403. The processing device 401, the ROM 402, and the RAM 403 are connected to each other through a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0138] Generally, the following devices can be connected to the I / O interface 405: input devices 406 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 407 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 408 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 409. The communication devices 409 can allow the electronic device 400 to communicate wirelessly or wired with other devices to exchange data. Although Figure 4 The electronic device 400 is shown with various devices, but it should be understood that all of the illustrated devices are not required to implement or have the electronic device. More or less devices can alternatively be implemented.

[0139] In particular, the processes described above with reference to the flowcharts can be implemented as a computer software program according to embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication devices 409, or installed from the storage devices 408, or installed from the ROM 402. When the computer program is executed by the processing device 401, the above-mentioned functions defined in the loop detection method of embodiments of the present disclosure are performed.

[0140] It should be noted that the computer-readable medium described above can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the above. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program used by or in connection with an instruction execution system, apparatus or device. In the present disclosure, the computer-readable signal medium can include a data signal propagated in baseband or propagated as a carrier wave in a propagated data signal, in which the computer-readable program code is contained. Such a propagated data signal can take many forms, including but not limited to, an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium that can send, propagate or transfer the program for use by or in connection with the instruction execution system, apparatus or device. The program code contained in the computer-readable medium can be transmitted by any suitable medium, including but not limited to, wire, cable, RF (radio frequency), etc., or any suitable combination of the above.

[0141] In some embodiments, the client, server, or both can communicate using any current known or future developed network protocol, such as HTTP (Hyper Text Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LAN"), wide area networks ("WAN"), the Internet, and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any current known or future developed networks.

[0142] The computer-readable medium described above can be included in the electronic device described above; or can exist separately from the electronic device, and not be assembled into the electronic device.

[0143] The computer readable medium described above carries one or more programs, when the one or more programs are executed by the electronic device, cause the electronic device to: in a playing process of a video, receive an information display triggering operation of a user; obtain at least two target information associated with the video; display first target information of the at least two target information in an information display area of a playing page of the video, wherein a size of the information display area is smaller than a size of the playing page; and receive a first switching triggering operation of the user, and switch the first target information displayed in the information display area to second target information of the at least two target information.

[0144] Computer program code for carrying out operations of the present disclosure can be written in any one or combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0145] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of present disclosure. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may

[0146] The units described in the embodiments of the present disclosure can be implemented by software, or by hardware, or by a combination of software and hardware. In some cases, the name of the unit does not constitute a limitation on the unit itself.

[0147] The functionality described above in this document can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Application-specific Integrated Circuits (ASICs), Application-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc.

[0148] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0149] According to one or more embodiments of the present disclosure, the present disclosure provides an electronic device, comprising:

[0150] a processor;

[0151] a memory for storing instructions executable by the processor;

[0152] the processor is configured to read the executable instructions from the memory and execute the instructions to implement any of the loop detection methods provided by the present disclosure.

[0153] According to one or more embodiments of the present disclosure, the present disclosure provides a computer-readable storage medium, the storage medium storing a computer program for executing any of the loop detection methods provided by the present disclosure.

[0154] The above description is merely exemplary of preferred embodiments of the present disclosure and of the application of the principles of the technology involved. It is to be understood that the disclosure is not limited to the specific combinations of technical features disclosed herein, but extends to other technical solutions that incorporate any combination of the disclosed technical features or equivalent features without departing from the scope of the disclosure. For example, the technical solutions formed by replacing the above-described features with other technical features disclosed in the present disclosure (but not limited to) having similar functions.

[0155] Further, although operations are depicted in a particular order, this should not be understood as requiring the operations to be performed in the particular order shown or in sequential order, and that certain features of the disclosure can be performed in parallel or concurrently with one another. Also, although individual embodiments have been discussed above, the present disclosure should not be limited to these individual embodiments. Individual features of one embodiment can be combined with individual features of the other embodiments or can be removed alone or in sub-combinations from the embodiments and can be replaced with other features serving the same, similar or other functions, in so doing, it is contemplated that the claims can be drafted to include features from different embodiments where varying and / or complementary features facilitate secure device authentication. Similarly, still other changes can be made within the scope of the disclosure, as readily understood by those of ordinary skill in the art, and equivalents are to be substituted for certain ingredients or procedures. Moreover, the whole disclosure gives examples of the components, procedures, and methods combined with one another to produce the components, procedures, and methods of the disclosure. That the combinations are a given example does not indicate that every demonstrated combination is essential to the practice of the disclosure. The patentable scope of the disclosure is defined by the claims and can include other combinations.

[0156] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

1. A loopback detection method, characterized in that, The method comprises the following steps: acquiring a frame image set of a scanned object; each frame image in the frame image set has a plurality of scanning mark points, and one scanning mark point corresponds to one mark point of the scanned object; generating an initial framework of the scanned object according to the frame image set; wherein the initial framework includes a plurality of initial framework points, each initial framework point is determined based on the plurality of scanning mark points, and one initial framework point corresponds to a scanning mark point in the plurality of frame images; acquiring a candidate framework point set of each initial framework point in the initial framework based on coordinate position information of each initial framework point; identifying a target framework point in the candidate framework point set of each initial framework point that meets a preset loop detection condition, and establishing a linking relationship between the initial framework point and the corresponding target framework point; performing a mark point global optimization process on the initial framework based on the linking relationship between the initial framework point and the corresponding target framework point to obtain a first target framework.

2. The loop detection method of claim 1, wherein, The method of generating an initial framework of a scanned object according to a frame image set comprises the following steps: acquiring an empty reference framework, acquiring a first frame image from the frame image set, and adding a framework point determined based on all scanning mark points corresponding to the first frame image to the reference framework; sequentially acquiring a second frame image from the frame image set, and acquiring a to-be-processed framework point from the reference framework that matches a scanning mark point corresponding to the second frame image; judging whether the scanning mark point corresponding to the second frame image and the matching to-be-processed framework point are the same mark point based on scanning mark point feature information; when the scanning mark point corresponding to the second frame image and the matching to-be-processed framework point are the same mark point, updating the feature information of the to-be-processed framework point, and when the scanning mark point corresponding to the second frame image and the matching to-be-processed framework point are not the same mark point, adding a framework point determined based on the scanning mark point corresponding to the second frame image to the reference framework; after frame-by-frame judgment on the frame image set, the initial framework is obtained.

3. The loop detection method of claim 1, wherein, The method of acquiring a candidate framework point set of each initial framework point in the initial framework based on coordinate position information of each initial framework point comprises the following steps: determining a position coordinate value of each initial framework point based on the coordinate position information of each initial framework point; determining a candidate position range based on the position coordinate value and a preset radius distance; acquiring framework points in the candidate position range as the candidate framework point set of each initial framework point.

4. The loop detection method of claim 1, wherein, The method of identifying a target framework point in the candidate framework point set of each initial framework point that meets a preset loop detection condition comprises the following steps: judging whether each initial framework point and a candidate framework point in the candidate framework point set belong to the same frame image; when each initial framework point and each candidate framework point do not belong to the same frame image, judging whether the feature information of each initial framework point and each candidate framework point matches; when the feature information of each initial framework point and each candidate framework point matches, determining the candidate framework point as the target framework point.

5. The loop detection method of claim 1, wherein, The link relationship between the initial frame points and the corresponding target frame points is used for global optimization processing of the initial frame, and a first target frame is obtained. The mapping relationship between the scanning markers in each frame image and the initial frame points in the initial frame, the coordinate position information of each initial frame point, and the minimum distance between the scanning markers in each frame image and the initial frame points in the initial frame, and the minimum distance between two initial frame points having the link relationship in the initial frame are used for calculation, and an updated position coordinate of the initial frame point is obtained. The position of the initial frame point is updated based on the updated position coordinate, and the first target frame is obtained.

6. The loop detection method according to any one of claims 1 to 5, wherein, Further comprising: A candidate target frame point corresponding to a first target frame point in the first target frame is obtained. It is determined that the first target frame point and the candidate target frame point are the same frame point based on the frame point feature information. A second target frame point is determined based on the first target frame point and the candidate target frame point, and all feature information corresponding to the same frame point is averaged to be feature information of the second target frame point, and a second target frame is obtained.

7. The loop detection method of claim 6, wherein, Further comprising: The mapping relationship between the scanning markers in each frame image and the second target frame points in the second target frame, the coordinate position information of each second target frame point, and the minimum distance between the scanning markers in each frame image and the second target frame points in the second target frame are used for calculation, and an updated position coordinate of the second target frame point is obtained. A third target frame and parameter information corresponding to each frame image are obtained based on the updated position coordinate of the second target frame point.

8. A loopback detection apparatus, characterized by, Comprising: A first acquisition module is configured to acquire a frame image set of a scanned object; each frame image in the frame image set has a plurality of scanning markers, and one scanning marker corresponds to one marker of the scanned object; A generation module is configured to generate an initial frame of the scanned object according to the frame image set; the initial frame includes a plurality of initial frame points, each initial frame point is determined based on the plurality of scanning markers, and one initial frame point corresponds to a scanning marker in a frame image; A second acquisition module is configured to acquire a candidate frame point set of each initial frame point in the initial frame based on coordinate position information of each initial frame point; An identification module is configured to identify a target frame point in the candidate frame point set of each initial frame point that meets a preset loop detection condition; A link establishment module is configured to establish a link relationship between each initial frame point and the corresponding target frame point; A processing module is configured to perform global optimization processing of the initial frame based on the link relationship between the initial frame points and the corresponding target frame points, and obtain a first target frame.

9. An electronic device, comprising: The electronic device comprises: A processor; A memory for storing instructions executable by the processor; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the loop detection method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program configured to implement the loop detection method according to any one of claims 1-7.

Images