Feature Point Matching Method and Related Device

Abstract

An embodiment of the present application provides a fiducial point matching method and related device. The method includes: obtaining the fiducial information of the fiducial point; if it is determined that the fiducial information includes scale factor information and image coordinate information, determining a first matching relationship according to the scale factor; determining a first target matching relationship according to the image coordinate information and the first matching relationship; and determining the encoding in time series of the first fiducial point in the current frame fiducial point image according to the first target matching relationship. It can be realized that when measuring on a mobile camera platform or measuring fiducial points with a large displacement, the scale factors of the fiducial points in the image are matched, thereby realizing the matching of the fiducial points, and the matching relationship of the scale factors is verified according to the motion constraints by the image coordinate information, which is beneficial to improving the accuracy of image measurement.

Description

Technical Field

[0001] This application belongs to the technical field of machine vision, and particularly relates to a fiducial point matching method and related device. Background Art

[0002] In the field of visual image measurement, matching the fiducial points in the images captured by a camera is an essential part. Only with the correct corresponding matching relationship can the pose of the camera platform, the displacement of the fiducial points, or the three-dimensional coordinates, etc. be solved correctly. Among the commonly used matching methods currently, the most common one is the coding matching method, which encodes and marks different fiducial points. For example, circular coding is used to identify the coding mark of the point while extracting the feature points, thereby realizing the matching of fiducial points. However, coding recognition has relatively high requirements for the quality of the image. When the camera undergoes a large displacement resulting in motion blur or the fiducial points are beyond the depth of field range, the dot-line features of the coded fiducial points will become blurred, leading to incorrect recognition of the coding mark, and thus resulting in poor accuracy of image measurement. Another one is the matching method based on feature points, such as the SIFT (Scale Invariant Feature Transform) matching method, which matches the feature points in two views by detecting the feature points in the image, such as corner points. However, this method depends on the texture information in the scene, and may be incorrectly matched when there are many repeated textures in the scene, thereby affecting the accuracy of image measurement. Summary of the Invention

[0003] The embodiments of this application provide a fiducial point matching method and related device, which can achieve matching of fiducial points by matching the scale factors of the fiducial points in the image when measuring on a moving camera platform or measuring fiducial points with a large displacement, and verify the matching relationship of the scale factors according to the motion constraints through the image coordinate information, which is beneficial to improving the accuracy of image measurement.

[0004] In a first aspect, the embodiments of this application provide a fiducial point matching method, which is applied to a processing device in an image measurement system. The image measurement system includes the processing device, a camera platform, fiducial points, and a camera disposed on the camera platform. The processing device is connected to the camera. The fiducial points include a first sub-marker and a second sub-marker, and there are scale features and direction features between the first sub-marker and the second sub-marker. The method includes:

[0005] Obtain the fiducial information of the fiducial points, where the fiducial information includes the scale factor information and / or image coordinate information and / or coding mark information of the fiducial points in multiple frames of fiducial point images and / or the first odometry information of the camera platform;

[0006] If it is determined that the flag information includes the scale factor information and the image coordinate information, then according to the scale factor information, determine a first matching relationship between a first scale factor of a first landmark point in the current frame landmark point image and a second scale factor of a second landmark point in the previous frame landmark point image;

[0007] According to the image coordinate information and the first matching relationship, determine a first target matching relationship between the first landmark point in the current frame landmark point image and the second landmark point in the previous frame landmark point image;

[0008] According to the first target matching relationship, determine the encoding of the first landmark point in the current frame landmark point image in terms of time sequence.

[0009] In a possible example, the determining, according to the scale factor information, a first matching relationship between a first scale factor of a first landmark point in the current frame landmark point image and a second scale factor of a second landmark point in the previous frame landmark point image includes:

[0010] Perform the following operations on the first scale factor of each first landmark point in the current frame landmark point image to obtain the first matching relationship:

[0011] Determine the differences between the currently processed first scale factor and each second scale factor to obtain a plurality of differences; determine the absolute values of the plurality of differences to obtain a plurality of target values;

[0012] Determine the target second scale factor corresponding to the smallest value among the plurality of target values;

[0013] If it is determined that the target value corresponding to the target second scale factor is less than a first preset threshold, then determine that there is a matching relationship between the currently processed first scale factor and the target second scale factor.

[0014] In a possible example, the determining, according to the image coordinate information and the first matching relationship, a first target matching relationship between the first landmark point in the current frame landmark point image and the second landmark point in the previous frame landmark point image includes:

[0015] Perform the following operations on each set of mutually matching first scale factor and second scale factor in the first matching relationship to determine a first target matching relationship between the first landmark point and the second landmark point:

[0016] According to the image coordinate information, determine the conversion relationship between the first landmark point corresponding to the currently processed first scale factor and the second landmark point corresponding to the currently processed second scale factor;

[0017] According to the conversion relationship, construct a target system of equations;

[0018] Solve the target equation set to obtain a rotation matrix;

[0019] According to the rotation matrix, perform reprojection error estimation to obtain a target reprojection error;

[0020] If it is determined that the target reprojection error is less than or equal to a sixth preset threshold, determine that there is a matching relationship between the first fiducial point corresponding to the first scale factor being currently processed and the second fiducial point corresponding to the second scale factor being currently processed.

[0021] In a possible example, after obtaining the fiducial information of the fiducial points, the method further includes: if it is determined that the fiducial information includes the scale factor information, the image coordinate information, and the first odometry information, then according to the second scale factor of the second fiducial point in the previous frame fiducial point image, the first odometry information, and a preset prediction model, predict the odometry prediction information and the scale factor prediction information of the current frame fiducial point image;

[0022] According to the scale factor prediction information and the first scale factor of the current frame fiducial point image, determine a second matching relationship between the third scale factor in the scale factor prediction information and the first scale factor of the current frame fiducial point image;

[0023] According to the second matching relationship, correct the preset prediction model to obtain a corrected preset prediction model;

[0024] According to the scale factor information, determine the first distance from each fiducial point in multiple frames of fiducial point images to the camera optical center;

[0025] According to the first distance, determine second odometry information;

[0026] According to the second odometry information and the scale factor information, determine the first position information of the fiducial points in multiple frames of fiducial point images;

[0027] According to the first odometry information and the scale factor information, determine the second position information of the first fiducial point;

[0028] Match the first position information and the second position information to obtain a second target matching relationship of the fiducial point positions.

[0029] In a possible example, after obtaining the fiducial information of the fiducial points, the method further includes:

[0030] If it is determined that the fiducial information includes the scale factor information, the image coordinate information, and the coded marker information, then according to the scale factor information and the image coordinate information, determine a first target matching relationship;

[0031] Determine the image quality information of each landmark point;

[0032] Filter out the target landmark points whose image quality information meets the preset image conditions;

[0033] Determine the first absolute coding information of the target landmark points according to the coding and marking information;

[0034] Determine the second absolute coding information of the landmark points in the current frame landmark point image according to the first target matching relationship and the first absolute coding information.

[0035] In a possible example, after obtaining the landmark information of the landmark points, the method further includes:

[0036] If it is determined that the landmark information includes the scale factor information, the image coordinate information, the first mileage information, and the coding and marking information, then determine a second target matching relationship according to the scale factor, the image coordinate information, and the first mileage information;

[0037] Determine the second absolute coding information according to the scale factor information, the image coordinate information, and the coding and marking information.

[0038] In a possible example, before obtaining the landmark information of the landmark points, the method further includes:

[0039] Perform the following operations on each first landmark point to obtain the first scale factor of each first landmark point:

[0040] Determine the physical scale value between the first sub-landmark and the second sub-landmark of the currently processed first landmark point;

[0041] Determine the pixel scale value between the first sub-landmark and the second sub-landmark of the currently processed first landmark point imaged in the current frame landmark point image;

[0042] Determine the first scale factor of the currently processed first landmark point according to the physical scale value and the pixel scale value.

[0043] In a possible example, after determining the coding in time sequence of the first landmark points in the current frame landmark point image according to the first target matching relationship, the method further includes:

[0044] If it is detected that there are first target landmark points in the current frame landmark point image that are not matched with second landmark points, then determine the fourth scale factor with the smallest scale factor value and the fifth scale factor with the largest scale factor value in the previous frame landmark point image;

[0045] Determine the difference between the first scale factor corresponding to the first target landmark point and the fourth scale factor to obtain a first target difference;

[0046] Determine the difference between the first scale factor corresponding to the first target landmark point and the fifth scale factor to obtain a second target difference;

[0047] If it is determined that the first target difference is less than a second preset threshold or the second target difference is greater than a third preset threshold, then determine the encoding of the first target landmark point according to the appearance order of the first target landmark point and the existing encoding information of the first landmark point in the current frame landmark point image.

[0048] In a second aspect, an embodiment of the present application provides a landmark point matching device, which is applied to a processing device in an image measurement system. The image measurement system includes the processing device, a camera platform, landmark points, and a camera disposed on the camera platform. The processing device is connected to the camera. The landmark points include a first sub-landmark and a second sub-landmark, and there are scale features and direction features between the first sub-landmark and the second sub-landmark. The device includes an acquisition unit and a determination unit; wherein,

[0049] The acquisition unit is configured to acquire landmark information, where the landmark information includes scale factor information and / or image coordinate information and / or encoding mark information of landmark points in multiple frames of landmark point images and / or first mileage information of the camera platform;

[0050] The determination unit is configured to, if it is determined that the landmark information includes the scale factor information and the image coordinate information, determine a first matching relationship between a first scale factor of a first landmark point in the current frame landmark point image and a second scale factor of a second landmark point in the previous frame landmark point image according to the scale factor information;

[0051] The determination unit is further configured to determine a first target matching relationship between the first landmark point in the current frame landmark point image and the second landmark point in the previous frame landmark point image according to the image coordinate information and the first matching relationship;

[0052] The determination unit is further configured to determine the encoding of the first landmark point in the current frame landmark point image in time sequence according to the first target matching relationship.

[0053] In a third aspect of the present application, an electronic device is provided, including: a processor and a memory; and one or more programs, where the one or more programs are stored in the memory and configured to be executed by the processor, and the programs include instructions for performing some or all of the steps described in the first aspect.

[0054] In a fourth aspect of the embodiments of the present application, a computer-readable storage medium is provided. The computer-readable storage medium is used to store a computer program, and the computer program enables a computer to execute instructions for performing some or all of the steps described in the first aspect of the embodiments of the present application.

[0055] In a fifth aspect of the embodiments of the present application, a computer program product is provided. The computer program product includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to enable a computer to execute some or all of the steps described in the first aspect of the embodiments of the present application. The computer program product may be a software installation package.

[0056] It can be seen that in the embodiments of the present application, the processing device may first obtain the marker information of the marker points. The marker information includes the scale factor information and / or the image coordinate information and / or the encoded marker information and / or the first odometry information of the camera platform in multiple frames of marker point images. Then, if it is determined that the marker information includes the scale factor information and the image coordinate information, according to the scale factor, a first matching relationship between the first scale factor of the first marker point in the current frame of marker point image and the second scale factor of the second marker point in the previous frame of marker point image is determined. Then, according to the image coordinate information and the first matching relationship, a first target matching relationship between the first marker point in the current frame of marker point image and the second marker point in the previous frame of marker point image is determined. Finally, according to the first target matching relationship, the encoding of the first marker point in the current frame of marker point image in time series is determined. It is possible to realize that when measuring on a mobile camera platform or measuring marker points with a large displacement, by matching the scale factors of the marker points in the image, the matching of the marker points is realized, and the matching relationship of the scale factors is verified according to the motion constraints through the image coordinate information, which is beneficial to improving the accuracy of image measurement. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained according to these drawings.

[0058] Figure 1 It is a schematic diagram of the architecture of an image measurement system provided by the embodiments of the present application;

[0059] Figure 2 It is a schematic flowchart of a marker point matching method provided by the embodiments of the present application;

[0060] Figure 3It is a schematic diagram of a fiducial point provided by an embodiment of the present application;

[0061] Figure 4 It is a schematic diagram of another fiducial point provided by an embodiment of the present application;

[0062] Figure 5 It is a schematic diagram of an object plane, an image plane, and an optical center provided by an embodiment of the present application;

[0063] Figure 6 It is a schematic diagram of a fiducial point and a camera provided by an embodiment of the present application;

[0064] Figure 7 It is a schematic diagram of a simulation curve provided by an embodiment of the present application;

[0065] Figure 8 It is a schematic diagram of another simulation curve provided by an embodiment of the present application;

[0066] Figure 9 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application;

[0067] Figure 10 It is a block diagram of the functional units of a fiducial point matching device provided by an embodiment of the present application. Detailed implementation manners

[0068] In order to enable those skilled in the art to better understand the solution of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0069] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but optionally further includes steps or units not listed, or optionally further includes other steps or units inherent to these processes, methods, products, or devices.

[0070] References to "embodiments" in this specification mean that specific features, structures, or characteristics described in connection with the embodiments can be included in at least one embodiment of the present application. The phrase appears in various places in the specification and does not necessarily refer to the same embodiment each time, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described herein can be combined with other embodiments.

[0071] The "and / or" in the embodiments of the present application describes the association relationship of associated objects and indicates that three relationships can exist. For example, A and / or B can represent the following three situations: A exists alone; both A and B exist simultaneously; B exists alone. Herein, A and B can be singular or plural.

[0072] In the embodiments of the present application, the symbol " / " can represent an "or" relationship between the preceding and following associated objects. Additionally, the symbol " / " can also represent a division sign, that is, for performing a division operation. For example, A / B can represent A divided by B.

[0073] The "at least one (item)" or similar expressions in the embodiments of the present application refer to any combination of these items, including any combination of a single item or multiple items, meaning one or more, and multiple means two or more. For example, at least one (item) of a, b, or c can represent the following seven situations: a, b, c, a and b, a and c, b and c, a, b, and c. Herein, each of a, b, and c can be an element or a set containing one or more elements.

[0074] The "equal to" in the embodiments of the present application can be used in combination with "greater than" and is applicable to the technical solutions adopted when it is greater than, or can also be used in combination with "less than" and is applicable to the technical solutions adopted when it is less than. When "equal to" is used in combination with "greater than", it is not used in combination with "less than"; when "equal to" is used in combination with "less than", it is not used in combination with "greater than".

[0075] To better understand the solutions of the embodiments of the present application, the electronic devices, related concepts, and background that may be involved in the embodiments of the present application will be introduced below.

[0076] The electronic device according to the embodiment of the present application is a device with wireless communication functions, which can be referred to as a terminal, user equipment (UE), mobile station (MS), mobile terminal (MT), access terminal device, in-vehicle terminal device, industrial control terminal device, UE unit, UE station, mobile station, remote station, remote terminal device, mobile device, UE terminal device, wireless communication device, UE agent or UE device, etc. The terminal device can be fixed or mobile. It should be noted that the terminal device can support at least one wireless communication technology, such as LTE, new radio (NR), wideband code division multiple access (WCDMA), etc. For example, the terminal device can be a mobile phone, tablet (pad), desktop computer, laptop computer, all-in-one computer, in-vehicle terminal, virtual reality (VR) terminal device, augmented reality (AR) terminal device, wireless terminal in industrial control, wireless terminal in self-driving, wireless terminal in remote medical surgery, wireless terminal in smart grid, wireless terminal in transportation safety, wireless terminal in smart city, wireless terminal in smart home, cellular phone, cordless phone, session initiation protocol (SIP) phone, wireless local loop (WLL) station, personal digital assistant (PDA), handheld device with wireless communication functions, electronic device or other processing devices connected to a wireless modem, wearable device, terminal device in a future mobile communication network or terminal device in a future evolved public land mobile network (PLMN), etc. The electronic device can be a processing device in an image measurement system.

[0077] Please refer to Figure 1 , Figure 1 which is a schematic diagram of the architecture of an image measurement system provided by the embodiment of the present application. As Figure 1As shown in the figure, the image measurement system 1 includes a processing device 10, a camera platform 20, a camera 30, and a fiducial point 40. The processing device 10 is connected to the camera 30.

[0078] Among them, the processing device 10 can be a server, a processor, etc., which are not limited herein.

[0079] Among them, the camera platform 20 can be a mobile platform, such as carriers like vehicles, drones, unmanned ships, etc. The camera 30 is fixed on the mobile platform to capture the stationary fiducial point 40; the camera platform 20 can also be a fixed platform, and the camera 30 is set on the fixed platform to capture the moving fiducial point 40. For example, when the bridge structure is the structure to be measured, during the bridge construction process using the incremental launching method, the beam structure needs to be pushed to the designed position, and during this process, the beam structure will have a large displacement; for example, when a conveyor belt or other structure is the structure to be measured, the conveyor belt is moving during the working process.

[0080] Among them, the fiducial point 40 has physical structural information, which can be a natural fiducial or an artificial fiducial, and is not limited herein. For example, natural fiducials include but are not limited to lane lines on roads, railway tracks, sleepers, spikes, etc.; artificial fiducials include but are not limited to luminous light source fiducials, reflective fiducials, flat panels with regular patterns, etc. The physical structural information not only includes the point-line features with geometric relationships of the fiducial point 40, but also includes the shape features of each part of the fiducial point 40. Among them, points include but are not limited to corner points, centroids, centers of mass, etc., lines include but are not limited to straight lines, curves, dotted lines, etc., and shapes include but are not limited to circles, ellipses, polygons, etc. The relative positions of these point-line features and shape features do not change with the pose change of the fiducial point 40, that is, during the measurement process, the fiducial point 40 always maintains fixed physical structural information. The geometric relationships formed by these point-line features and shape features can be used to find the structural features with scale information. The information with scale features includes but is not limited to the distance between points, the distance between a point and a line, the distance between lines, the scale of a line, the size of a shape, the distance between the centers of mass of shapes, etc. For example, the length and width of a highway lane line, the length and width of a railway sleeper, the distance between the center points of two lane lines, the distance between the center points of two spikes, etc. The aforementioned scale information can be used to calibrate the scale factor in real time. The scale factor refers to the ratio of the image resolution to the physical space resolution, and the scale factor is related to the equivalent focal length of the camera 30 and the distance between the camera 30 and the object to be measured, and represents the magnification relationship between the change value of the image pixel points at each object to be measured in the image collected by the camera 30 and the actual physical displacement of the object to be measured.

[0081] For example, the scale information of the landmark point 40 to be measured is the distance between the center points of two lane lines. The actual distance between the center points of the two lane lines is D. In the image captured by the camera 30, the pixel scale between the center points of the two lane lines is d. Then the scale factor k = d / D. The scale factor can establish the change amount of the image coordinates and the true physical displacement between them, that is .

[0082] Among them, the camera 30 is used to capture the image of the landmark point 40 and transmit the captured image to the processing device 10.

[0083] Among them, the image measurement system 1 further includes a device for obtaining the first mileage information. The visual odometer or the mobile platform equipped with multi-source sensors such as an accelerometer, an inertial measurement unit IMU, an inertial navigation system, a global positioning system GPS, an encoder odometer, etc. can be used to obtain the first mileage information of the camera platform 20. The first mileage information includes but is not limited to the speed, acceleration, moving distance, moving direction, etc. of the camera platform 20.

[0084] Among them, the landmark point matching system may include a memory. The memory is connected to the processing device 10. The memory is used to store the matching relationship between the landmark points 40 in two adjacent frames of landmark point images determined by the processing device 10 and the absolute coding information of the landmark points 40.

[0085] In a possible example, the processing device 10 may first obtain the landmark information of the landmark point 40. The landmark information includes the scale factor information and / or image coordinate information and / or coding mark information of the landmark point 40 in multiple frames of landmark point images and / or the first mileage information of the camera platform 20. Then, if the processing device 10 determines that the landmark information includes the scale factor information and the image coordinate information, it determines the first matching relationship between the first scale factor of the first landmark point in the current frame of landmark point image and the second scale factor of the second landmark point in the previous frame of landmark point image according to the scale factor. Then, the processing device 10 determines the first target matching relationship between the first landmark point in the current frame of landmark point image and the second landmark point in the previous frame of landmark point image according to the image coordinate information and the first matching relationship. Finally, the processing device 10 determines the coding of the first landmark point in the current frame of landmark point image in terms of time sequence according to the first target matching relationship. It can be realized that when measuring on the mobile camera platform 20 or measuring the landmark point 40 with a large displacement amount, by matching the scale factors of the landmark points 40 in the image, the matching of the landmark points 40 is further realized, and the matching relationship of the scale factors is verified according to the motion constraint by the image coordinate information, which is beneficial to improving the accuracy of image measurement.

[0086] Please refer to Figure 2 ,Figure 2 FIG. Figure 2 is a schematic flowchart of a fiducial point matching method provided by an embodiment of the present application, which is applied to a processing device in an image measurement system. The image measurement system includes the processing device, a camera platform, fiducial points, and a camera disposed on the camera platform. The processing device is connected to the camera. The fiducial points include a first sub-marker and a second sub-marker, and there are scale features and direction features between the first sub-marker and the second sub-marker. The method includes:

[0087] Step S201, obtaining fiducial information of the fiducial point, where the fiducial information includes scale factor information and / or image coordinate information and / or encoded marker information of the fiducial point in multiple frames of fiducial point images and / or first odometry information of the camera platform.

[0088] Among them, the scale features include, but are not limited to, the distance between points, the distance between a point and a line, the distance between lines, the scale of a line, the size of a shape, the distance between the centroids of shapes, etc., which are not limited herein. Among them, points include, but are not limited to, corner points, centroids, centers of mass, centers, etc., lines include, but are not limited to, straight lines, curves, dotted lines, etc., and shapes include, but are not limited to, circles, ellipses, polygons, irregular figures, etc. Exemplarily, the first sub-marker and the second sub-marker are circles, and the first scale feature is that the distance between the centers of the first sub-marker and the second sub-marker is equal to a preset distance. The preset distance can be set manually or defaulted by the system, which is not limited herein. The diameter of the first sub-marker is a first preset value, and the diameter of the second sub-marker is a second preset value. The first preset value and the second preset value can be set manually or defaulted by the system, which is not limited herein.

[0089] Please refer to Figure 3 , Figure 3 FIG. Figure 3 is a schematic diagram of a fiducial point provided by an embodiment of the present application. The fiducial point includes a first sub-marker and a second sub-marker. The first sub-marker and the second sub-marker are circles. The scale information of the fiducial point includes that the length of the fiducial point is 98 mm, the width is 60 mm, the diameter of the first sub-marker is 50 mm, the diameter of the second sub-marker is 34 mm, the distance from the center of the second sub-marker to the bottom edge of the fiducial point is 27 mm, the distance between the centers of the first sub-marker and the second sub-marker is 55 mm, and an encoded marker is added inside the first sub-marker.

[0090] Among them, the direction feature refers to the arrangement direction of the first sub-marker and the second sub-marker, and this arrangement direction is consistent with the direction calibrated by the scale factor of the fiducial point. For example, when it is necessary to calibrate the scale factor in the vertical direction, the arrangement direction is the vertical direction. Please refer to Figure 4 , Figure 4It is a schematic diagram of another type of fiducial point provided by an embodiment of the present application. The arrangement directions of the first sub-marker and the second sub-marker can cover all directions, so that the scale factors in multiple directions can be calibrated at one time.

[0091] Among them, the multiple-frame fiducial point images include the current-frame fiducial point image and the historical-frame fiducial point images. The historical-frame fiducial point images include the previous-frame fiducial point image. The fiducial point image is an image of the fiducial points captured by the camera.

[0092] Among them, a single fiducial point corresponds to a scale factor, which can be calibrated in real time.

[0093] Among them, the image coordinate information includes the coordinates of each fiducial point in the image coordinate system.

[0094] Among them, the encoded marker information includes the encoded marker of the fiducial point, the scale information between the first sub-marker and the second sub-marker, the length and width of the fiducial point, the distances from the centroid of the first sub-marker to the bottom side and the side of the fiducial point, the distances from the centroid of the second sub-marker to the bottom side and the side of the fiducial point, etc.

[0095] Among them, the first mileage information includes the speed, acceleration, moving distance, moving direction, etc. of the camera platform when each frame of image is captured.

[0096] Step S202, if it is determined that the marker information includes the scale factor information and the image coordinate information, then according to the scale factor information, determine the first matching relationship between the first scale factor of the first fiducial point in the current-frame fiducial point image and the second scale factor of the second fiducial point in the previous-frame fiducial point image.

[0097] For example: The calibration direction of the scale factor of the fiducial point is the vertical direction. The first-frame fiducial point image includes three fiducial points, which appear in sequence from top to bottom. Then the three fiducial points are encoded as 1, 2, and 3 according to the appearance time sequence. The second-frame fiducial point image includes four fiducial points, and the fourth fiducial point appears below the fiducial point image. Then the encoding of the fourth fiducial point is 4, and the four fiducial points are encoded as 1, 2, 3, and 4 according to the appearance time sequence.

[0098] Step S203, according to the image coordinate information and the first matching relationship, determine the first target matching relationship between the first fiducial point in the current-frame fiducial point image and the second fiducial point in the previous-frame fiducial point image.

[0099] Among them, the proximity matching is performed according to the magnitude of the scale factor.

[0100] Step S204, according to the first target matching relationship, determine the encoding of the first fiducial point in the current-frame fiducial point image in terms of time sequence.

[0101] After determining the encoding of the first fiducial point in the current frame fiducial point image in terms of time sequence according to the first target matching relationship, the following steps are further included: If it is detected that there is a first target fiducial point in the current frame fiducial point image that does not match the second fiducial point, determine the fourth scale factor with the smallest scale factor value and the fifth scale factor with the largest scale factor value in the previous frame fiducial point image; determine the difference between the first scale factor corresponding to the first target fiducial point and the fourth scale factor to obtain a first target difference; determine the difference between the first scale factor corresponding to the first target fiducial point and the fifth scale factor to obtain a second target difference; if it is determined that the first target difference is less than the second preset threshold or the second target difference is greater than the third preset threshold, determine the encoding of the first target fiducial point according to the appearance order of the first target fiducial point and the existing encoding information of the first fiducial point in the current frame fiducial point image.

[0102] Exemplarily, the current frame fiducial point image includes five fiducial points with scale factors of 1, 8, 20, 63, and 120 respectively, and the previous frame fiducial point image includes four fiducial points with encodings of 4, 5, 6, and 7 respectively and scale factors of 2, 9, 21, and 64 respectively. The first preset threshold is 5, the second preset threshold is -3, and the third preset threshold is 20. Then the first matching relationship of the scale factors is as follows: According to the condition that the absolute value of the difference in scale factors is less than the first preset threshold, the fiducial point with a scale factor of 1 in the current frame fiducial point image matches the fiducial point with a scale factor of 2 in the previous frame fiducial point image, so the encoding of the fiducial point with a scale factor of 1 in the current frame fiducial point image is 4; the fiducial point with a scale factor of 8 in the current frame fiducial point image matches the fiducial point with a scale factor of 9 in the previous frame fiducial point image, so the encoding of the fiducial point with a scale factor of 8 in the current frame fiducial point image is 5; the fiducial point with a scale factor of 20 in the current frame fiducial point image matches the fiducial point with a scale factor of 21 in the previous frame fiducial point image, so the encoding of the fiducial point with a scale factor of 20 in the current frame fiducial point image is 6; the fiducial point with a scale factor of 63 in the current frame fiducial point image matches the fiducial point with a scale factor of 64 in the previous frame fiducial point image, so the encoding of the fiducial point with a scale factor of 63 in the current frame fiducial point image is 7. And the fiducial point with a scale factor of 120 in the current frame fiducial point image (the first target fiducial point) has no corresponding matching second fiducial point. Since the second target difference between the fiducial point with a scale factor of 120 in the current frame fiducial point image and the largest scale factor 64 (the fifth scale factor) in the previous frame fiducial point image is greater than the third preset threshold, that is, 120 - 64 > 20, the fiducial point with a scale factor of 120 in the current frame fiducial point image is a newly appeared point and is encoded as 8.

[0103] Another possible example is that the current frame fiducial point image includes four fiducial points with scale factors of 2, 9, 21, and 64 respectively, the previous frame fiducial point image includes three fiducial points with encodings of 7, 6, and 5 respectively, and the scale factors are 8, 20, and 63 respectively. The first preset threshold is 5, the second preset threshold is -3, and the third preset threshold is 20. Then the first matching relationship of the scale factors is as follows: According to the condition that the absolute value of the difference in scale factors is less than the first preset threshold, the fiducial point with a scale factor of 9 in the current frame fiducial point image matches the fiducial point with a scale factor of 8 in the previous frame fiducial point image, so the encoding of the fiducial point with a scale factor of 9 in the current frame fiducial point image is 7; the fiducial point with a scale factor of 21 in the current frame fiducial point image matches the fiducial point with a scale factor of 20 in the previous frame fiducial point image, so the encoding of the fiducial point with a scale factor of 21 in the current frame fiducial point image is 6; the fiducial point with a scale factor of 64 in the current frame fiducial point image matches the fiducial point with a scale factor of 63 in the previous frame fiducial point image, so the encoding of the fiducial point with a scale factor of 64 in the current frame fiducial point image is 5. However, the fiducial point with a scale factor of 2 in the current frame fiducial point image (the first target fiducial point) has no corresponding matching second fiducial point. Since the first target difference between the fiducial point with a scale factor of 2 in the current frame fiducial point image and the smallest scale factor 8 (the fourth scale factor) in the previous frame fiducial point image is less than the second preset threshold, that is, 2 - 8 < -3, the fiducial point with a scale factor of 2 in the current frame fiducial point image is a newly emerged point and is encoded as 8.

[0104] Among them, the image coordinate information of the fiducial point is used to determine whether the matching situation is credible according to the motion constraint. The motion constraint means that in the case of high-speed photography or low-speed movement, the change of the camera platform between two frames of fiducial point images is approximately only an angular change, the displacement change can be ignored, and the reprojection error is less than the sixth preset threshold.

[0105] Among them, the first fiducial point and the second fiducial point that match each other in the first target matching relationship can be considered as the same fiducial point.

[0106] It can be seen that in the embodiment of the present application, the processing device may first obtain the landmark information of the landmark points. The landmark information includes the scale factor information and / or the image coordinate information and / or the encoded marker information of the landmark points in multiple frames of landmark point images and / or the first odometry information of the camera platform. Then, if it is determined that the landmark information includes the scale factor information and the image coordinate information, according to the scale factor, the first matching relationship between the first scale factor of the first landmark point in the current frame landmark point image and the second scale factor of the second landmark point in the previous frame landmark point image is determined. Then, according to the image coordinate information and the first matching relationship, the first target matching relationship between the first landmark point in the current frame landmark point image and the second landmark point in the previous frame landmark point image is determined. Finally, according to the first target matching relationship, the encoding of the first landmark point in the current frame landmark point image in time sequence is determined. It can be realized that when measuring on a mobile camera platform or measuring landmark points with a large displacement, by matching the scale factors of the landmark points in the image, the matching of the landmark points is further realized, and the matching relationship of the scale factors is verified according to the motion constraints by the image coordinate information, which is beneficial to improving the accuracy of image measurement.

[0107] In a possible example, in terms of determining the first matching relationship between the first scale factor of the first landmark point in the current frame landmark point image and the second scale factor of the second landmark point in the previous frame landmark point image according to the scale factor information, the above method may include the following steps: performing the following operations on the first scale factor of each first landmark point in the current frame landmark point image to obtain the first matching relationship: determining the difference between the currently processed first scale factor and each second scale factor to obtain a plurality of differences; determining the absolute values of the plurality of differences to obtain a plurality of target values; determining the target second scale factor corresponding to the smallest value among the plurality of target values; if it is determined that the target value corresponding to the target second scale factor is less than the first preset threshold, it is determined that there is a matching relationship between the currently processed first scale factor and the target second scale factor.

[0108] Among them, the current frame landmark image has three first landmarks with scale factors of 9, 21, and 64 respectively. The previous frame landmark image has three second landmarks, and the scale factors corresponding to the three landmarks are 8, 20, and 63 respectively. The first preset threshold is 1.5. The multiple differences between the first scale factor of the first landmark with a scale factor of 9 and the second scale factors of each second landmark are 1, -11, and -54 respectively. The multiple target values obtained after taking the absolute values of the multiple differences are 1, 11, and 54. The smallest value is 1, 1 < 1.5, and the target second scale factor corresponding to 1 is 8. Then, at this time, the first scale factor 9 and the second scale factor 8 have a matching relationship. Similarly, it can be obtained that the first scale factor 21 and the second scale factor 20 match, and the first scale factor 64 and the second scale factor 63 match. Then, the finally obtained first matching relationship is that the first scale factor 9 and the second scale factor 8 match, the first scale factor 21 and the second scale factor 20 match, and the first scale factor 64 and the second scale factor 63 match.

[0109] Among them, the first preset threshold can be set manually or default by the system, and no limitation is made here.

[0110] It can be seen that in this example, by matching the scale factors of the landmarks in the image, the matching of the landmarks is realized, which is beneficial to improving the accuracy of image measurement.

[0111] In a possible example, in terms of determining the first target matching relationship between the first landmark in the current frame landmark image and the second landmark in the previous frame landmark image according to the image coordinate information and the first matching relationship, the above method may include the following steps: perform the following operations on each group of mutually matching first scale factor and second scale factor in the first matching relationship to determine the first target matching relationship between the first landmark and the second landmark: according to the image coordinate information, determine the conversion relationship between the first landmark corresponding to the currently processed first scale factor and the second landmark corresponding to the currently processed second scale factor; according to the conversion relationship, construct a target equation set; solve the target equation set to obtain a rotation matrix; according to the rotation matrix, perform reprojection error estimation to obtain a target reprojection error; if it is determined that the target reprojection error is less than or equal to the sixth preset threshold, it is determined that the first landmark corresponding to the currently processed first scale factor and the second landmark corresponding to the currently processed second scale factor have a matching relationship.

[0112] Among them, assuming that the camera only has angular changes, the pose change of two frames of the camera can be described by a rotation matrix R composed of Euler angles in three directions:

[0113]

[0114] Let the normalized coordinates of the matching points in two images be:

[0115]

[0116] Among them, represents the normalized coordinates of the first fiducial point, represents the normalized coordinates of the second fiducial point,

[0117] The two coordinates satisfy:

[0118]

[0119] Expanding it gives:

[0120]

[0121] Among them, K is the camera internal parameter matrix, which is obtained by the camera calibration method.

[0122] Among them, each pair of the first fiducial point and the second fiducial point can provide two equations, and 3 pairs of the first fiducial points and the second fiducial points can solve 9 parameters. An overdetermined system of equations, that is, the target system of equations, is constructed using at least 3 pairs of the above matching points, and the least squares solution is obtained by singular value decomposition to get the rotation matrix R:

[0123]

[0124] Project the first fiducial point of the current frame fiducial point image onto the previous frame fiducial point image through the calculated rotation matrix R to obtain the projected point , calculate the projected point and the actual point to estimate the reprojection error by calculating the Euclidean distance:

[0125]

[0126] Among them, Error represents the reprojection error.

[0127] Among them, the sixth preset threshold can be set manually or by default in the system, and it is not limited here.

[0128] Among them, if it is determined that the target reprojection error is greater than the sixth preset threshold, it is determined that there is no matching relationship between the first fiducial point corresponding to the current processed first scale factor and the second fiducial point corresponding to the current processed second scale factor.

[0129] It can be seen that in this example, verifying the matching relationship of the scale factor according to the motion constraint based on the image coordinate information is beneficial to improving the accuracy of image measurement.

[0130] In a possible example, after obtaining the landmark information of the landmark points, the above method may include the following steps: If it is determined that the landmark information includes the scale factor information, the image coordinate information, and the first mileage information, then according to the second scale factor of the second landmark point in the previous frame landmark point image, the first mileage information, and a preset prediction model, predict the mileage prediction information and the scale factor prediction information of the current frame landmark point image; according to the scale factor prediction information and the first scale factor of the current frame landmark point image, determine the second matching relationship between the third scale factor in the scale factor prediction information and the first scale factor of the current frame landmark point image; according to the second matching relationship, correct the preset prediction model to obtain a corrected preset prediction model; according to the scale factor information, determine the first distance from each landmark point in multiple frames of landmark point images to the camera optical center; according to the first distance, determine the second mileage information; according to the second mileage information and the scale factor information, determine the first position information of the landmark points in multiple frames of landmark point images; according to the first mileage information and the scale factor information, determine the second position information of the first landmark point; match the first position information and the second position information to obtain the second target matching relationship of the landmark point positions.

[0131] Among them, methods such as Kalman filtering can be used to predict the mileage of the current frame camera platform and the scale factor of the landmark points using a preset prediction model, and then numerically approximate match the measured value (the first scale factor) and the predicted value (the third scale factor) of the scale factor of the current frame landmark point image to obtain the second matching relationship between the first scale factor and the third scale factor. If the current camera frame is the first frame, then use the preset prediction model based on motion constraints for Kalman filtering with the initial state variables and the initial covariance matrix to predict the scale factor of the next frame of landmark points. If the current camera frame is not the first frame, then the set of the first scale factors of the current frame is numerically approximated matched with the set of predicted third scale factors, and the state variables and the covariance matrix of the current preset prediction model are corrected.

[0132] Among them, since the relationship between the scale factor and the equivalent focal length is , where represents the distance from the object to the camera optical center along the optical axis direction. Therefore, for each detected scale factor of a landmark point, the distance from this point to the optical center can be calculated. Subtract the distances from the corresponding points of two frames to the optical center direction to obtain the moving distance of the camera / landmark points between the two frames . At the same time, if the frame rate fps of the camera acquisition is known, that is, it is considered that the time difference between two frames is 1 / fps, the moving speed of the camera / landmark points of this frame can be obtained as Therefore, each time the matching relationship of the landmark points between two frames is obtained, the moving distance of the camera or the landmark points can be calculated through the scale factor. and the moving speed v. Regarding the characteristics that the moving distance and moving speed can be directly obtained from the scale factor, the scale factors of all landmark points in the camera frame are tracked based on the Kalman filter: the state vector is defined as the displacement and the speed :

[0133]

[0134] The prediction model is:

[0135]

[0136] where the state transition matrix is:

[0137]

[0138] where the control input matrix is:

[0139]

[0140] where is the control input (or external input), representing the active control quantity of the system (such as the applied force, acceleration, etc.). This variable is the external factor that causes the state variables to change; in this embodiment, the state variables are displacement and speed. Therefore, the control input usually corresponds to the acceleration, and this acceleration can be caused by factors such as the throttle, brake, motor, or manual push by a person.

[0141] where the process noise obeys the Gaussian distribution:

[0142]

[0143] The observation equation is:

[0144]

[0145] where the observation matrix is: , and the observation noise obeys the Gaussian distribution: The prediction step of the Kalman filter: , , that is, predicting the displacement of the current frame as , converting the predicted displacement into the predicted scale factor , and performing a proximity match in size with the actual scale factor of the landmark points in the current frame. Among them, is the state error covariance matrix, and Q is the error covariance matrix of the noise. After confirming the matching relationship, assume that the scale factor of the landmark points in the previous frame is , and the scale factor of the landmark points in the current frame is . Then the observation equation has a displacement of , and a velocity of . The correction steps of the Kalman filter: , , , with the first frame of the camera / landmark points being stationary as the initial value: . Repeating the process of predicting the scale factor and correcting the error covariance matrix can achieve the tracking of the landmark points.

[0146] For example, the second set of scale factors of the landmark point image in the previous frame , the predicted third set of scale factors , corresponds to the predicted , corresponds to the predicted , corresponds to the predicted , corresponds to the predicted , the first set of scale factors of the landmark point image in the current frame . If the result of the proximity match between the third set of scale factors and the first set of scale factors is that the second matching relationship is matches with , matches with , matches with , is the smallest scale factor in the third set of scale factors, is the largest scale factor in the second set of scale factors, or , then the corresponding landmark point is a newly emerged landmark point, and its time series encoding can be performed.

[0147] Among them, from the second matching relationship, the third matching relationship between the first scale factor in the landmark point image of the current frame and the second scale factor of the landmark points in the previous frame can be obtained. For example, the second matching relationship is matches with , matches with , matches with , then the third matching relationship is matches with , Match with match, Match with .

[0148] Among them, since the relationship between the scale factor and the equivalent focal length is , where represents the distance from the object to the optical center of the camera along the optical axis. Therefore, for each detected scale factor of a fiducial point, the distance from the fiducial point to the optical center, i.e., the first distance, can be calculated. By subtracting the distances of the corresponding points in two frames in the direction to the optical center, the camera movement distance between the two frames can be obtained , since multiple fiducial points can be observed between two frames, the scale factors of all the fiducial points observed in the two frames can be involved in the calculation to obtain multiple camera movement distances. The least squares optimization is performed on the multiple camera movement distances to obtain the best camera movement distance between the two frames, i.e., the second odometry information. For example, the mean value of the multiple camera movement distances can be obtained and used as the best camera movement distance between the two frames

[0149] Among them, the first position information includes the distances of the fiducial points that appear in the fiducial point images of all frames relative to the first camera frame, and the first position information is used as the reference for global matching. The second position information includes the distances of the fiducial points in the current frame fiducial point image relative to the first camera frame. Due to the influence of feature point extraction or noise, there is a certain probability of false matching in obtaining the third matching relationship, and even the number of measured fiducial points does not match the actual number. It is also necessary to further optimize the result of the temporal coding matching using the odometry information. The distance of the fiducial point relative to the first camera frame (assuming that the nth fiducial point is observed in the mth frame). Similarly, since the same fiducial point can be observed by multiple camera frames, the least squares optimization can be performed on the multiple distances of the fiducial point relative to the first camera frame (the odometry of the first camera frame is zero) calculated by different camera frames to obtain the best distance of each fiducial point relative to the first camera frame, i.e., the first position information. For example: the mean value of the multiple distances is obtained as the best distance of each fiducial point relative to the first camera frame

[0150] For example, the first position information includes fiducial points 1, 2, 3, 4, 5, and the distances of each fiducial point relative to the first camera frame are 10, 20, 30, 40, 50. The second position information includes the distances of three fiducial points in the current frame fiducial point image relative to the first camera frame (second position information) as 29, 38, 49. At this time, the first position information and the second position information are subjected to proximity matching of numerical magnitudes, and the obtained second target matching relationship is that 29 matches 30, 38 matches 40, and 49 matches 50. From this, it can be known that the fiducial point with a distance of 29 relative to the first camera frame in the current frame fiducial point image is the same fiducial point as the fiducial point with code 3 in the first position information, the fiducial point with a distance of 38 relative to the first camera frame in the current frame fiducial point image is the same fiducial point as the fiducial point with code 4 in the first position information, and the fiducial point with a distance of 49 relative to the first camera frame in the current frame fiducial point image is the same fiducial point as the fiducial point with code 5 in the first position information.

[0151] It can be seen that in this example, when the actual valid marking information includes scale factor information, image coordinate information, and first odometry information, two least squares optimizations are performed when obtaining the first position information, and the first position information is used as a reference for global matching. Therefore, when obtaining the matching relationship of fiducial points through position information matching, it is beneficial to improve the accuracy of matching.

[0152] In a possible example, after obtaining the fiducial information of the fiducial points, the above method may include the following steps: If it is determined that the fiducial information includes the scale factor information, the image coordinate information, and the encoded marking information, then determine a first target matching relationship according to the scale factor information and the image coordinate information; determine the image quality information of each fiducial point; screen out target fiducial points whose image quality information meets a preset image condition; determine the first absolute encoding information of the target fiducial points according to the encoded marking information; and determine the second absolute encoding information of the fiducial points in the current frame fiducial point image according to the first target matching relationship and the first absolute encoding information.

[0153] Among them, the image quality information includes the contrast, gradient, and other indicators for judging image quality, which are not limited herein.

[0154] Among them, the preset image condition may be that the image contrast of the fiducial point is greater than or equal to a fourth preset threshold and the gradient is greater than or equal to a fifth preset threshold. The fourth preset threshold and the fifth preset threshold can be set manually or default by the system, which are not limited herein.

[0155] Among them, the absolute coding information is the number of pre-set landmark points. Different landmark points can correspond to different coding marks. Therefore, a correspondence relationship can be established between the coding marks and the absolute coding. After the coding mark of the landmark point is recognized, the absolute coding of the landmark point can be obtained accordingly.

[0156] Among them, according to the first target matching relationship and the absolute coding information of the target landmark point, the timing coding of the current frame landmark point image is replaced with the absolute coding. For example, the current frame landmark point image includes four landmark points, which are defined as the first landmark point, the second landmark point, the third landmark point, and the fourth landmark point from top to bottom. The previous frame landmark point image includes landmark points 1, 2, 3, and 4. The first target matching relationship is that the first landmark point in the current frame landmark point image matches the landmark point with coding 2 in the previous frame landmark point image, the second landmark point in the current frame landmark point image matches the landmark point with coding 3 in the previous frame landmark point image, and the third landmark point in the current frame landmark point image matches the landmark point with coding 4 in the previous frame landmark point image. The first absolute coding information identifies that the absolute codings of the first landmark point and the fourth landmark point are 77 and 69. The pre-stored absolute coding information table shows that the landmark point codings are 77, 11, 18, and 22 in sequence. Then, the pre-stored absolute coding information table can be updated to 77, 11, 18, 22, 69. At the same time, it is determined that the codings of the landmark points in the current frame landmark point image in terms of timing are 2, 3, 4, and 5. At the same time, the codings in this timing can be replaced with 11, 18, 22, and 69, that is, the second absolute coding information is obtained.

[0157] It can be seen that in this example, when the actual valid landmark information includes the scale factor, the image coordinate information, and the coding mark information, while matching the timing coding, the image quality judgment is introduced, and the timing coding is replaced with the absolute coding information, which is beneficial to improving the accuracy of landmark point matching.

[0158] In a possible example, after obtaining the landmark information of the landmark point, the above method may include the following steps: If it is determined that the landmark information includes the scale factor information, the image coordinate information, the first mileage information, and the coding mark information, then determine the second target matching relationship according to the scale factor, the image coordinate information, and the first mileage information; determine the second absolute coding information according to the scale factor information, the image coordinate information, and the coding mark information.

[0159] Among them, when the valid and available flag information includes scale factor information, image coordinate information, first mileage information, and coding mark information, the above two marker point matching methods can be used simultaneously to respectively determine the second target matching relationship and the second absolute coding information. If there is only the timing coding matching method, the interval between marker points may be incorrect due to mis-matching. If there is only the coding matching method, the correct coding may not be detected due to the poor clarity of the marker points. In addition, the marker point position information and the second absolute coding information can be compared, and damaged or missing marker points are abnormal marker points.

[0160] For example, in one measurement, if the second absolute coding information is 22 and 8, while the original absolute coding sequence should be 22, 69, and 8, and it is also known from the analysis of the marker point position information that the distance interval between the 22nd marker point and the 8th marker point is much larger than the pre-set distance interval, then it is considered that the marker point with the absolute coding of 69 is damaged or missing.

[0161] It can be seen that in this example, when the flag information includes scale factor information, image coordinate information, first mileage information, and coding mark information, the second target matching relationship and the second absolute coding information can be determined in parallel, which is beneficial to improving the accuracy of marker point matching.

[0162] In a possible example, before obtaining the flag information of the marker points, the above method may include the following steps: performing the following operations on each first marker point to obtain the first scale factor of each first marker point: determining the physical scale value between the first sub-marker and the second sub-marker of the currently processed first marker point; determining the pixel scale value between the first sub-marker and the second sub-marker of the currently processed first marker point imaged in the current frame of marker point image; and determining the first scale factor of the currently processed first marker point according to the physical scale value and the pixel scale value.

[0163] Please refer to Figure 5 , Figure 5 which is a schematic diagram of an object plane, an image plane, and an optical center provided by an embodiment of the present application. Figure 5 It shows the optical center C, the optical axis, the object plane, and the image plane. The physical scale of the marker points in the object plane is d, which can be the physical scale between the first sub-marker and the second sub-marker. For example, the distance between the centroid of the first sub-marker and the centroid of the second sub-marker is d. The pixel scale of the marker points imaged in the image plane is , which can be the pixel scale between the first sub-marker and the second sub-marker imaged in the image plane. For example, the distance between the centroid of the first sub-marker and the centroid of the second sub-marker imaged in the image plane is , then the scale factor of the marker points is When the camera observes the object to be measured each time, the foregoing method can be used to calibrate the scale factor. This means that each time the camera makes an observation, the scale factor is calibrated, that is, real-time calibration of the scale factor is achieved.

[0164] Among them, the centroid scale can replace the center point scale for the following reasons: Assume that the shape of a component in the fiducial point is circular with a radius of r. The origin of the world coordinate system is established at the center of the circle, and the plane where the circle is located is set as the W-XY plane of the world coordinate system, as Figure 6 shown, Figure 6 is a schematic diagram of a fiducial point and a camera provided by an embodiment of the present application, Figure 6 in which a circular fiducial point is shown. The circular fiducial point becomes an ellipse in the image. Given that the quadratic curve equation is , its matrix form can be written as , where the quadratic curve coefficient matrix C is:

[0165]

[0166] Then the equation of the circle with the center at the origin of the world coordinate system is:

[0167]

[0168] Converting it to matrix form, the coefficient matrix C1 is:

[0169]

[0170] Assume that the world coordinate system W-XYZ first rotates around the Z axis by , then rotates around the current X axis by , and finally rotates around the current Y axis by . The directions of the axes of the rotated coordinate system are parallel to the axes of the camera coordinate system . The rotation matrix R is:

[0171]

[0172] where are respectively:

[0173]

[0174]

[0175]

[0176] Then, the coordinate axes are translated to coincide with the camera coordinate system through the translation vector . Assume that the equivalent focal lengths in the x and y directions of the camera are 、 , and the principal point coordinates in the x and y directions are respectively , Then, the projection matrix M from the world coordinate system W-XYZ to the image coordinate system O-xy can be obtained as follows:

[0177]

[0178] where O is a 3×3 zero matrix.

[0179] Since the points on the circle lie in the same plane, the transformation relationship between the points in the image coordinate system and the points in the world coordinate system can be represented by the homography matrix H: , where H is

[0180]

[0181] where represents the distance from the world point to the camera optical center along the optical axis direction.

[0182] The expression of C1 becomes:

[0183] That is, the matrix form of the elliptic curve in the image can be represented by . The centroid coordinates (u, v) of the ellipse are given by:

[0184]

[0185] Substituting the elements of into the aforementioned centroid coordinate expression, we get and . Further simplifying the formula. Due to the symmetry of the circle, rotating around the Z-axis of the world coordinate system will not change its image. Therefore, we can set . The principal point coordinates of the camera will change the pixel coordinates of the center of the ellipse in the image, but will not change the difference value between the centroid of the ellipse and the projection point of the center of the circle. Therefore, we can set , . The simplified centroid pixel coordinates of the ellipse are:

[0186]

[0187]

[0188] Since in most photogrammetry systems, the centroid pixel coordinates of the ellipse can be further simplified to:

[0189]

[0190] And the image point coordinates of the center of the circle are:

[0191]

[0192]

[0193] It can be known the distance between the centroid coordinates and the center - point image coordinates in the image mainly lies in

[0194]

[0195] and

[0196]

[0197] From these two items, a deviation coefficient is defined

[0198]

[0199] This coefficient characterizes the deviation degree between the centroid coordinates and the center - point image coordinates in the image.

[0200] Explore the deviation coefficient through numerical simulation and the deviation between the centroid coordinates and the center - point image coordinates The relationship between them. Considering the default simulation parameters under relatively harsh non - vertical shooting conditions are shown in Table 1 below:

[0201] Table 1

[0202]

[0203] By the method of controlling variables, separately change the magnitude of , calculate the corresponding deviation coefficient and the distance between the centroid coordinates and the center - point image coordinates , draw a curve with and as independent variables and as the dependent variable. Please refer to Figure 7 and Figure 8 , Figure 7 is a schematic diagram of a simulation curve provided by an embodiment of the present application, Figure 8 is a schematic diagram of another simulation curve provided by an embodiment of the present application.

[0204] Among them, the simulation results show that as increases, sharply decreases and then approaches zero, showing a negative - power - function law; as increases, Gradually increasing with an increasing growth rate, showing a positive power function growth pattern. Under the default simulation conditions, when the distance in the optical axis direction from the camera to the measurement point reaches more than 4000 mm, that is, when the deviation coefficient is within 12500, the deviation between the centroid coordinates and the center-of-circle image point coordinates can be controlled at the sub-pixel level. Therefore, it can be considered that when the camera is far enough from the marker (similarly, when the equivalent focal length of the camera and the radius of the marker point are small enough), and when the deviation coefficient is small enough, it is advisable to use the centroid scale instead of the center-of-circle scale to obtain the scale factor.

[0205] It can be seen that in this example, the scale factor can be calibrated through the centroid scale of the marker point, which is beneficial to improving the accuracy of scale factor calibration.

[0206] Please refer to Figure 9 , Figure 9 which is a schematic structural diagram of an electronic device provided by an embodiment of the present application, applied to a processing device in an image measurement system. The image measurement system includes the processing device, a camera platform, a marker point, and a camera disposed on the camera platform. The processing device is connected to the camera. The marker point includes a first sub-marker and a second sub-marker, and there are scale features and direction features between the first sub-marker and the second sub-marker; as Figure 9 shown, the electronic device includes a processor, a memory, a communication interface, and one or more programs. Among them, the above one or more programs are stored in the above memory, and the above one or more programs are configured with instructions for the above processor to execute the following steps:

[0207] Obtain the marker information of the marker point, where the marker information includes the scale factor information and / or image coordinate information and / or encoded marker information of the marker point in multiple frames of marker point images and / or the first mileage information of the camera platform;

[0208] If it is determined that the marker information includes the scale factor information and the image coordinate information, then according to the scale factor information, determine a first matching relationship between a first scale factor of a first marker point in the current frame of marker point image and a second scale factor of a second marker point in the previous frame of marker point image;

[0209] According to the image coordinate information and the first matching relationship, determine a first target matching relationship between the first marker point in the current frame of marker point image and the second marker point in the previous frame of marker point image;

[0210] According to the first target matching relationship, determine the encoding of the first marker point in the current frame of marker point image in terms of time sequence.

[0211] It can be seen that in the embodiment of the present application, the electronic device can first obtain the marker information of the marker points. The marker information includes the scale factor information and / or the image coordinate information and / or the encoded marker information and / or the first odometry information of the camera platform in multiple frames of marker point images. Then, if it is determined that the marker information includes the scale factor information and the image coordinate information, according to the scale factor, a first matching relationship between the first scale factor of the first marker point in the current frame marker point image and the second scale factor of the second marker point in the previous frame marker point image is determined. Then, according to the image coordinate information and the first matching relationship, a first target matching relationship between the first marker point in the current frame marker point image and the second marker point in the previous frame marker point image is determined. Finally, according to the first target matching relationship, the encoding of the first marker point in the current frame marker point image in time series is determined. It can be realized that when measuring on a mobile camera platform or measuring marker points with a large displacement, by matching the scale factors of the marker points in the image, the matching of the marker points is further realized, and the matching relationship of the scale factors is verified according to the motion constraint by the image coordinate information, which is beneficial to improving the accuracy of image measurement.

[0212] In a possible example, in terms of determining the first matching relationship between the first scale factor of the first marker point in the current frame marker point image and the second scale factor of the second marker point in the previous frame marker point image according to the scale factor information, the above program includes instructions for performing the following steps:

[0213] Perform the following operations on the first scale factor of each first marker point in the current frame marker point image to obtain the first matching relationship:

[0214] Determine the difference between the currently processed first scale factor and each second scale factor to obtain a plurality of differences; determine the absolute values of the plurality of differences to obtain a plurality of target values;

[0215] Determine the target second scale factor corresponding to the smallest value among the plurality of target values;

[0216] If it is determined that the target value corresponding to the target second scale factor is less than the first preset threshold, it is determined that there is a matching relationship between the currently processed first scale factor and the target second scale factor.

[0217] In a possible example, in terms of determining the first target matching relationship between the first marker point in the current frame marker point image and the second marker point in the previous frame marker point image according to the image coordinate information and the first matching relationship, the above program includes instructions for performing the following steps:

[0218] Perform the following operations on each set of mutually matching first and second scale factors in the first matching relationship to determine the first target matching relationship between the first landmark point and the second landmark point:

[0219] Determine the conversion relationship between the first landmark point corresponding to the currently processed first scale factor and the second landmark point corresponding to the currently processed second scale factor according to the image coordinate information;

[0220] Construct a target system of equations according to the conversion relationship;

[0221] Solve the target system of equations to obtain the rotation matrix;

[0222] Perform reprojection error estimation according to the rotation matrix to obtain the target reprojection error;

[0223] If it is determined that the target reprojection error is less than or equal to the sixth preset threshold, it is determined that there is a matching relationship between the first landmark point corresponding to the currently processed first scale factor and the second landmark point corresponding to the currently processed second scale factor.

[0224] In a possible example, after obtaining the landmark information of the landmark points, the above program includes instructions for performing the following steps:

[0225] If it is determined that the landmark information includes the scale factor information, the image coordinate information, and the first odometry information, then according to the second scale factor of the second landmark point in the previous frame landmark point image, the first odometry information, and a preset prediction model, predict the odometry prediction information and the scale factor prediction information of the current frame landmark point image;

[0226] Determine the second matching relationship between the third scale factor in the scale factor prediction information and the first scale factor of the current frame landmark point image according to the scale factor prediction information and the first scale factor of the current frame landmark point image;

[0227] Correct the preset prediction model according to the second matching relationship to obtain the corrected preset prediction model;

[0228] Determine the first distance from each landmark point in multiple frames of landmark point images to the camera optical center according to the scale factor information;

[0229] Determine the second odometry information according to the first distance;

[0230] Determine the first position information of the landmark points in multiple frames of landmark point images according to the second odometry information and the scale factor information;

[0231] Determine the second position information of the first landmark point according to the first mileage information and the scale factor information;

[0232] Match the first position information and the second position information to obtain the second target matching relationship of the landmark point position.

[0233] In a possible example, after obtaining the landmark information of the landmark point, the above program further includes instructions for performing the following steps:

[0234] If it is determined that the landmark information includes the scale factor information, the image coordinate information, and the coding mark information, determine the first target matching relationship according to the scale factor information and the image coordinate information;

[0235] Determine the image quality information of each landmark point;

[0236] Filter out the target landmark points whose image quality information meets the preset image conditions;

[0237] Determine the first absolute coding information of the target landmark point according to the coding mark information;

[0238] Determine the second absolute coding information of the landmark points in the current frame landmark point image according to the first target matching relationship and the first absolute coding information.

[0239] In a possible example, after obtaining the landmark information of the landmark point, the above program further includes instructions for performing the following steps:

[0240] If it is determined that the landmark information includes the scale factor information, the image coordinate information, the first mileage information, and the coding mark information, determine the second target matching relationship according to the scale factor, the image coordinate information, and the first mileage information;

[0241] Determine the second absolute coding information according to the scale factor information, the image coordinate information, and the coding mark information.

[0242] In a possible example, before obtaining the landmark information of the landmark point, the above program further includes instructions for performing the following steps:

[0243] Perform the following operations on each first landmark point to obtain the first scale factor of each first landmark point:

[0244] Determine the physical scale value between the first sub-landmark and the second sub-landmark of the currently processed first landmark point;

[0245] Determine the pixel scale value between the first sub-marker and the second sub-marker of the first marker being currently processed imaged in the current frame marker point image;

[0246] Determine the first scale factor of the first marker being currently processed according to the physical scale value and the pixel scale value.

[0247] In a possible example, after determining the encoding of the first marker in the current frame marker point image in time sequence according to the first target matching relationship, the above program further includes instructions for performing the following steps:

[0248] If it is detected that there is a first target marker in the current frame marker point image that is not matched with the second marker, determine the fourth scale factor with the smallest scale factor value and the fifth scale factor with the largest scale factor value in the previous frame marker point image;

[0249] Determine the difference between the first scale factor corresponding to the first target marker and the fourth scale factor to obtain a first target difference;

[0250] Determine the difference between the first scale factor corresponding to the first target marker and the fifth scale factor to obtain a second target difference;

[0251] If it is determined that the first target difference is less than a second preset threshold or the second target difference is greater than a third preset threshold, determine the encoding of the first target marker according to the appearance order of the first target marker and the existing encoding information of the first marker in the current frame marker point image.

[0252] The above mainly introduces the solution of the embodiment of the present application from the perspective of the execution process on the method side. It can be understood that in order for an electronic device to implement the above functions, it includes the corresponding hardware structure and / or software module for executing each function. Those skilled in the art should easily realize that, combining the units and algorithm steps of each example described in the embodiments provided in this article, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the way of hardware or computer software driving hardware depends on the specific application and design constraint conditions of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0253] The embodiments of the present application can divide the functional units of the electronic device according to the above method examples. For example, each functional unit can be divided corresponding to each function, or two or more functions can be integrated into one processing unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit. It should be noted that the division of units in the embodiments of the present application is illustrative, only a logical function division, and there can be other division methods in actual implementation.

[0254] In the case of dividing each functional module corresponding to each function, Figure 10 is a block diagram of the functional unit composition of a feature point matching device provided by an embodiment of the present application. As Figure 10 shown, the device includes an acquisition unit 1001 and a determination unit 1002; wherein,

[0255] The acquisition unit 1001 is configured to acquire the feature information of the feature points, and the feature information includes the scale factor information and / or the image coordinate information and / or the coding mark information of the feature points in multiple frames of feature point images and / or the first odometry information of the camera platform;

[0256] The determination unit 1002 is configured to, if it is determined that the feature information includes the scale factor information and the image coordinate information, determine a first matching relationship between a first scale factor of a first feature point in the current frame of feature point image and a second scale factor of a second feature point in the previous frame of feature point image according to the scale factor information;

[0257] The determination unit 1002 is further configured to determine a first target matching relationship between the first feature point in the current frame of feature point image and the second feature point in the previous frame of feature point image according to the image coordinate information and the first matching relationship;

[0258] The determination unit 1002 is further configured to determine the coding of the first feature point in the current frame of feature point image in time sequence according to the first target matching relationship.

[0259] It can be seen that in the embodiments of the present application, the landmark matching device may first obtain the landmark information of the landmarks. The landmark information includes the scale factor information and / or the image coordinate information and / or the encoded marker information of the landmarks in multiple frames of landmark images and / or the first odometry information of the camera platform. Then, if it is determined that the landmark information includes the scale factor information and the image coordinate information, according to the scale factor, a first matching relationship between the first scale factor of the first landmark in the current frame landmark image and the second scale factor of the second landmark in the previous frame landmark image is determined. Then, according to the image coordinate information and the first matching relationship, a first target matching relationship between the first landmark in the current frame landmark image and the second landmark in the previous frame landmark image is determined. Finally, according to the first target matching relationship, the encoding of the first landmark in the current frame landmark image in terms of time sequence is determined. It can be realized that when measuring on a mobile camera platform or measuring landmarks with a large displacement, by matching the scale factors of the landmarks in the image, the matching of the landmarks is further realized, and the matching relationship of the scale factors is verified according to the motion constraints by the image coordinate information, which is beneficial to improving the accuracy of image measurement.

[0260] In a possible example, in terms of determining the first matching relationship between the first scale factor of the first landmark in the current frame landmark image and the second scale factor of the second landmark in the previous frame landmark image according to the scale factor information, the determining unit 1002 is specifically configured to:

[0261] Perform the following operations on the first scale factor of each first landmark in the current frame landmark image to obtain the first matching relationship:

[0262] Determine the difference between the currently processed first scale factor and each second scale factor to obtain a plurality of differences; determine the absolute values of the plurality of differences to obtain a plurality of target values;

[0263] Determine the target second scale factor corresponding to the smallest value among the plurality of target values;

[0264] If it is determined that the target value corresponding to the target second scale factor is less than the first preset threshold, determine that there is a matching relationship between the currently processed first scale factor and the target second scale factor.

[0265] In a possible example, in terms of determining the first target matching relationship between the first landmark in the current frame landmark image and the second landmark in the previous frame landmark image according to the image coordinate information and the first matching relationship, the determining unit 1002 is specifically configured to:

[0266] Perform the following operations on each set of mutually matching first and second scale factors in the first matching relationship to determine the first target matching relationship between the first fiducial point and the second fiducial point:

[0267] Determine the conversion relationship between the first fiducial point corresponding to the currently processed first scale factor and the second fiducial point corresponding to the currently processed second scale factor according to the image coordinate information;

[0268] Construct a target equation set according to the conversion relationship;

[0269] Solve the target equation set to obtain a rotation matrix;

[0270] Perform reprojection error estimation according to the rotation matrix to obtain a target reprojection error;

[0271] If it is determined that the target reprojection error is less than or equal to the sixth preset threshold, determine that there is a matching relationship between the first fiducial point corresponding to the currently processed first scale factor and the second fiducial point corresponding to the currently processed second scale factor.

[0272] In a possible example, after obtaining the fiducial information of the fiducial point, the determining unit 1002 is specifically configured to:

[0273] If it is determined that the fiducial information includes the scale factor information, the image coordinate information, and the first mileage information, predict the mileage prediction information and the scale factor prediction information of the current frame fiducial point image according to the second scale factor of the second fiducial point in the previous frame fiducial point image, the first mileage information, and a preset prediction model;

[0274] Determine the second matching relationship between the third scale factor in the scale factor prediction information and the first scale factor of the current frame fiducial point image according to the scale factor prediction information and the first scale factor of the current frame fiducial point image;

[0275] Correct the preset prediction model according to the second matching relationship to obtain a corrected preset prediction model;

[0276] Determine the first distance from each fiducial point in multiple frames of fiducial point images to the camera optical center according to the scale factor information;

[0277] Determine the second mileage information according to the first distance;

[0278] Determine the first position information of the fiducial points in multiple frames of fiducial point images according to the second mileage information and the scale factor information;

[0279] Determine the second position information of the first landmark point according to the first mileage information and the scale factor information;

[0280] Match the first position information and the second position information to obtain a second target matching relationship of the landmark point positions.

[0281] In a possible example, after obtaining the landmark information of the landmark point, the determining unit 1002 is further specifically configured to:

[0282] If it is determined that the landmark information includes the scale factor information, the image coordinate information, and the coding marker information, determine a first target matching relationship according to the scale factor information and the image coordinate information;

[0283] Determine the image quality information of each landmark point;

[0284] Filter out target landmark points whose image quality information meets a preset image condition;

[0285] Determine the first absolute coding information of the target landmark point according to the coding marker information;

[0286] Determine the second absolute coding information of the landmark points in the current frame landmark point image according to the first target matching relationship and the first absolute coding information.

[0287] In a possible example, after obtaining the landmark information of the landmark point, the determining unit 1002 is further specifically configured to:

[0288] If it is determined that the landmark information includes the scale factor information, the image coordinate information, the first mileage information, and the coding marker information, determine a second target matching relationship according to the scale factor, the image coordinate information, and the first mileage information;

[0289] Determine the second absolute coding information according to the scale factor information, the image coordinate information, and the coding marker information.

[0290] In a possible example, before obtaining the landmark information of the landmark point, the determining unit 1002 is further specifically configured to:

[0291] Perform the following operations on each first landmark point to obtain the first scale factor of each first landmark point:

[0292] Determine the physical scale value between the first sub-landmark and the second sub-landmark of the first landmark point being currently processed;

[0293] Determine the pixel scale value between the first sub-marker and the second sub-marker of the first marker being currently processed imaged in the current frame marker point image;

[0294] Determine the first scale factor of the first marker being currently processed according to the physical scale value and the pixel scale value.

[0295] In a possible example, after determining the encoding of the first marker in the current frame marker point image in time sequence according to the first target matching relationship, the determining unit 1002 is further specifically configured to:

[0296] If it is detected that there is a first target marker in the current frame marker point image that is not matched with the second marker, determine the fourth scale factor with the smallest scale factor value and the fifth scale factor with the largest scale factor value in the previous frame marker point image;

[0297] Determine the difference between the first scale factor corresponding to the first target marker and the fourth scale factor to obtain a first target difference;

[0298] Determine the difference between the first scale factor corresponding to the first target marker and the fifth scale factor to obtain a second target difference;

[0299] If it is determined that the first target difference is less than the second preset threshold or the second target difference is greater than the third preset threshold, determine the encoding of the first target marker according to the appearance order of the first target marker and the existing encoding information of the first marker in the current frame marker point image.

[0300] It should be noted that all relevant contents of the steps involved in the above method embodiments can be cited in the function descriptions of the corresponding functional modules, and will not be repeated here.

[0301] The electronic device provided in this embodiment is used to execute the above marker point matching method, and thus can achieve the same effect as the above implementation method.

[0302] In the case of adopting an integrated unit, the electronic device may include a processing module, a storage module, and a communication module. Among them, the processing module can be used to control and manage the actions of the electronic device. For example, it can be used to support the electronic device to execute the steps performed by the above functional units. The storage module can be used to support the electronic device to execute the storage of program codes and data, etc. The communication module can be used to support the communication of the electronic device with other devices.

[0303] Among them, the processing module can be a processor or a controller. It can implement or execute various exemplary logic blocks, modules, and circuits described in connection with the disclosure of this application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, and so on. The storage module can be a memory. The communication module can specifically be a device that interacts with other electronic devices, such as a radio frequency circuit, a Bluetooth chip, a Wi-Fi chip, etc.

[0304] An embodiment of this application also provides a computer storage medium. Among them, this computer storage medium stores a computer program for electronic data exchange, and this computer program enables a computer to execute some or all of the steps of any one of the methods described in the above method embodiments. The above computer includes an electronic device.

[0305] An embodiment of this application also provides a computer program product. The above computer program product includes a non-transitory computer-readable storage medium storing a computer program, and the above computer program is operable to enable a computer to execute some or all of the steps of any one of the methods described in the above method embodiments. This computer program product can be a software installation package, and the above computer includes a control platform.

[0306] It should be noted that for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that this application is not limited by the described action sequence, because according to this application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0307] In the above embodiments, the descriptions of each embodiment have their own emphases. For the parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0308] In several embodiments provided by this application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the above unit division is only a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some interfaces. The indirect coupling or communication connection of the device or unit can be in an electrical or other form.

[0309] The units described above as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed over multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0310] In addition, the functional units in the various embodiments of this application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0311] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the above methods in the various embodiments of this application. The aforementioned memory includes: USB flash drives, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), mobile hard disks, magnetic disks, or optical discs and other media that can store program codes.

[0312] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing relevant hardware through a program. This program can be stored in a computer-readable memory, and the memory can include: flash drives, read-only memories, random access memories, magnetic disks, or optical discs, etc.

[0313] The above has introduced the embodiments of this application in detail. Specific examples are used in this article to elaborate on the principle and implementation manner of this application. The description of the above embodiments is only used to help understand the method and its core idea of this application; at the same time, for those of ordinary skill in the art, according to the idea of this application, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to this application.

Claims

1. A landmark point matching method, characterized in that: A processing device applied to an image measurement system, the image measurement system comprising the processing device, a camera platform, a marker point and a camera arranged on the camera platform, the processing device being connected to the camera, the marker point comprising a first sub-marker and a second sub-marker, the first sub-marker and the second sub-marker having a scale feature and a direction feature; the method comprising: Acquire marker information of the marker point, wherein the marker information includes scale factor information and / or image coordinate information and / or coding mark information of the marker point in multiple frames of marker point images and / or first mileage information of the camera platform; If it is determined that the mark information includes the scale factor information and the image coordinate information, determining a first matching relationship between a first scale factor of a first mark point in a mark point image of a current frame and a second scale factor of a second mark point in a mark point image of a previous frame according to the scale factor information; Determine a first target matching relationship between a first marker point in the marker point image of the current frame and a second marker point in the marker point image of the previous frame according to the image coordinate information and the first matching relationship; Determine, according to the first target matching relationship, a temporal encoding of a first marker point in the marker point image of the current frame; The determining, based on the scale factor information, a first matching relationship between a first scale factor of a first marker point in a current frame marker point image and a second scale factor of a second marker point in a previous frame marker point image comprises: performing the following operations on the first scale factor of each first marker point in the current frame marker point image to obtain the first matching relationship: determining a difference between a currently processed first scale factor and each second scale factor to obtain a plurality of differences; determining an absolute value of the plurality of differences to obtain a plurality of target values; determining a target second scale factor corresponding to a minimum value among the plurality of target values; and if it is determined that a target value corresponding to the target second scale factor is less than a first preset threshold, determining that a matching relationship exists between the currently processed first scale factor and the target second scale factor.

2. The method according to claim 1, characterized in that The determining, according to the image coordinate information and the first matching relationship, a first target matching relationship between a first marker point in the marker point image of the current frame and a second marker point in the marker point image of the previous frame includes: Perform the following operation on each set of mutually matching first proportional factors and second proportional factors in the first matching relationship to determine a first target matching relationship between the first marker point and the second marker point: Determine, according to the image coordinate information, a conversion relationship between a first marker point corresponding to a first scale factor currently being processed and a second marker point corresponding to a second scale factor currently being processed; According to the conversion relationship, construct a target equation group; Solving the target equation group to obtain a rotation matrix; According to the rotation matrix, a reprojection error is estimated to obtain a target reprojection error; If it is determined that the target reprojection error is less than or equal to the sixth preset threshold, it is determined that a first marker point corresponding to the currently processed first scale factor and a second marker point corresponding to the currently processed second scale factor have a matching relationship.

3. The method according to claim 1, characterized in that After acquiring the mark information of the mark point, the method further includes: If it is determined that the marker information includes the scale factor information, the image coordinate information and the first mileage information, predicting the mileage prediction information and the scale factor prediction information of the marker point image of the current frame according to the second scale factor of the second marker point in the marker point image of the previous frame, the first mileage information and a preset prediction model; Determine, according to the scale factor prediction information and the first scale factor of the marker point image of the current frame, a second matching relationship between a third scale factor in the scale factor prediction information and the first scale factor of the marker point image of the current frame; Correcting the preset prediction model according to the second matching relationship to obtain a corrected preset prediction model; Determine, according to the scale factor information, a first distance from each marker point in the multiple-frame marker point images to the optical center of the camera; Determining second mileage information according to the first distance; Determine first position information of the marker point in the multiple-frame marker point image according to the second mileage information and the scale factor information; Determine second position information of the first marker point according to the first mileage information and the scale factor information; The first position information and the second position information are matched to obtain a second target matching relationship of the marker point position.

4. The method according to claim 1, characterized in that: After acquiring the mark information of the mark point, the method further includes: If it is determined that the mark information includes the scale factor information, the image coordinate information and the encoding mark information, determining a first target matching relationship according to the scale factor information and the image coordinate information; Determine image quality information for each landmark point; Filter out target landmark points whose image quality information meets preset image conditions; Determining first absolute coding information of the target marker point according to the coding mark information; According to the first target matching relationship and the first absolute coding information, second absolute coding information of the marker point in the marker point image of the current frame is determined.

5. The method according to claim 1, characterized in that After acquiring the mark information of the mark point, the method further includes: If it is determined that the mark information includes the scale factor information, the image coordinate information, the first mileage information and the coded mark information, determining a second target matching relationship according to the scale factor, the image coordinate information and the first mileage information; The second absolute coding information is determined according to the scale factor information, the image coordinate information and the coding mark information.

6. The method according to claim 1, characterized in that Before acquiring the mark information of the mark point, the method further includes: The following operation is performed on each first marker point to obtain the first scale factor of each first marker point: Determine a physical dimension value between a first sub-marker and a second sub-marker of a first marker point currently being processed; Determine a pixel scale value between a first sub-marker and a second sub-marker of the first marker point currently being processed imaged in the marker point image of the current frame; A first scale factor of the first marker point currently being processed is determined according to the physical scale value and the pixel scale value.

7. The method according to claim 1, characterized in that After determining the temporal encoding of the first marker point in the marker point image of the current frame according to the first target matching relationship, the method further includes: If it is detected that there is a first target marker point that does not match the second marker point in the marker point image of the current frame, determine a fourth scale factor with the smallest scale factor value and a fifth scale factor with the largest scale factor value in the marker point image of the previous frame; Determine a difference between a first scale factor corresponding to the first target landmark point and the fourth scale factor to obtain a first target difference; Determine a difference between a first scale factor corresponding to the first target landmark and the fifth scale factor to obtain a second target difference; If it is determined that the first target difference is less than the second preset threshold or the second target difference is greater than the third preset threshold, the encoding of the first target marker point is determined based on the order of appearance of the first target marker points and the existing encoding information of the first marker point in the current frame marker point image.

8. A landmark point matching device, characterized in that: A processing device applied to an image measurement system, the image measurement system comprising the processing device, a camera platform, a marker point and a camera arranged on the camera platform, the processing device is connected to the camera, the marker point comprises a first sub-marker and a second sub-marker, the first sub-marker and the second sub-marker have a scale feature and a direction feature; the marker point matching device comprises an acquisition unit and a determination unit, wherein, The acquisition unit is used to acquire the mark information of the mark point, wherein the mark information includes the scale factor information and / or the image coordinate information and / or the coding mark information and / or the first mileage information of the camera platform of the mark point in the multiple frames of the mark point image; The determining unit is used to determine, if it is determined that the mark information includes the scale factor information and the image coordinate information, a first matching relationship between a first scale factor of a first mark point in a mark point image of a current frame and a second scale factor of a second mark point in a mark point image of a previous frame according to the scale factor information, including: performing the following operations on the first scale factor of each first mark point in the mark point image of the current frame to obtain the first matching relationship: determining the difference between the currently processed first scale factor and each second scale factor to obtain a plurality of differences; determining the absolute values ​​of the plurality of differences to obtain a plurality of target values; determining a target second scale factor corresponding to the smallest value among the plurality of target values; and determining that a matching relationship exists between the currently processed first scale factor and the target second scale factor if it is determined that the target value corresponding to the target second scale factor is less than a first preset threshold value; The determining unit is further configured to determine a first target matching relationship between a first marker point in the marker point image of the current frame and a second marker point in the marker point image of the previous frame according to the image coordinate information and the first matching relationship; The determining unit is further configured to determine the temporal encoding of the first marker point in the marker point image of the current frame according to the first target matching relationship.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory is used to store one or more programs and is configured to be executed by the processor, wherein the program comprises instructions for executing the steps in the method according to any one of claims 1 to 6.

Images