Speed ​​measurement method and speed measurement device using multiple cameras

The use of multiple cameras with synchronized image sequences and position transformation matrices addresses the inaccuracy of wheel encoder-based speed measurement in mobile robots, ensuring precise and environment-aware speed estimation.

JP7580614B2Active Publication Date: 2024-11-11SHANGHAI AIFANSHIYU TECH CO LTD
View PDF 15 Cites 0 Cited by

Patent Information

Application Number
JP2023540732
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2021-06-30
Filing Date
2022-03-22
Publication Date
2024-11-11
Estimated Expiration
2042-03-22

AI Technical Summary

Technical Problem

Current methods for measuring the speed of mobile robots using wheel encoders are inaccurate due to lack of environmental feedback, leading to errors in self-position and operation estimation, especially when wheels slip or float, compromising safety and stability.

Method used

A speed measurement method using multiple cameras with fixed relative positions, synchronized image sequences, and image matching to calculate position transformation matrices, enabling accurate speed measurement independent of external influences.

Benefits of technology

The method provides highly accurate speed measurement by ensuring feedback with the real environment, preventing incorrect judgments even when wheels slip or float, and enhancing the precision of robot operation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007580614000018
    Figure 0007580614000018
  • Figure 0007580614000019
    Figure 0007580614000019
  • Figure 0007580614000020
    Figure 0007580614000020
Patent Text Reader

Abstract

The present invention provides a method and device for measuring speed using multiple cameras, including the steps of: S1: calibrating the position of each camera in a camera set with a fixed relative position installed on a moving platform; S2: recording the synchronous image sequence and time stamp of each camera in the camera set; and S3: processing the matching of images taken by each camera in the camera set, and if the matching result of the two images obtained is greater than a first threshold, calculating the position transformation matrix of the two images, obtaining the position transformation matrix of the moving platform based on the positional relationship of the camera taking the two images with respect to the moving platform, and finally calculating the speed information of the moving platform based on the position transformation matrix and the shooting time interval of the two images. By processing the matching of images taken by different cameras to calculate the speed of the moving platform, it is possible to solve the contradiction between the shooting distance and the image overlap degree when measuring the speed with a single camera, and obtain the moving speed information with high accuracy.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to the field of speed measurement technology, and more particularly to a speed measurement method and device using multiple cameras. [Background technology]

[0002] With the continuous development of information electronics, automatic control and other technologies, robots have already been involved in people's production and life. Among them, mobile robots with functions such as movement, environment recognition and transportation are representative and widely used in various industries. Not only household cleaning robots, but also AGVs (automated guided vehicles), smart forklifts, and service robots for hotels and hospitals, which are widely used in industry, are concrete examples of mobile robot technology.

[0003] In order to control a mobile robot well, it is a technical problem to obtain the speed of the robot itself in real time. Currently, most mobile robots use rotary encoders to measure the speed of the left and right wheels. However, this method is essentially unable to form feedback with the real environment. Therefore, if the wheels slip or the wheels are not on the ground but floating, the mobile robot's speed cannot be measured accurately, which causes errors in estimating its own position and driving situation, and ultimately leads to unknown errors. This is a major challenge in the safety and stability of the robot's operation. Summary of the Invention [Problem to be solved by the invention]

[0004] An object of the present invention is to provide a speed measurement method and device using multiple cameras that is accurate in measuring speed and is not subject to external influences. [Means for solving the problem]

[0005] In order to achieve the above object, the present invention provides S1: A step of calibrating the position of each camera in a camera set with a fixed relative position provided on a moving platform to obtain a positional relationship between each camera and a positional relationship between each camera and the moving platform; S2: acquiring a synchronized image sequence for each camera in the camera set and recording a timestamp of the synchronized image sequence; and S3: processing a matching process for images taken by each camera in the camera set, and if the matching result of the obtained two images is greater than a first threshold, calculating a position transformation matrix of the two images, calculating a position transformation matrix of the moving platform during the shooting time interval of the two images based on the positional relationship between the cameras taking the two images, the positional relationship of the cameras taking the two images relative to the moving platform, and the position transformation matrices of the two images, and finally calculating speed information of the moving platform based on the position transformation matrix of the moving platform and the shooting time interval of the two images.

[0006] Preferably, step S3 specifically includes the steps of: S31: acquiring position information of the moving platform at a current time, and calculating a first position of the first camera at the current time based on the position information of the moving platform at the current time and a positional relationship of a first camera with respect to the moving platform; S32: based on the first position, traverse a second position when another camera captures an image, and if the distance between the second position and the first position is less than a second threshold, determine that the search is successful, record the captured images corresponding to the first position and the second position as a search image and a candidate image, respectively, record the timestamps of the search image and the candidate image, and then proceed to step S33; S33: performing matching between the search image and the candidate image, and if the obtained matching result is greater than the first threshold, determining that the matching is successful, calculating the position transformation matrix between the search image and the candidate image, and then proceeding to step S34; and S34: calculating a position transformation matrix of the first camera based on the positional relationship between the cameras of the search image and the candidate image and the position transformation matrix, then calculating a position transformation matrix of the moving platform based on the position transformation matrix of the first camera and the positional relationship of the first camera with respect to the moving platform, and finally calculating speed information of the moving platform based on the position transformation matrix of the moving platform and the shooting time interval of the two images; Includes.

[0007] Preferably, the step of acquiring position information of the moving platform at a current time comprises: The method includes a step of matching a search image of the first camera with a previous frame image to obtain a self-position transformation matrix of the first camera, and obtaining position information of the moving platform based on the self-position transformation matrix and the positional relationship of the first camera with respect to the moving platform.

[0008] Preferably, the search image of the first camera is matched with a previous frame image to obtain a self-location transformation matrix of the first camera, and when obtaining position information of the mobile platform based on the self-location transformation matrix and the positional relationship of the first camera with respect to the mobile platform, a weight corresponding to each camera is set based on the matching result of each camera, and the position information of the mobile platform is calculated using a weighted average method.

[0009] Preferably, the matching method is a 2D image feature matching method, which calculates a position transformation matrix of two images by extracting feature points of two images and quickly matching them.

[0010] Preferably, the matching method is a 2D image frequency domain information matching method, which calculates the rotation, translation and scale factors of the two images through the Fourier-Mellin transform algorithm, and further calculates the position transformation matrix of the two images.

[0011] Preferably, the matching method is a 3D matching method, which calculates a position, orientation and velocity of three degrees of freedom in space through 3D point coordinates corresponding to two images, and further calculates a position transformation matrix of the two images.

[0012] Preferably, the position information of the mobile platform at the current time is obtained directly via a position sensor.

[0013] Preferably, when performing step S32 or S33, if the search or matching fails, the process proceeds to step S35, which specifically includes the steps of:

[0014] The method includes a step of obtaining position information of the moving platform at the current time, position information at a previous time, and a shooting time interval between the current time and the previous time, and estimating a speed of the camera set relative to the object.

[0015] In order to achieve the above object, the present invention provides a moving platform movable relative to the subject; a camera set including at least two cameras mounted on the moving platform, the relative positions of the cameras being fixed; A processor arranged to perform the multiple camera speed measurement method as described above.

[0016] Preferably, the camera set includes a light source that matches the camera; a camera equipped with a near-infrared light source and a lens that matches the near-infrared light source and is capable of transmitting the wavelength of the light emitted by the near-infrared light source; Or, a far-infrared light source and a matching far-infrared camera, Or an ultraviolet light source and a matching ultraviolet camera, Alternatively, it includes a polarized light source and a camera equipped with a lens that matches the polarized light and can transmit the polarized light.

[0017] Preferably, the image capture times of each camera are synchronized or each camera's capture time is time stamped.

[0018] Preferably, the multiple camera speed measurement device further includes a calibration tool provided on the moving platform, the calibration tool calibrating the relative positional relationship between each camera and the positional relationship of each camera to the moving platform.

[0019] Preferably, the calibration tool is a checkerboard, a laser range finder, a laser radar, a TOF sensor, or an encoder.

[0020] Preferably, the camera set includes at least a pair of binocular cameras.

[0021] Preferably, the camera set includes at least two TOF cameras positioned along a main direction of movement of said moving platform.

[0022] Preferably, the camera set includes at least a pair of line scan cameras positioned along a main direction of movement of the moving platform.

[0023] Preferably, the camera set further includes at least one global shutter area camera, and the processor is adapted to compare an image captured by the global shutter area camera with an image captured by the line scan camera and correct the image captured by the line scan camera.

[0024] Preferably, if the matching result of the two acquired images is less than a first threshold, the processor can estimate the velocity of the camera set relative to the subject based on the position information of the moving platform at the current time, the position information at the previous time, and the shooting time interval between the current time and the previous time.

[0025] Preferably, the multiple camera speed measurement device further includes at least one laser beam measuring device for measuring distances between different objects.

[0026] Preferably, the laser beam meter is a line laser beam meter or a cross-hair laser beam meter.

[0027] The present invention provides a speed measurement method and a speed measurement device using multiple cameras, which obtain two images with high similarity by processing matching of images taken by different cameras. Since the measurement distance between the two images is the shortest and the overlapping degree of the images is the largest, the accuracy of the speed information of the moving platform calculated by the speed measurement method using multiple cameras is higher than the accuracy of the speed information of the object calculated by continuous image tracking using a single camera. In addition, the speed measurement method and the speed measurement device using multiple cameras according to the present invention form a feedback with the real environment to solve the problem of the robot making an erroneous judgment when the wheels slip or the wheels are tilted. [Brief description of the drawings]

[0028] [Figure 1] FIG. 2 is a step diagram of a speed measurement method using multiple cameras according to an embodiment of the present invention. [Diagram 2] 2 is a flowchart of a speed measurement method using multiple cameras according to an embodiment of the present invention. [Diagram 3] 1 is a diagram showing the structure of a speed measurement device using multiple cameras according to an embodiment of the present invention; [Figure 4]FIG. 2 is a diagram showing a search image and a candidate image according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0029] Those of ordinary skill in the art should understand that the accompanying drawings are for a better understanding of the present invention, and are not intended to limit the scope of the present invention. As described in the Background Art, most mobile robots currently use a method of measuring the speed of the left and right wheels using rotary encoders. However, this method is essentially unable to form feedback with the real environment. Therefore, when the wheels slip or the wheels are not in contact with the ground but floating, the mobile robot cannot accurately measure its speed, which causes errors in estimating its own position and driving situation, ultimately leading to unknown errors. This poses a major challenge to the safety and stability of the robot's operation.

[0030] It is known to be a very effective method to point a camera at the ground and calculate the camera's moving speed from a video of the ground image. However, the accuracy of the speed measurement is determined by the shooting position distance for comparing the two calculated images and the degree of overlap of the images, so how to reduce the shooting distance of the two images and improve the degree of overlap of the two images are important factors in improving the performance of visual speed measurement. Since the visible area of ​​a single camera is limited, the further away the shooting locations of the two images taken by this camera are, the lower the degree of overlap of the two images will be, which will further affect the accuracy of visual speed measurement.

[0031] Based on this, the present invention provides a multiple camera speed measurement method and speed measurement device that captures images using at least two cameras with fixed relative positions and compares the images from the different cameras to obtain highly accurate moving speed information.

[0032] In order to make the objects, advantages and features of the present invention clearer, the present invention will be described in more detail below in combination with the accompanying drawings and specific embodiments. Please note that the drawings are provided in a very simplified form and are not necessarily drawn to scale, in order to more conveniently and clearly describe the embodiments of the present invention. The structures shown in the drawings are usually part of the actual structures. In particular, each drawing may be drawn to a different scale, since the main points to be disclosed are different.

[0033] As used herein, the singular terms "a", "one" and "the" include plural referents unless expressly indicated otherwise. For example, as used herein, the term "or" is generally used in the sense of including "and / or" unless expressly indicated otherwise. For example, as used herein, the term "several" is generally used in the sense of "at least one" unless expressly indicated otherwise. For example, as used herein, the term "at least two" is generally used in the sense of "two or more" unless expressly indicated otherwise. Furthermore, the use of the terms "first", "second" and "third" is for illustrative purposes only and should not be construed as indicating or suggesting a relative importance or as implicitly indicating a quantity of the technical features described. Thus, a feature qualified with "first", "second" and "third" will explicitly or implicitly indicate the presence of one or at least two of the feature.

[0034] The technical idea of ​​the present invention is to provide a speed measurement method and device using multiple cameras that solves the problem of authenticity and accuracy of speed measurement in current robots.

[0035] The speed measurement method using multiple cameras includes: S1: A step of calibrating the position of each camera in a camera set with a fixed relative position provided on a moving platform to obtain a positional relationship between each camera and a positional relationship between each camera and the moving platform; S2: acquiring a synchronized image sequence for each camera in the camera set and recording a timestamp of the synchronized image sequence; and S3: Processing matching of images taken by each camera in the camera set, and if the matching result of the obtained two images is greater than a first threshold, calculating position transformation matrices of the two images, obtaining a position transformation matrix of the moving platform during the shooting time interval of the two images based on the positional relationship between the cameras taking the two images, the positional relationship of the cameras taking the two images relative to the moving platform, and the position transformation matrices of the two images, and finally calculating speed information of the moving platform based on the position transformation matrix of the moving platform and the shooting time interval of the two images;

[0036] As another aspect of the present invention, the speed measurement device using multiple cameras comprises: a moving platform movable relative to the subject; a camera set including at least two cameras mounted on the moving platform, the relative positions of the cameras being fixed; and a processor arranged to perform the multiple camera speed measurement method as described above.

[0037] With this arrangement, two highly similar images are obtained by processing the matching of the images taken by each camera, and the measurement distance between the two highly similar images is the shortest and the image overlap is the largest, so that the accuracy of the speed information of the moving platform calculated by the speed measurement method using multiple cameras is higher than the accuracy of the speed information of the object calculated by continuous image tracking using a single camera.In addition, the speed measurement method and the speed measurement device using multiple cameras according to the present invention form a feedback with the real environment to solve the problem of the robot making an erroneous judgment when the wheels slip or the wheels are tilted.

[0038] The following description will be given with reference to the accompanying drawings. As shown in Figures 1 to 3, Figure 1 is a step diagram of a speed measurement method using multiple cameras according to an embodiment of the present invention, Figure 2 is a flowchart of the speed measurement method using multiple cameras according to an embodiment of the present invention, and Figure 3 is a diagram showing the structure of a speed measurement device using multiple cameras according to an embodiment of the present invention.

[0039] In this embodiment, S1: A step of calibrating the position of each camera 102 in a camera set 101 with a fixed relative position provided on a moving platform 111 to obtain a positional relationship between each camera 102 and a positional relationship between each camera 102 and the moving platform 111; S2: acquiring a synchronized image sequence of each camera 102 in the camera set 101 and recording a timestamp of the synchronized image sequence; and S3: processing the matching of images taken by each camera 102 of the camera set 101, and if the matching result of the obtained two images is greater than a first threshold, calculating the position transformation matrix of the two images, and obtaining the position transformation matrix of the moving platform 111 during the shooting time interval of the two images based on the positional relationship between the cameras 102 that took the two images, the position transformation matrix of the two images, and the positional relationship of the camera 102 that took the two images relative to the moving platform 111, and finally calculating speed information of the moving platform 111 based on the position transformation matrix of the moving platform 111 and the shooting time interval of the two images.

[0040] In this embodiment, referring to FIG. 3, the speed measurement device using the multiple cameras 102 includes: a moving platform 111 movable relative to the subject; A camera set 101 including at least two cameras 102 mounted on the moving platform 111, the relative positions of the cameras 102 being fixed; and a processor 110 arranged to perform the multiple camera 102 speed measurement method as described above.

[0041] First, step S1 is performed to calibrate the position of each camera 102 in the camera set 101 to obtain the positional relationship between each camera 102 and the positional relationship between each camera 102 and the moving platform 111. When performing positional calibration, it is first necessary to establish a world coordinate system W and a body coordinate system C of the moving platform 111. The world coordinate system W can have any point as its origin, and the body coordinate system C has the center of the moving platform 111 as its origin. In this embodiment, the coordinate system used is a right-handed coordinate system, in which the forward direction toward the moving platform 111 is the X-axis, the rightward direction toward the moving platform 111 is the Y-axis, and the vertical direction toward the ground is the Z-axis.

[0042] The moving speed of the moving platform 111 only considers translation and rotation in a two-dimensional plane, so the position information is shown only considering the horizontal plane. The positions of the moving platform 111 and each camera 102 in the camera set 101 are expressed in a world coordinate system W, where the position of the moving platform 111 is expressed at its physical center. The position of each camera 102 in the camera set 101 is expressed with the camera 102 at the center.

[0043] Specifically, at time t k If the coordinates of the position of the moving platform 111 in are c(x, y) and the angle is θ, then the position matrix of the moving platform 111 in the world coordinate system W is expressed as follows:

[0044] JPEG0007580614000001.jpg35170

[0045] The position of each camera 102 in the camera set 101 relative to the moving platform 111 is fixed, and the positional relationship of the i-th camera 102 relative to the moving platform 111 is P ciwhere i=1, 2, . . . , M, and M is the number of cameras 102 in the camera set 101.

[0046] JPEG0007580614000002.jpg39170

[0047] Here, θ ci is the angle of the i-th camera 102 with respect to the moving platform 111, and △x ci and △y ci are the coordinates of the i-th camera 102 in the body coordinate system C of the moving platform 111.

[0048] In this embodiment, the relative positional relationship between each camera 102 and the positional relationship of each camera 102 to the moving platform 111 can be calibrated by a checkerboard, a laser range finder, a laser radar, a TOF sensor or an encoder.

[0049] Next, step S2 is performed to acquire a sequence of synchronized images for each camera 102 in the camera set 101 and record the timestamps of the synchronized images. k In the camera set 101, each camera 102 captures an image F i k are obtained synchronously (i = 1, 2, , M), where F i k is the time t k represents image data acquired by the i-th camera 102 in the i-th camera 102 in the camera set 101, and M represents the number of cameras 102 in the camera set 101. k Get the.

[0050] Next, step S3 is executed to process matching of images taken by the cameras 102 in the camera set 101, and if the matching result of the obtained two images is greater than a first threshold, the position transformation matrices of the two images are calculated, and based on the positional relationship between the cameras 102 that took the two images, the position transformation matrices of the two images, and the positional relationship of the camera 102 relative to the moving platform 111, the position transformation matrix of the moving platform 111 during the shooting time interval of the two images is obtained, and finally, the speed information of the moving platform 111 is calculated based on the position change matrix of the moving platform 111 and the shooting time interval of the two images.

[0051] Furthermore, in step S3, S31: obtaining position information of the moving platform 111 at a current time, and calculating a first position of the first camera 102 at the current time based on the position information of the moving platform 111 at the current time and a positional relationship of the first camera 102 to the moving platform 111; S32: based on the first position, traverse a second position when another camera 102 captures an image, and if the distance between the second position and the first position is less than a second threshold, determine that the search is successful, record the captured images corresponding to the first position and the second position as a search image and a candidate image, respectively, record the timestamps of the search image and the candidate image, and then proceed to step S33; S33: performing matching between the search image and the candidate image, and if the obtained matching result is greater than the first threshold, determining that the matching is successful, calculating the position transformation matrix between the search image and the candidate image, and then proceeding to step S34; and S34: Calculating a position transformation matrix of the first camera 102 based on the positional relationship between the cameras 101 of the search image and the candidate image and the position transformation matrix, then calculating a position transformation matrix of the moving platform 111 based on the position transformation matrix of the first camera 102 and the positional relationship of the first camera 102 to the moving platform 111, and finally calculating speed information of the moving platform 111 based on the position transformation matrix of the moving platform 111 and the shooting time interval of the two images;

[0052] Specifically, after executing step S2, step S31 is subsequently executed to obtain the position information of the moving platform 111 at the current time, and store the search image and the corresponding position information in each camera 102.

[0053] Furthermore, the step of acquiring the position information of the moving platform 111 at the current time specifically includes: The method includes a step of matching the search image of the first camera 102 with a previous frame image to obtain a self-position transformation matrix of the first camera 102, and obtaining position information of the moving platform 111 based on the self-position transformation matrix and the positional relationship of the first camera 102 with respect to the moving platform 111.

[0054] That is, the position of the moving platform 111 can be estimated by each camera 102 in the camera set 101. For example, for each camera 102 in the camera set 101, at time t k An image F taken by a camera 102 at i k After acquiring the search image F i k and the previous frame image F i k-1 is matched and calculated to obtain the self-location transformation matrix of the camera 102.

[0055] As a first preferred example of this embodiment, the matching method is a 2D image feature matching method, which extracts feature points of two images and quickly matches them to obtain a position transformation matrix P i k,k-1 Calculate P i k,k-1 is the time t k-1 and t k represents the position transformation matrix of the i-th camera 102 in

[0056] As a second preferred embodiment of the present invention, the matching method is a 2D image frequency domain information matching method, which calculates the rotation, translation and scale factors of two images through a Fourier-Mellin transform algorithm, and further calculates a position transformation matrix P i k,k-1 Get the.

[0057] As a third preferred embodiment of the present invention, the matching method is a 3D matching method, which calculates the position, orientation, and velocity of three degrees of freedom in space through the 3D point coordinates corresponding to two images, and further calculates a position transformation matrix P i k,k-1 For example, if the i-th camera 102 is a TOF camera 102, or a laser range finder, or a single-line laser radar, or a multi-line laser radar device, the image data includes depth information, and a 3D matching method can be used to estimate the current position of the i-th camera 102. The iterative closest point (ICP) method is used to calculate a three-degree-of-freedom position and orientation transformation in space through the 3D point coordinates corresponding to the two images, or the PnP algorithm is used to calculate the position transformation matrix P i k,k-1 Get the.

[0058] The i-th camera 102 is k The position transformation matrix P of the front and rear frame images calculated byi k,k-1 is the position transformation relationship P of the moving platform 111 c k,k-1 and by rigid body transformation, the following position transformation relationship is obtained:

[0059] JPEG0007580614000003.jpg31170

[0060] P ci -1 , P ci is the inverse matrix of P ci is the positional relationship of the i-th camera 102 with respect to the known moving platform 111, so the position transformation matrix P i k,k-1 The position transformation matrix P of the moving platform 111 is expressed as c k,k-1 can be calculated directly.

[0061] time t k Position P of the moving platform 111 at c k is the time t k-1 Position P of the moving platform 111 at c k-1 and the position transformation matrix P c k,k-1 The calculation formula is as follows:

[0062] JPEG0007580614000004.jpg27170

[0063] For each camera 102, at time t k Position P of the moving platform 111 at c k Based on the positional relationship of the i-th camera 102 with respect to the moving platform 111, at time t k The position of the camera 102 in the image is obtained by the following calculation formula:

[0064] JPEG0007580614000005.jpg30170

[0065] Each camera 102 captures a search image F i k Save the time t k The search image F i k The position of the camera 102 corresponding to i k It is.

[0066] In this embodiment, the position information of the moving platform 111 at the current time can also be directly obtained through a position sensor, which includes but is not limited to an encoder, an inertial element or an odometer.

[0067] Furthermore, the search image of the first camera 102 is matched with the previous frame image to obtain the self-location transformation matrix of the first camera 102, and when calculating the position information of the moving platform 111 based on the self-location transformation matrix and the positional relationship of the first camera 102 with respect to the moving platform 111, a weight corresponding to each camera 102 is set based on the matching result of each camera 102, and the position information of the moving platform 111 is calculated using a weighted average method.

[0068] That is, using the i-th camera 102 in the camera set 101, the position P of one moving platform 111 is calculated based on the matching results of the front and rear frame images obtained by equations (3) and (4) and the position transformation matrix. c k In order to improve the accuracy of the position of the moving platform 111, the calculation results obtained by each camera 102 (e.g., the position P c k ), where the weights of the weighted average are determined by the matching results of each camera 102.

[0069] time t k The matching results of each camera 102 in iThen, the corresponding P of each camera 102 c k The weights of are as follows:

[0070] JPEG0007580614000006.jpg38170

[0071] where j goes from 1 to M.

[0072] time t k The position of the mobile platform 111 at is calculated as follows:

[0073] JPEG0007580614000007.jpg36170

[0074] The weighted average method provides a more accurate position of the mobile platform 111 .

[0075] After storing the search image and the corresponding position information in each camera 102, step S32 is executed, and each camera 102 stores the current position P i k Based on this, we search for and select candidate images of other cameras 102 at the current position at past times. k Position P of each camera 102 of the camera set 101 at i k is the position P of all candidate images of other cameras. l k’ Traverse P l k’ is time t k’ is the position of the l-th camera 102 (l ≠ i) in i k and the closest P l k’ If satisfies the following relationship,

[0076] JPEG0007580614000008.jpg30170

[0077] where D is the second threshold, i.e., the predetermined distance threshold.

[0078] Corresponding image F l k’ is a candidate image of the l-th camera 102, and the timestamp t k’ Record the following.

[0079] After the search is successful, step S33 is executed to match the search image with the candidate image.

[0080] If the matching result satisfies the following relationship:

[0081] JPEG0007580614000009.jpg26170

[0082] S i,l k,k’ is the search image F i k and candidate image F l k’ is the matching score of , and ε is a first threshold, i.e., a pre-defined score threshold.

[0083] If the obtained matching result is greater than the first threshold, it is determined that the matching is successful, and a position transformation matrix between the search image and the candidate image is calculated.

[0084] According to equation (4), the position transformation matrix from which the search image and the candidate image are obtained satisfies the following relationship.

[0085] JPEG0007580614000010.jpg31170

[0086] If there is a candidate image that has been successfully matched, the process proceeds to step S34. In step S34, a position transformation matrix of the first camera 102 is calculated based on the positional relationship between the cameras 102 of the search image and the candidate image and the position transformation matrices of the search image and the candidate image, then a position transformation matrix of the moving platform 111 is calculated based on the position transformation matrix of the first camera 102 and the positional relationship of the first camera 102 with respect to the moving platform 111, and finally, speed information of the moving platform 111 is calculated based on the position transformation matrix of the moving platform 111 and the shooting time interval of the two images.

[0087] Referring to FIG. 4, FIG. 4 is a diagram showing a search image and a candidate image according to an embodiment of the present invention, in which the positional relationship between the cameras 102 corresponding to the search image and the candidate image is fixed.

[0088] time t k and t k’ The position transformation matrix P of the i-th camera 102 in i k,k’ satisfies:

[0089] JPEG0007580614000011.jpg25170

[0090] According to equation (10), the search image F i k and its candidate image F l k’ Position deviation P between i,l k,k’ The relative positional relationship P li is known (where, at time t k’ The position P of the l-th camera 102 in l k’ is also known), and P i k,k’ can be calculated.

[0091] Furthermore, equation (3)Pc k,k’ =P ci ×P i k,k’ ×P ci -1 The time t calculated by the i-th camera 102 is k and time t k’ The position change of the moving platform 111 at time t k A search image F taken by the i-th camera 102 is i k and a candidate image F taken by the l-th camera 102. l k’ Since the measurement distance is the shortest and the image overlap is the largest, P i,m k,k’ The calculation accuracy of is higher than that of continuous image tracking using a single camera 102, and further, P i k,k’ and P c k,k’ To improve the calculation accuracy of

[0092] P c k,k’ When decomposed into a matrix form using equation (1),

[0093] JPEG0007580614000012.jpg36170

[0094] time t k The moving speed V of the moving platform 111 calculated by the i-th camera 102 is i is expressed as follows:

[0095] JPEG0007580614000013.jpg33170

[0096] The moving speed V of the moving platform 111 i is the velocity of the camera set 101 relative to the object.

[0097] For the same reason, the matching weight in equation (6) can be used to obtain a more accurate moving speed. The equation is as follows:

[0098] JPEG0007580614000014.jpg36170

[0099] Furthermore, when performing step S32 or S33, if the search or matching fails, proceed to step S35, which specifically includes the following steps: The step includes obtaining position information of the moving platform 111 at the current time, position information at the previous time, and the shooting time interval (shooting time difference) between the current time and the previous time, and estimating the speed of the camera set 101 relative to the subject.

[0100] That is, if all the cameras 102 in the camera set 101 cannot find a candidate image, i.e., if none of the cameras 102 can find a candidate image that satisfies the formula (8), it is determined that the search has failed, and the process proceeds to step S35. Otherwise, it is determined that the search has succeeded, and the process proceeds to step S33. Similarly, if all the candidate images fail to match, the process proceeds to step S35. The position information P c k , previous position information P c k-1 and the time difference between the current time and the previous time (t k -t k-1 ) can be obtained, and further, the position transformation matrix of the moving platform 111 at two adjacent times can be obtained, and the formula is specifically as follows:

[0101] JPEG0007580614000015.jpg33170

[0102] Of course, by equation (3), P c k,k-1 It is also possible to obtain the above P c k and P c k-1 can also be calculated using equation (7).

[0103] Pc k,k-1 Decomposing into matrix form, JPEG0007580614000016.jpg36170

[0104] Finally, at time t k The moving speed of the moving platform 111 at is obtained and expressed as:

[0105] JPEG0007580614000017.jpg33170

[0106] In an embodiment of the present invention, by processing the matching of images taken by each camera 102, two images with high similarity are obtained, and the two images have the shortest measurement distance and the largest image overlap, so that the accuracy of the calculated velocity information of the moving platform 111 is higher than the accuracy of the velocity information of the subject obtained by continuous image tracking by a single camera 102.

[0107] Based on this, and referring to FIG. 2, an embodiment of the present invention includes: a moving platform 111 movable relative to the subject; A camera set 101 including at least two cameras 102 mounted on the moving platform 111, the relative positions of the cameras 102 being fixed; There is further provided an apparatus for measuring speed using multiple cameras 102, the apparatus including: a processor 110 arranged to execute the method for measuring speed using multiple cameras 102 as described above.

[0108] In this embodiment, the camera set 101 includes a light source 103 that matches the camera 102, A camera 102 equipped with a near-infrared light source 103 and a lens that matches the near-infrared light source 103 and can transmit the wavelength of the light emitted by the near-infrared light source 103; Alternatively, a far-infrared light source 103 and a matching far-infrared camera 102; Or, an ultraviolet light source 103 and a matching ultraviolet camera 102; Alternatively, it includes a polarized light source 103 and a camera 102 equipped with a matching lens capable of transmitting the polarized light.

[0109] In this embodiment, the image capture times of each camera 102 are synchronized, or the capture times of each camera 102 have a time stamp, which can be obtained in real time by the processor 110 to easily calculate the movement speed.

[0110] In this embodiment, the speed measurement device using multiple cameras 102 further includes a calibration tool provided on the moving platform 111, which calibrates the relative positional relationship between each camera 102 and the positional relationship of each camera 102 to the center of the moving platform 111, and the positional relationship can be acquired by the processor 110.

[0111] Furthermore, the calibration tool is a checkerboard, a laser range finder, a laser radar, a TOF sensor, or an encoder.

[0112] As a first preferred example of this embodiment, the camera set 101 includes at least a pair of binocular cameras, which refers to binocular cameras that include a common field of view and can obtain a three-dimensional image so that the camera set 101 can calculate the position, orientation, and velocity of the camera set 101 with three or more degrees of freedom in space based on the distance of each point in the image.

[0113] As a second preferred example of this embodiment, the camera set 101 includes at least two TOF cameras arranged along the main moving direction of the moving platform 111, thereby ensuring that the images used for matching the two front and rear cameras are stereoscopic images, and after matching the images, higher accuracy can be obtained depending on the distance and / or shape of the object to be photographed.

[0114] As a third preferred example of this embodiment, the camera set 101 includes at least a pair of line scan cameras arranged along the main moving direction of the moving platform 111, and the speed information of the camera set 101 can be obtained by comparing the images of each pair of line scan cameras.

[0115] Preferably, the camera set 101 further includes at least one global shutter area camera, and the processor 110 is adapted to compare an image captured by the global shutter area camera with an image captured by the line scan camera and correct the image captured by the line scan camera.

[0116] Furthermore, the multiple camera speed measurement device further includes at least one laser beam measuring device for measuring distances between different objects.

[0117] Furthermore, the laser beam measuring instrument is a line laser beam measuring instrument or a cross-shaped laser beam measuring instrument.

[0118] In this embodiment, the processor 110 can obtain highly accurate speed information by comparing images from different cameras, and can also obtain speed information through successive images from each camera, which can be used to roughly calculate the appearance time of images taken by different cameras that can be used for matching, thereby improving the matching speed.

[0119] In this embodiment, if the matching result of the two acquired images is less than a first threshold, the processor 110 can estimate the speed of the camera set 101 relative to the subject based on the position information of the center of the moving platform 111 at the current time, the position information of the previous time, and the shooting time difference between the current time and the previous time.

[0120] As described above, the embodiment of the present invention provides a speed measurement method and a speed measurement device using multiple cameras that obtain two images with high similarity by processing matching of images taken by different cameras, and since the measurement distance between the two images is the shortest and the overlapping degree of the images is the largest, the accuracy of the speed information of the moving platform calculated by the speed measurement method using multiple cameras is higher than the accuracy of the speed information of the object calculated by continuous image tracking using a single camera. In addition, the speed measurement method and the speed measurement device using multiple cameras according to the present invention form a feedback with the real environment to solve the problem of the robot making an erroneous judgment when the wheels slip or the wheels are tilted.

[0121] The above description shows the best mode of the present invention, and does not limit the present invention. Those skilled in the art can make many possible modifications and decorations or equivalent modifications to the technical means of the present invention using the above technical content without departing from the scope of the technical means of the present invention. Therefore, any simple modifications, equivalent changes and decorations to the above-mentioned embodiments based on the technical ideas of the present invention within the content of the technical means of the present invention should be considered to fall within the scope of protection by the technical means of the present invention. [Explanation of symbols]

[0122] 101 Camera Set 102 Camera 103 Light source 110 Processor 111 Moving Platform W World coordinate system C centered coordinate system

Claims

1. S1: A step of calibrating the position of each camera in a camera set with a fixed relative position provided on a moving platform to obtain a positional relationship between each camera and a positional relationship between each camera and the moving platform; S2: acquiring a synchronized image sequence for each camera in the camera set and recording a timestamp of the synchronized image sequence; S3: Processing matching of images captured by each camera in the camera set, and when the matching result of the obtained two images is greater than a first threshold, calculating position transformation matrices of the two images, and obtaining a position transformation matrix of the moving platform during a shooting time interval of the two images according to a positional relationship between the cameras capturing the two images, a positional relationship of the cameras capturing the two images relative to the moving platform, and the position transformation matrices of the two images; and finally, calculating speed information of the moving platform according to the position transformation matrix of the moving platform and the shooting time interval of the two images; The step S3 S31: obtaining position information of the moving platform at a current time; and calculating a first position of the first camera at a current time based on the position information of the moving platform at the current time and a positional relationship of a first camera to the moving platform, where the first camera is any one of the cameras in the camera set; S32: searching for a second position stored when another camera captured an image at a past time based on the first position, and if the distance between the second position and the first position is less than a second threshold, determining that the search is successful, recording the captured images corresponding to the first position and the second position as a search image and a candidate image, respectively, recording timestamps of the search image and the candidate image, and then proceeding to step S33; S33: performing matching between the search image and the candidate image, and if the obtained matching result is greater than the first threshold, determining that the matching is successful, calculating a position transformation matrix between the search image and the candidate image, and then proceeding to step S34; and S34: Calculating a position transformation matrix of the first camera based on the positional relationship between the cameras of the search image and the candidate image and the position transformation matrix; then calculating a position transformation matrix of the moving platform based on the position transformation matrix of the first camera and the positional relationship of the first camera with respect to the moving platform; and finally calculating speed information of the moving platform based on the position transformation matrix of the moving platform and the shooting time interval of the two images; A speed measurement method using multiple cameras.

2. The speed measurement method using multiple cameras as described in claim 1, characterized in that the step of acquiring position information of the moving platform at the current time includes a step of matching the search image of the first camera with a previous frame image to acquire a self-position transformation matrix of the first camera, and acquiring position information of the moving platform based on the self-position transformation matrix and the positional relationship of the first camera with respect to the moving platform.

3. The speed measurement method using multiple cameras as described in claim 2, characterized in that when obtaining the self-position transformation matrix of the first camera by matching the search image of the first camera with a previous frame image, and obtaining the position information of the moving platform based on the self-position transformation matrix and the positional relationship of the first camera with respect to the moving platform, a weight corresponding to each camera is set based on the matching result of each camera, and the position information of the moving platform is calculated using a weighted average method.

4. The speed measurement method using multiple cameras according to claim 1 or 3, characterized in that the matching method is a 2D image feature matching method, which calculates a position transformation matrix of the two images by extracting feature points of the two images and quickly matching them.

5. The method for measuring speed using multiple cameras according to claim 1 or 3, characterized in that the matching method is a 2D image frequency domain information matching method, which calculates the rotation, translation, and scale factors of the two images through a Fourier-Mellin transform algorithm, and further calculates a position transformation matrix of the two images.

6. The method for measuring speed using multiple cameras according to claim 1 or 3, characterized in that the matching method is a 3D matching method, which calculates a position, orientation and velocity of three degrees of freedom in space through 3D point coordinates corresponding to two images, and further calculates a position transformation matrix of the two images.

7. The method for measuring speed using multiple cameras according to claim 1, further comprising directly acquiring position information of the moving platform at the current time via a position sensor.

8. The speed measurement method using multiple cameras as described in claim 1, characterized in that when performing step S32 or S33, if the search or matching fails, proceeds to step S35, which includes a step of obtaining position information of the moving platform at the current time, position information at the previous time, and the shooting time interval between the current time and the previous time, and estimating speed information of the moving platform.

9. a moving platform movable relative to the subject; a camera set including at least two cameras mounted on the moving platform, the relative positions of the cameras being fixed; A processor arranged to carry out the method of claim 1.

10. The camera set includes a light source that matches the camera; a camera equipped with a near-infrared light source and a lens that matches the near-infrared light source and is capable of transmitting the wavelength of the light emitted by the near-infrared light source; Or, a far-infrared light source and a matching far-infrared camera, Or an ultraviolet light source and a matching ultraviolet camera, Alternatively, the speed measurement device using multiple cameras according to claim 9, further comprising a camera equipped with a polarized light source and a lens that matches the polarized light and can transmit the polarized light.

11. 10. The speed measuring device according to claim 9, wherein the image capture times of each camera are synchronized or each camera has a time stamp.

12. The multiple camera speed measurement device according to claim 9, further comprising a calibration tool provided on the moving platform, the calibration tool calibrating the relative positional relationship between each camera and the positional relationship of each camera to the moving platform.

13. The multiple camera speed measurement device according to claim 12 , wherein the calibration tool is a checkerboard, a laser range finder, a laser radar, a TOF sensor, or an encoder.

14. 10. The multiple camera speed measurement device according to claim 9, wherein the camera set includes at least a pair of binocular cameras.

15. 10. The multiple camera speed measurement device of claim 9, wherein the camera set includes at least two TOF cameras positioned along a main direction of movement of the moving platform.

16. 10. The multiple camera speed measurement device of claim 9, wherein the camera set includes at least one pair of line scan cameras positioned along a main direction of movement of the moving platform.

17. 17. The speed measurement device using multiple cameras as described in claim 16, characterized in that the camera set further includes at least one global shutter area camera, and the processor is capable of comparing an image taken by the global shutter area camera with an image taken by the line scan camera to correct the image taken by the line scan camera.

18. The multiple camera speed measurement device of claim 9, characterized in that if the matching result of the two acquired images is less than a first threshold, the processor can estimate speed information of the moving platform based on the position information of the moving platform at the current time, the position information at the previous time, and the shooting time interval between the current time and the previous time.

19. 10. The speed measurement device according to claim 9, further comprising at least one laser beam measuring device for measuring distances between different objects.

20. 20. The speed measurement device using multiple cameras according to claim 19, wherein the laser beam measurement device is a line laser beam measurement device or a cross-line laser beam measurement device.

Citation Information

Patent Citations

  • Sequence image rapid three-dimensional reconstruction method based on mobile platform shooting

    CN105825518A

  • Method for robot fast positioning and attitude estimation based on depth camera

    CN106403924A

  • Mobile robot positioning device based on multiple cameras

    CN108426566A

  • Obstacle recognition unit for vehicle

    JP1995318652A

  • Speedometer and method for measuring speed

    JP2000097968A