Relevance matching method and relevance matching system

Abstract

The invention provides a relevance matching method and a relevance matching system. In the method, a first reference coordinate of a first reference position in a first image on a first local map is determined, and a second reference coordinate of a second reference position in a second image on a second local map is determined. And mapping the first local map and the second local map to a global map. The first reference coordinate of the first reference position and / or the second reference coordinate of the second reference position are / is mapped to the coordinate of the global map according to the conversion relation. And according to the third reference positions in the one or more third images and the fourth reference positions in the one or more fourth images, mapping to coordinates of the global map through the conversion relation to correct the conversion relation. Therefore, the object matching efficiency can be improved.

Description

Technical Field

[0001] The present invention relates to an image processing technology, and in particular to a correlation matching method and a correlation matching system. Background Art

[0002] In the development of smart cities, sensors scattered at various intersections play a very important role. These sensors can sense regional information in real time, allowing the system to analyze and grasp the overall situation and even further perform contingency processing. With the help of information transmission through communication technology, a large number of sensors can be installed to expand the monitoring field of view to more areas. If the camera equipment is equipped with an embedded platform with computing power, it can not only feedback image signals, but also obtain more effective information through various post-processing, thereby reducing the burden on the server.

[0003] The existing single-camera object detection and tracking technology has been developed to a considerable extent. However, for a wide area such as an intersection, a single camera cannot detect the entire area. In practical applications, the location information of the object may be lost due to occlusion. Therefore, effective tracking target association technology with multiple cameras with overlapping fields of view is necessary.

[0004] However, in the past, the multi-camera object tracking monitoring system can only record the camera number sequence of the object, but cannot obtain the exact position information of each object in each frame.

[0005] In addition, previous multi-camera object tracking monitoring systems used visual information such as image color and appearance as the basis for multi-camera object matching. However, these matching bases may be affected by factors such as light and changes in the perspective of moving objects, causing feature changes and further affecting the matching effect. Summary of the invention

[0006] The present invention provides a correlation matching method and a correlation matching system, which can use relative reference coordinates to perform target correlation to reduce the influence of appearance features.

[0007] The associative matching method of the embodiment of the present invention includes (but is not limited to) the following steps: determining the first reference coordinates of the first reference position in the first image in the first local map (Local Map), and determining the second reference coordinates of the second reference position in the second image in the second local map. The first image is obtained by a first image acquisition device, and the second image is obtained by a second image acquisition device. Mapping the first local map and the second local map to a global map (Global Map). The first reference coordinates of the first reference position and / or the second reference coordinates of the second reference position are mapped to the coordinates of the global map according to a conversion relationship. Correcting the conversion relationship according to the third reference position in one or more third images and the fourth reference position in one or more fourth images mapped to the coordinates of the global map through the conversion relationship. The third image is obtained by the first image acquisition device, and the fourth image is obtained by the second image acquisition device, and the corrected conversion relationship minimizes the error distance between the coordinates of the third reference position mapped to the global map and the coordinates of the fourth reference position mapped to the global map.

[0008] The associative matching system of the embodiment of the present invention includes (but is not limited to) a computing device. The computing device determines the first reference coordinates of the first reference position in the first image in the first local map, determines the second reference coordinates of the second reference position in the second image in the second local map, maps the first local map and the second local map to the global map, and corrects the conversion relationship based on the third reference position in one or more third images and the fourth reference position in one or more fourth images mapped to the coordinates of the global map through the conversion relationship. The first image is obtained by the first image acquisition device, and the second image is obtained by the second image acquisition device. The first reference coordinates of the first reference position and / or the second reference coordinates of the second reference position are mapped to the coordinates of the global map according to the conversion relationship. The third image is obtained by the first image acquisition device, and the fourth image is obtained by the second image acquisition device, and the corrected conversion relationship minimizes the error distance between the coordinates of the third reference position mapped to the global map and the coordinates of the fourth reference position mapped to the global map.

[0009] Based on the above, the correlation matching method and correlation matching system of the embodiment of the present invention can convert the two reference coordinates corresponding to the same object acquired by two image acquisition devices into the coordinates of the global map, and adjust the conversion relationship from the local map to the global map according to other reference coordinates. In this way, the influence of light and perspective changes on appearance feature matching can be reduced, thereby improving the performance of object tracking.

[0010] In order to make the above features and advantages of the present invention more clearly understood, embodiments are given below with reference to the accompanying drawings for detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a block diagram of components of a relevance matching system according to an embodiment of the present invention.

[0012] Figure 2 is a flow chart of a correlation matching method according to an embodiment of the present invention.

[0013] Figure 3A It is a schematic diagram of the positional relationship in a top view according to an embodiment of the present invention.

[0014] Figure 3B is a schematic diagram of a first image according to an embodiment of the present invention.

[0015] Figure 3C is a schematic diagram of a second image according to an embodiment of the present invention.

[0016] Figure 4A It is a schematic diagram of the positional relationship in a top view according to another embodiment of the present invention.

[0017] Figure 4B is a schematic diagram of a first image according to another embodiment of the present invention.

[0018] Figure 4C is a schematic diagram of a second image according to another embodiment of the present invention.

[0019] Figure 5 FIG. 4 is a schematic diagram of converting pixel coordinates to local map coordinates according to an embodiment of the present invention.

[0020] Fig. 6A is a schematic diagram of a first local map according to an embodiment of the present invention.

[0021] Figure 6B is a schematic diagram of a second local map according to an embodiment of the present invention.

[0022] Fig. 7A is a schematic diagram of a third image according to an embodiment of the present invention.

[0023] Figure 7B is a schematic diagram of a fourth image according to an embodiment of the present invention.

[0024] Explanation of symbols

[0025] 1: Relevance matching system

[0026] 10: Computing device

[0027] 11: Memory

[0028] 12: Communication transceiver

[0029] 13: Processor

[0030] 21, 22: Image acquisition device

[0031] S210~S230: Steps

[0032] O1~O4: Object

[0033] V: Mobile Vehicle

[0034] X, Y, Z, U, V, X1, Y1, X2, Y2: axes

[0035] IP: Image Plane

[0036] (x,y,z), (u,v): coordinates

[0037] P c : Projection point

[0038] P: Object

[0039] O: origin

[0040] O5 t1 、O5 t2 、O5 t3 : Reference position DETAILED DESCRIPTION

[0041] Figure 1 is a block diagram of components of a correlation matching system 1 according to an embodiment of the present invention. Figure 1 The correlation matching system 1 includes (but is not limited to) a computing device 10 and image acquisition devices 21 and 22. The correlation matching system 1 can be applied in application scenarios such as transportation, indoor, sports or military.

[0042] The computing device 10 may be a smart phone, a tablet computer, a server, a cloud host, a computer host, a wearable device or other electronic devices. The computing device 10 includes (but is not limited to) a memory 11 , a communication transceiver 12 and a processor 13 .

[0043] The memory 11 can be any form of fixed or removable random access memory (RAM), read only memory (ROM), flash memory, hard disk drive (HDD), solid-state drive (SSD) or similar components. In one embodiment, the memory 11 is used to store program code, software modules, configurations, data (e.g., images, coordinates, or conversion relationships) or files, and its embodiments will be described in detail later.

[0044] The communication transceiver 12 may be a communication transceiver supporting, for example, the fourth generation (4G) or other generations of mobile communications, Wi-Fi, Bluetooth, infrared, radio frequency identification (RFID), Ethernet, optical fiber network, etc., a serial communication interface (such as RS-232), or a universal serial bus (USB), Thunderbolt, or other communication transmission interface. In one embodiment, the communication transceiver 12 is used to transmit or receive data with other electronic devices (such as image acquisition devices 21, 22).

[0045] The processor 13 is coupled to the memory 11 and the communication transceiver 12. The processor 13 may be a central processing unit (CPU), a graphics processing unit (GPU), or other programmable general-purpose or special-purpose microprocessor (Microprocessor), a digital signal processor (Digital Signal Processor, DSP), a programmable controller, a field programmable gate array (Field Programmable Gate Array, FPGA), an application-specific integrated circuit (Application-Specific Integrated Circuit, ASIC), a neural network accelerator or other similar elements or a combination of the above elements. In one embodiment, the processor 13 is used to execute all or part of the operations of the computing device 10, and can load and execute various program codes, software modules, files and data stored in the memory 11. In some embodiments, the functions of the processor 13 can be implemented by software or chips.

[0046] The image acquisition devices 21 and 22 may be cameras, video cameras, monitors, smart phones or road side units (RSU) with image acquisition functions, and acquire one or more images within a specified field of view.

[0047] In some embodiments, any one of the image acquisition devices 21 and 22 can be integrated with the computing device 10 into an independent device. Figure 1 Two image acquisition devices 21 and 22 are used as an example for description, but the number is not limited thereto.

[0048] Hereinafter, the method described in the embodiment of the present invention will be described in conjunction with various devices, components and modules in the correlation matching system 1. Each process of the method can be adjusted according to the implementation situation.

[0049] Figure 2 is a flow chart of a correlation matching method according to an embodiment of the present invention. Figure 2 , the processor 13 determines the first reference position in the first image at the first reference coordinates of the first local map, and determines the second reference position in the second image at the second reference coordinates of the second local map (step S210). Specifically, the first image is obtained by the image acquisition device 21, and the second image is obtained by the image acquisition device 22. The time of shooting the first image and the second image can be simultaneous or separated by an allowable period (for example, 50 milliseconds (ms), 100ms or 500ms). The first reference position is the position of the object in the first image, and the second reference position is the position of the same object in the second image. The object can be a person, an animal, a vehicle, a machine, a building or other specific objects. In one embodiment, for a single frame of an image, an object detection technology can be used to identify the object. There are many algorithms for object detection. For example, YOLO (You Only Look Once), SSD (Single Shot Detector) or R-CNN, but not limited thereto. The detection result of the object can be representative information of the object. For example, object type, bounding box, identification code and / or coordinates. In another embodiment, the first reference position and the second reference position may also be positions specified by the user's input operation at the first image and / or the second image. For example, a touch screen receives a touch operation at a specific position, or a mouse receives an operation of moving to a specific position and clicking.

[0050] In one embodiment, the processor 13 may define that the field of view of the image acquisition device 21 partially overlaps the field of view of the image acquisition device 22. The field of view refers to the range of the scene that can be captured by the image acquisition devices 21 and 22. Since the first reference position and the second reference position are for the same object in the first image and the second image, the fields of view of the two image acquisition devices 21 and 22 at least partially overlap. However, depending on the hardware specifications of the image acquisition devices 21 and 22 (e.g., the focal length or viewing angle of the lens, or the size of the photosensitive element) and the application scenario (e.g., the environment or task content), the size and shape of the overlapping area of ​​the two fields of view may be different.

[0051] In one embodiment, the image acquisition device 21 is located within the field of view of the image acquisition device 22, and the image acquisition device 22 is located within the field of view of the image acquisition device 21. The field of view can be adjusted by changing the shooting direction, lens specifications, image cropping or other methods. Therefore, the image acquisition device 22 is located in the first image captured by the image acquisition device 21, and the image acquisition device 21 is located in the second image captured by the image acquisition device 22. For example, Figure 3Ais a schematic diagram of the positional relationship in a top view according to an embodiment of the present invention. Figure 3A , image acquisition devices 21, 22 (such as roadside units) are arranged at two diagonal positions of the intersection. The image acquisition devices 21, 22 shoot towards each other so that the overlapping field of view of the image acquisition devices 21, 22 roughly covers the intersection (the area at the bottom of the oblique bottom line as shown in the figure). It should be noted that Figure 3A The field of view is only used as an example, and its shape, size, position and setting environment may change. For example, the image acquisition devices 21 and 22 are monitors of a store, a hall or a parking lot, and may be set at any position in the field.

[0052] Figure 3B is a schematic diagram of a first image according to an embodiment of the present invention. Figure 3A and Figure 3B The first image acquired by the image acquisition device 21 includes three objects O1, O2, and O3 within the overlapping field of view.

[0053] Figure 3C is a schematic diagram of a second image according to an embodiment of the present invention. Figure 3A and Figure 3C , the second image acquired by the image acquisition device 22 includes the three objects O1, O2, O3 in the overlapping field of view. That is, when the objects O1, O2, O3 are located in the overlapping field of view, the first image and the second image both include the images of the objects O1, O2, O3.

[0054] In another embodiment, the image acquisition device 22 is not located in the field of view of the image acquisition device 21, and the image acquisition device 21 is not located in the field of view of the second image acquisition device 22. The field of view can be adjusted by changing the shooting direction, lens specifications, image cropping or other methods. Therefore, the image acquisition device 22 is not located in the first image captured by the image acquisition device 21, and the image acquisition device 21 is not located in the second image captured by the image acquisition device 22. For example, Figure 4A is a schematic diagram of the positional relationship in a top view according to another embodiment of the present invention. Figure 4A , both image acquisition devices 21 and 22 are arranged on one side of a mobile vehicle V (e.g., a car, a motorcycle, or a bus). The image acquisition devices 21 and 22 face the same direction for shooting, so that the overlapping fields of view of the image acquisition devices 21 and 22 are approximately located at the bottom of the oblique bottom line shown in the figure. It should be noted that Figure 4A The field of view is only used as an example, and its shape, size, position and setting environment may change. For example, the image acquisition devices 21 and 22 are monitors of a store, a hall or a parking lot, and may be set at any position in the field.

[0055] Figure 4Bis a schematic diagram of a first image according to another embodiment of the present invention. Figure 4A and Figure 4B The first image acquired by the image acquisition device 21 includes the object O4 within the overlapping field of view.

[0056] Figure 4C is a schematic diagram of a second image according to another embodiment of the present invention. Figure 4A and Figure 4C , the second image acquired by the image acquisition device 22 includes the object O4 in the overlapping field of view. That is, when the object O4 is located in the overlapping field of view, both the first image and the second image include the image of the object O4.

[0057] On the other hand, the coordinates of the first local map are defined by the relative distance from the image acquisition device 21, and the coordinates of the second local map are defined by the relative distance from the image acquisition device 22. The two local maps include a local coordinate system established with a vertical axis and a horizontal axis. The relative distance includes a vertical distance and a horizontal distance. The location of the image acquisition device 21 is taken as the origin of the vertical axis and the horizontal axis, the vertical axis corresponds to the vertical distance relative to the location of the image acquisition device 21, and the horizontal axis corresponds to the horizontal distance of the location of the image acquisition device 21. Similarly, the location of the image acquisition device 22 is taken as the origin of the vertical axis and the horizontal axis, the vertical axis corresponds to the vertical distance relative to the location of the image acquisition device 22, and the horizontal axis corresponds to the horizontal distance of the location of the image acquisition device 22.

[0058] In one embodiment, the processor 13 may convert the pixel coordinates of the first reference position in the first image into the first reference coordinates and / or convert the pixel coordinates of the second reference position in the second image into the second reference coordinates through a pinhole camera model. Figure 5 is a schematic diagram of the pixel coordinates (u, v) to the coordinates (x, y, z) of the local map according to an embodiment of the present invention. Figure 5 , the local / world coordinates include the X-axis, Y-axis and Z-axis. The image coordinates include the U-axis and the V-axis. The image acquisition device 21 or 22 is located at the origin O of the local coordinate system. Assume that the coordinates of the object P in the local coordinate system are (x, y, z). The pinhole camera model is to project an object in the three-dimensional world onto a two-dimensional image plane IP through an ideal pinhole camera (whose pinhole aperture is infinitely small), and its mathematical expression is:

[0059]

[0060] , (x, y, z) are the x, y, z coordinates of the position of object P according to the local / world coordinate system, and (u, v) are the projected point P in pixels.c The pixel coordinates, f x 、f y is the number of pixels in the U-axis direction and the V-axis direction of the focal length, O x , O y are the number of pixels displaced by the origin O in the U-axis direction and the V-axis direction (i.e., the pixel coordinates of the origin O of the local coordinate system on the image plane IP), r 11 ~r 33 is the element in the rotation matrix for transforming the local / world coordinate system to the camera coordinate system, t1-t3 are the elements in the translation vector for transforming the local / world coordinate system to the camera coordinate system, and s (optional) is the skew parameter and is applicable to the situation where the U axis and the V axis are not perpendicular. Except for the coordinates (x, y, z) and (u, v), the remaining parameters can be stored in the memory 12 for use by the processor 13. Then, the coordinates (u, v) are substituted into equation (1) to obtain the coordinates (x, y, z). That is, the first reference coordinate is the pixel coordinate of the first reference position in the first image transformed by equation (1), and the second reference coordinate is the pixel coordinate of the second reference position in the second image transformed by equation (1).

[0061] Fig. 6A is a schematic diagram of a first local map according to an embodiment of the present invention. Figure 3B and Fig. 6A The first local map includes an X1 axis and a Y1 axis (corresponding to the X axis and Y axis of the aforementioned local / world coordinate system, respectively). Figure 3B The pixel coordinates of the first reference positions of the objects O1 - O3 in the first image may be converted into coordinates of the X1 axis and the Y1 axis (the Z axis coordinate is ignored).

[0062] Figure 6B is a schematic diagram of a second local map according to an embodiment of the present invention. Figure 3C and Figure 6B The second local map includes an X2 axis and a Y2 axis (corresponding to the X axis and Y axis of the aforementioned local / world coordinate system, respectively). Figure 3C The pixel coordinates of the second reference positions of the objects O1-O3 in the second image can be converted into coordinates of the X2 axis and the Y2 axis (the coordinate of the Z axis is ignored).

[0063] The pinhole camera model assumes an ideal imaging model, but the lens imaging may be distorted. For example, radial and tangential distortion. In one embodiment, based on the lens specifications of the image acquisition devices 21 and 22, the processor 13 can perform distortion correction on the first reference coordinate and / or the second reference coordinate. For example, the radial and tangential distortions are described by corresponding distortion parameters, and a correction equation is derived accordingly. The processor 13 can substitute the first reference coordinate and / or the second reference coordinate into the correction equation to correct the first reference coordinate and / or the second reference coordinate.

[0064] In some embodiments, the processor 13 may further determine first reference coordinates of first reference positions corresponding to more objects in the first image in the first reference coordinates of the first local map, and / or may determine second reference coordinates of second reference positions corresponding to more objects in the second image in the second reference coordinates of the second local map.

[0065] Please refer to Figure 2 , the processor 13 maps the first local map and the second local map to the global map (GlobalMap) (step S220). Specifically, the coordinate system of the global map also adopts the world coordinate system, that is, the coordinates are defined using the X-axis corresponding to the horizontal direction and the Y-axis corresponding to the vertical direction. The local coordinate systems used by the first local map and the second local map are defined by the relative positions relative to the two image acquisition devices 21 and 22, respectively. Then, the two local coordinate systems are mapped to the same global coordinate system, so that the reference coordinates of the two objects can be matched.

[0066] In one embodiment, at least one of the first reference coordinates of the first reference position and the second reference coordinates of the second reference position can be mapped to the coordinates of the global map according to the conversion relationship. It is worth noting that the coordinate conversion of different coordinate systems can be achieved by rotating and / or translating the coordinates. The conversion relationship is how to rotate and / or how to translate the coordinates. For example, rotate 30 degrees clockwise and translate three units to the right, or rotate 90 degrees counterclockwise and translate one unit upward.

[0067] In one embodiment, the processor 13 may define the first local map as a reference map. That is, the local coordinate system of the first local map is used as the global coordinate system. The first reference coordinate is the coordinate of the first reference position in the global map. Then, the processor 13 may determine the conversion relationship of the second reference coordinate to the first reference coordinate in the reference map. The first reference position and the second reference position are the positions of the same object in the first image and the second image respectively. Therefore, the second reference coordinate of the second reference position on the second local map mapped to the coordinate of the reference map through the conversion relationship should be the same as the first reference coordinate. The processor 13 can determine how the second reference coordinate is rotated and / or translated to be equivalent to the first reference coordinate.

[0068] In one embodiment, the transformation relationship is a transformation matrix. As described above for the pinhole camera model, the transformation of different coordinate systems can use a transformation matrix. The transformation matrix may include a rotation matrix and / or a translation vector. The mathematical expression of the transformation matrix is:

[0069]

[0070] , (x1, y1) is the first reference coordinate, (x2, y2) is the second reference coordinate, is the rotation matrix, θ is the rotation angle, is the translation vector, Tx is the translation amount on the X axis, and Ty is the translation amount on the Y axis.

[0071] In one embodiment, the processor 13 may substitute the first reference coordinates and the second reference coordinates of the plurality of objects into equation (2) to deduce θ, Tx, and Ty. That is, the unknowns in the simultaneous equations are solved. In another embodiment, the processor 13 may assume values ​​of θ, Tx, and Ty, and approximate the other of the first reference coordinate and the second reference coordinate by substituting one of the first reference coordinate and the second reference coordinate and changing the values ​​of θ, Tx, and Ty.

[0072] In another embodiment, the conversion relationship is a conversion table or a conversion equation. The processor 13 can query the first reference coordinate corresponding to the second reference coordinate from the conversion table, or substitute the second reference coordinate into the conversion equation to obtain the first reference coordinate.

[0073] In another embodiment, the conversion relationship includes a first conversion relationship and a second conversion relationship. The first reference coordinates of the first reference position can be mapped to the coordinates of the global map according to the first conversion relationship, and the second reference position can be mapped to the coordinates of the global map according to the second conversion relationship. In other words, the first conversion relationship is the relationship of how to rotate and / or how to translate the coordinates of the first local map, and the second conversion relationship is the relationship of how to rotate and / or how to translate the coordinates of the second local map. Similarly, the first conversion relationship and the second conversion relationship can be obtained by calculating the aforementioned conversion matrix, conversion comparison table and / or conversion equation.

[0074] Please refer to Figure 2, the processor 13 maps the third reference position in one or more third images and the fourth reference position in one or more fourth images to the coordinate correction transformation relationship of the global map through the transformation relationship (step S230). Specifically, one or more third images are obtained by the image acquisition device 21, and one or more fourth images are obtained by the image acquisition device 22. The image acquisition devices 21 and 22 can synchronously shoot the time of the third image and the fourth image or the time interval between the corresponding frames within the allowable period (for example, 50ms, 100ms or 500ms). The third reference position is the position of the object in the third image, and the fourth reference position is the position of the same object in the fourth image. The object can be a person, an animal, a vehicle, a machine, a building or other specific object.

[0075] In one embodiment, the processor 13 can identify an object in one or more third images and one or more fourth images through object detection technology or input operation marking, define the position of the object on the third image as the third reference position, and define the position of the object on the fourth image as the fourth reference position. Similarly, as described in step S210, the reference position of the same object can be marked through object detection technology or user input operation.

[0076] The conversion relationship obtained in step S220 may have errors. For the same object, the pixel positions of the third reference position and / or the fourth reference position may obtain different coordinates through the conversion relationship obtained in step S220. The distance between these two coordinates can be defined as the error distance. Therefore, it is necessary to further correct this conversion relationship. The corrected conversion relationship can minimize the error distance between the coordinates of the third reference position mapped to the global map and the coordinates of the fourth reference position mapped to the global map.

[0077] In one embodiment, the processor 13 may define an objective function based on the error distance. For example, the distance between the third reference coordinate and the fourth reference coordinate of a single object at a certain time point is defined as the objective function. For another example, the distance between the third reference coordinate and the fourth reference coordinate of a single object at multiple time points is defined as the objective function. The third reference coordinate is the pixel coordinate of the third reference position in the third image mapped to the coordinate of the global map through the conversion relationship of step S220, and the fourth reference coordinate is the pixel coordinate of the fourth reference position in the fourth image mapped to the coordinate of the global map through the conversion relationship of step S220.

[0078] The processor 13 can minimize the objective function to determine the modified conversion relationship. Taking equation (2) as an example, θ, Tx, and Ty are unknowns, and the values ​​of θ, Tx, and Ty are estimated by the least square method or other regression algorithms. Alternatively, by substituting different values ​​of θ, Tx, and Ty, the value of the objective function is minimized.

[0079] In one embodiment, the third images include third images at multiple time points, and the fourth images include fourth images at multiple time points. Fig. 7A is a schematic diagram of a third image according to an embodiment of the present invention, and Figure 7B is a schematic diagram of a fourth image according to an embodiment of the present invention. Fig. 7A and Figure 7B , the reference position of the same object at three time points t1, t2, and t3 is O5 t1 、O5 t2 、O5 t3 The image acquisition devices 21 and 22 can synchronously record videos to respectively generate third images and fourth images at multiple time points.

[0080] The processor 13 may define an objective function based on the sum of the error distances between the third reference positions at these time points and the fourth reference positions at those time points mapped to the two coordinates of the global map through the conversion relationship of step S220. For example, the objective function is TP is the number of time points, and ΔDistance is the error distance. Fig. 7A and Figure 7B For example, TP is 3, k=0 corresponds to time point t1, k=1 corresponds to time point t2, and k=2 corresponds to time point t3. That is, it is used to find the optimal solution (eg, the values ​​of θ, Tx, and Ty) that minimizes the sum of the error distances at multiple time points.

[0081] In one embodiment, the processor 13 may perform the correction of step S230 regularly or based on an event to provide a more accurate conversion relationship.

[0082] In one embodiment, based on the modified transformation relationship, the processor 13 may utilize object tracking technology to track subsequently detected objects. For multiple continuous images or videos, (multiple) object tracking technologies may be utilized. The main function of object tracking is to track the same object framed by the previous and next frames. There are many object tracking algorithms. For example, optical flow, SORT (Simple Online and Realtime Tracking) or Deep SORT, Joint Detection and Embedding (JDE). The processor 13 may map the image coordinates of the reference position of the object on the image to the global map through the modified transformation relationship to obtain the coordinates of the object on the global map. Then, the processor 13 may continue to track the same object in other frames based on the coordinates of the global map.

[0083] In one embodiment, based on the modified conversion relationship, the processor 13 can obtain other motion information, such as speed, acceleration, or rotation angle. The processor 13 can calculate the motion information based on the coordinates of the global map based on the modified conversion relationship. For example, the moving speed and acceleration are determined based on the coordinates of multiple frames.

[0084] In one embodiment, Figure 4A For example, the embodiment of the present invention can also be applied to the bird's eye view (Bird View View) of the Advanced Driver-Assistance System (ADAS). For example, the coordinates of the global map based on the modified conversion relationship are used to mark the object O4 in the surrounding view of the mobile vehicle V or provide a visual prompt corresponding to the relative distance. For another example, the satellite positioning coordinates of the object O4 are determined based on the satellite positioning coordinates of the mobile vehicle V and the coordinates of the global map.

[0085] It is worth noting that since the embodiment of the present invention directly matches the relative position relationship of different local maps, there are fewer restrictions on the location of the image acquisition devices 21 and 22. For example, the image acquisition devices 21 and 22 can be placed at Figure 4A The mobile vehicle V is shown.

[0086] In summary, in the relevance matching method and relevance matching system of the embodiment of the present invention, the reference coordinates in the local map corresponding to the images of the two image acquisition devices are mapped to a single global map, and the conversion relationship used for mapping is corrected. In this way, the visual feature differences caused by the hardware and viewing angle differences of the camera can be avoided, and it can also be applied to the scenario of a mobile vehicle.

[0087] Although the present invention has been disclosed as above by way of embodiments, it is not intended to limit the present invention. Any person skilled in the art may make some changes and modifications without departing from the concept and scope of the present invention. Therefore, the protection scope of the present invention shall be determined by the claims.

Claims

1. A relevance matching method, comprising: Determine a first reference coordinate of a first reference position in a first image at a first local map, and determine a second reference coordinate of a second reference position in a second image at a second local map, wherein the first image is acquired by a first image acquisition device, and the second image is acquired by a second image acquisition device; Mapping the first local map and the second local map to a global map, wherein at least one of the first reference coordinate of the first reference position and the second reference coordinate of the second reference position is mapped to a coordinate of the global map according to a conversion relationship; as well as The transformation relationship is corrected based on a third reference position in at least one third image and a fourth reference position in at least one fourth image mapped to coordinates of the global map through the transformation relationship, wherein the at least one third image is acquired by the first image acquisition device, and the at least one fourth image is acquired by the second image acquisition device, and the corrected transformation relationship minimizes the error distance between the coordinates of the third reference position mapped to the global map and the coordinates of the fourth reference position mapped to the global map.

2. The correlation matching method as claimed in claim 1, wherein the step of modifying the conversion relationship comprises: defining an objective function according to the error distance; as well as The objective function is minimized to determine the modified conversion relationship.

3. The associative matching method of claim 2, wherein the at least one third image comprises third images at a plurality of time points, the at least one fourth image comprises fourth images at the plurality of time points, and the step of defining the objective function according to the error distance comprises: The objective function is defined according to the sum of error distances between two coordinates of the global map mapped to the third reference position at the multiple time points and the fourth reference position at the multiple time points.

4. The correlation matching method as claimed in claim 1, wherein the step of modifying the conversion relationship comprises: identifying an object in the at least one third image and the at least one fourth image; as well as The position of the object on the at least one third image is defined as the third reference position, and the position of the object on the at least one fourth image is defined as the fourth reference position.

5. The associative matching method as claimed in claim 1, wherein the step of mapping the first local map and the second local map to the global map comprises: defining the first local map as a reference map; as well as The transformation relationship from the second reference coordinate to the first reference coordinate in the reference map is determined. 6 . The associative matching method as claimed in claim 1 , wherein the transformation relationship is a transformation matrix, and the transformation matrix is ​​used to perform at least one of rotation and translation on the coordinates.

7. The correlation matching method according to claim 1, further comprising: The field of view of the first image acquisition device is defined to partially overlap the field of view of the second image acquisition device. 8 . The associative matching method as claimed in claim 7 , wherein the second image acquisition device is located within the field of view of the first image acquisition device, and the first image acquisition device is located within the field of view of the second image acquisition device. 9 . The associative matching method as claimed in claim 7 , wherein the second image acquisition device is not located in the field of view of the first image acquisition device, and the first image acquisition device is not located in the field of view of the second image acquisition device.

10. The associative matching method as claimed in claim 1, wherein the step of determining the first reference coordinates of the first reference position in the first image in the first local map comprises: The pixel coordinates of the first reference position in the first image are converted into the first reference coordinates through a pinhole camera model.

11. A relevance matching system, comprising: a computing device, Determine a first reference coordinate of a first reference position in a first image at a first local map, and determine a second reference coordinate of a second reference position in a second image at a second local map, wherein the first image is acquired by a first image acquisition device, and the second image is acquired by a second image acquisition device; Mapping the first local map and the second local map to a global map, wherein at least one of the first reference coordinate of the first reference position and the second reference coordinate of the second reference position is mapped to a coordinate of the global map according to a conversion relationship; as well as The transformation relationship is corrected based on a third reference position in at least one third image and a fourth reference position in at least one fourth image mapped to coordinates of the global map through the transformation relationship, wherein the at least one third image is acquired by the first image acquisition device, and the at least one fourth image is acquired by the second image acquisition device, and the corrected transformation relationship minimizes the error distance between the coordinates of the third reference position mapped to the global map and the coordinates of the fourth reference position mapped to the global map.

12. The relevance matching system as claimed in claim 11, wherein the computing device: defining an objective function according to the error distance; and The objective function is minimized to determine the modified conversion relationship.

13. The associative matching system of claim 12, wherein the at least one third image comprises third images at a plurality of time points, the at least one fourth image comprises fourth images at the plurality of time points, and the computing device: The objective function is defined according to the sum of error distances between two coordinates of the global map mapped to the third reference position at the multiple time points and the fourth reference position at the multiple time points.

14. The relevance matching system as claimed in claim 11, wherein the computing device: identifying an object in the at least one third image and the at least one fourth image; and The position of the object on the at least one third image is defined as the third reference position, and the position of the object on the at least one fourth image is defined as the fourth reference position.

15. The relevance matching system according to claim 11, wherein the computing device: defining the first local map as a reference map; and The transformation relationship from the second reference coordinate to the first reference coordinate in the reference map is determined. 16 . The associative matching system as claimed in claim 11 , wherein the transformation relationship is a transformation matrix, and the transformation matrix is ​​used to at least one of rotate and translate the coordinates.

17. The relevance matching system as claimed in claim 11, wherein the computing device: The field of view of the first image acquisition device is defined to partially overlap the field of view of the second image acquisition device.

18. The relevance matching system according to claim 17, comprising: The first image acquisition device and the second image acquisition device, wherein the second image acquisition device is located in the field of view of the first image acquisition device, and the first image acquisition device is located in the field of view of the second image acquisition device.

19. The relevance matching system according to claim 17, comprising: The first image acquisition device and the second image acquisition device, wherein the second image acquisition device is not located in the field of view of the first image acquisition device, and the first image acquisition device is not located in the field of view of the second image acquisition device.

20. The relevance matching system as claimed in claim 11, wherein the computing device: The pixel coordinates of the first reference position in the first image are converted into the first reference coordinates through a pinhole camera model.

Images