Image target matching method, terminal equipment and storage medium
By constructing a homography matrix using the intrinsic parameters of a multi-view camera, detection boxes are directly projected between multi-view camera images, solving the mismatch problem caused by inaccurate extrinsic parameters in multi-view camera target matching, improving the accuracy and stability of matching, and simplifying the calibration operation.
Patent Information
- Application Number
- CN202410789240.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-18
- Publication Date
- 2024-11-12
AI Technical Summary
Existing multi-view cameras rely on camera extrinsic parameters for target matching, which leads to mismatch problems. Furthermore, the inaccuracy of the extrinsic parameters affects the robustness and accuracy of the perception results, and frequent calibration operations are required.
By constructing a homography matrix using camera intrinsics, the detection boxes of the second camera are directly projected onto the image of the first camera, and the matching relationship is determined based on the positional relationship between the detection boxes, thus avoiding dependence on extrinsic parameters.
It improves the accuracy and stability of target matching, reduces false matching, simplifies calibration operations, and enhances the stability of multi-view perception fusion algorithms.
Smart Images

Figure CN118941823A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer vision technology, and in particular to an image target matching method, a terminal device, and a storage medium. Background Art
[0002] With the rapid development of artificial intelligence and sensor technology, intelligent driving has become an important research field in academia and industry. Perception is an important part of intelligent driving, and the environmental information it obtains is the basis for the successful application of intelligent driving technology.
[0003] In intelligent driving systems, the most common sensor is the camera. Due to the limited field of view and detection range of a camera with a single focal length, the perception solution of multi-cameras is currently used. The focal lengths of different cameras in a multi-camera are different, which effectively increases the field of view and detection range of the multi-camera. Before fusing the perception results of different cameras in a multi-camera, a target matching operation is required. The relevant matching operation will involve additional features, fine parameter adjustments, etc., which may result in certain mismatches, greatly affecting the robustness and accuracy of the perception result matching. Summary of the invention
[0004] The present application provides an image target matching method, a terminal device and a storage medium.
[0005] A technical solution adopted by the present application is to provide an image target matching method; the method comprises:
[0006] Acquire a first image taken by a first camera in the multi-camera at the same time, and acquire a second image taken by a second camera in the multi-camera, and obtain at least one first detection frame corresponding to the first image, and at least one second detection frame corresponding to the second image; wherein the focal length corresponding to the first image is smaller than the focal length corresponding to the second image; and there is an image target in each detection frame;
[0007] Using a homography matrix, projecting at least one second detection frame into the first image, to obtain at least one third detection frame of the at least one second detection frame in the first image; wherein the homography matrix is determined based on a first intrinsic parameter of the first eyepiece camera and a second intrinsic parameter of the second eyepiece camera;
[0008] According to the positional relationship between each first detection frame and each third detection frame, a first matching relationship between each first detection frame and each third detection frame is determined; and based on the first matching relationship, a second matching relationship between at least one first detection frame and at least one second detection frame is determined.
[0009] Optionally, determining a first matching relationship between each first detection frame and each third detection frame according to a positional relationship between each first detection frame and each third detection frame includes:
[0010] Determine an offset matrix between a second center point of each second detection frame and a center point of each first detection frame;
[0011] Using the offset matrix, adjusting each first detection frame to obtain a fourth detection frame corresponding to each first detection frame;
[0012] In response to the existence of an intersection-and-union ratio between any fourth detection frame and any third detection frame being greater than a preset intersection-and-union ratio, it is determined that any first detection frame is successfully matched with any third detection frame.
[0013] Optionally, in response to the existence of an intersection-and-union ratio between any fourth detection frame and any third detection frame being greater than a preset intersection-and-union ratio, before the step of determining that any first detection frame matches any third detection frame successfully, the method further includes:
[0014] Obtaining a first target category corresponding to each fourth detection frame and a second target category corresponding to each third detection frame;
[0015] In response to the existence of a first target category corresponding to a first number of fourth detection frames, which is the same as a second target category corresponding to a second number of third detection frames, an intersection-over-union ratio between the first number of fourth detection frames and the second number of third detection frames is obtained.
[0016] Optionally, in response to the existence of an intersection-and-union ratio between any fourth detection frame and any third detection frame being greater than a preset intersection-and-union ratio, before the step of determining that any first detection frame matches any third detection frame successfully, the method further includes:
[0017] Obtaining the Euclidean distance between the fourth center point of each fourth detection frame and the third center point of each third detection frame;
[0018] Obtain an intersection-over-union ratio between each fourth detection frame and each third detection frame having the smallest Euclidean distance.
[0019] Optionally, determining an offset matrix between a second center point of each second detection frame and a center point of each first detection frame includes:
[0020] Obtaining the lateral offset between each second center point and each first center point to determine a lateral offset matrix, and the longitudinal deviation between each second center point and each first center point to determine a longitudinal offset matrix;
[0021] Using the offset matrix, each first detection frame is adjusted to obtain a fourth detection frame corresponding to each first detection frame, including:
[0022] Each first detection frame is translated using the horizontal offset matrix and the vertical offset matrix to obtain a fourth detection frame corresponding to each first detection frame.
[0023] Optionally, in response to the existence of an intersection-and-union ratio between any fourth detection frame and any third detection frame being greater than a preset intersection-and-union ratio, determining that any first detection frame matches any third detection frame successfully includes:
[0024] In response to the existence of an intersection-and-union ratio between any fourth detection frame and any third detection frame being greater than a preset intersection-and-union ratio, obtaining a first detection frame perimeter and a first detection frame area corresponding to any fourth detection frame, and a second detection frame perimeter and a second detection frame area corresponding to any third detection frame;
[0025] Determine a detection frame difference between any fourth detection frame and any third detection frame by using the first detection frame perimeter, the second detection frame perimeter, the first detection frame area, and the second detection frame area;
[0026] In response to the detection frame difference being less than or equal to the difference threshold, it is determined that any first detection frame is successfully matched with any third detection frame.
[0027] Optionally, determining the detection frame difference between any fourth detection frame and any third detection frame by using the first detection frame perimeter, the second detection frame perimeter, the first detection frame area, and the second detection frame area includes:
[0028] Obtaining a difference between the perimeter of the first detection frame and the perimeter of the second detection frame to obtain the perimeter difference, and a sum of the perimeter of the first detection frame and the perimeter of the second detection frame to obtain the perimeter sum;
[0029] Obtaining a difference between an area of the first detection frame and an area of the second detection frame to obtain an area difference, and a sum of an area of the first detection frame and an area of the second detection frame to obtain an area sum;
[0030] Calculate the quotient of the perimeter difference and the perimeter sum to obtain the first parameter, and the quotient of the area difference and the area sum to obtain the second parameter;
[0031] The weighted sum of the first parameter and the second parameter is calculated to obtain the detection box difference.
[0032] Optionally, using a homography matrix, projecting the at least one second detection frame into the first image to obtain at least one third detection frame of the at least one second detection frame in the first image includes:
[0033] Obtaining an inverse matrix of a second intrinsic parameter and at least one initial corner point of a second detection box in the second image;
[0034] Calculate the matrix product of the first inner parameter and the inverse matrix to obtain the homography matrix;
[0035] Using a homography matrix, projecting at least one initial corner point into the first image to obtain a projected corner point;
[0036] At least one third detection frame is obtained according to the at least one projection corner point.
[0037] Another technical solution adopted by the present application is to provide a terminal device, the terminal device comprising a memory and a processor connected to the memory;
[0038] The memory is used to store program data, and the processor is used to execute the program data to implement the above-mentioned image target matching method.
[0039] Another technical solution adopted by the present application is to provide a computer storage medium, which is used to store program data. When the program data is executed by a computer, it is used to implement the above-mentioned image target matching method.
[0040] The beneficial effects of the present application are: using the first intrinsic parameter of the first eyepiece camera and the second intrinsic parameter of the second eyepiece camera to determine the homography matrix; using the homography matrix to project at least one second detection frame from the second image to the first image to obtain at least one third detection frame; according to the positional relationship between each third detection frame and each first detection frame, determining the first matching relationship between each third detection frame and each first detection frame, and then determining the second matching relationship between at least one first detection frame and at least one second detection frame. The target matching method provided by the present application does not need to use camera extrinsic parameters to determine the homography matrix, reduces mismatching caused by inaccurate camera extrinsic parameters, and thus improves the accuracy and stability of target matching. Furthermore, it helps to improve the stability of subsequent multi-eye perception fusion algorithms. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0042] Figure 1 It is a flowchart of an embodiment of an image target matching method provided by the present application;
[0043] Figure 2 is a schematic diagram of an embodiment of a first image;
[0044] Figure 3 is a schematic diagram of an embodiment of a second image;
[0045] Figure 4 is a schematic diagram of an embodiment of a first detection frame in a first image;
[0046] Figure 5 is a schematic diagram of an embodiment of a second detection frame in a second image;
[0047] Figure 6 It is a flowchart of another embodiment of the image target matching method provided by the present application;
[0048] Figure 7 It is a structural diagram of an embodiment of a terminal device provided by the present application;
[0049] Figure 8 It is a structural diagram of an embodiment of a computer storage medium provided by the present application. DETAILED DESCRIPTION
[0050] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0051] The terms "first", "second", etc. in this application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods or devices.
[0052] In addition, although the terms "first", "second", etc. are used many times in this application to describe various data (or various applications or various instructions or various operations), etc., these data (or applications or instructions or operations) should not be limited by these terms. These terms are only used to distinguish one type of data (or application or instruction or operation) from another type of data (or application or instruction or operation).
[0053] Smart driving vehicles often use multi-cameras to perceive the surrounding driving environment. Multi-cameras include cameras with different focal lengths. Cameras with short focal lengths can perceive objects within a wider range, and cameras with long focal lengths can better perceive objects at a longer distance. In order to obtain better perception effects, the perception results (for example, target detection results) of cameras with different focal lengths are often fused.
[0054] In the related technology, the calibrated external parameters of the multi-eye camera are mainly used to calculate the homography matrix between the multi-eye images, and the detection frame of one image is projected into another image for matching. This method relies on the external parameters of the multi-eye camera. If the external parameters are inaccurate, it will cause a large error and affect the accuracy of the matching. In addition, with the long-term use of the multi-eye camera, the actual external parameters of the multi-eye camera will also change to a certain extent. In order to ensure the accuracy of the matching, it is necessary to perform complicated and lengthy camera calibration operations at intervals, which causes certain inconveniences.
[0055] This application mainly designs a set of methods to improve the accuracy of target matching. Unlike traditional methods, this application constructs a homography matrix from the perspective of camera intrinsic parameters, without using camera extrinsic parameters. Each second detection frame in the second image is projected into the first image, and the matching relationship between each first detection frame and each second detection frame is determined based on the positional relationship between the projected second detection frame and the first detection frame. This can avoid mismatching caused by inaccurate camera extrinsic parameters and improve the stability of the multi-camera perception fusion algorithm.
[0056] Please refer to Figure 1 , Figure 1 It is a flowchart of an embodiment of an image target matching method provided by the present application.
[0057] like Figure 1 As shown, the image target matching method of the embodiment of the present application may specifically include the following steps:
[0058] S1, obtaining a first image taken by a first camera in a multi-eye camera at the same time, and obtaining a second image taken by a second camera in the multi-eye camera, and obtaining at least one first detection frame corresponding to the first image, and at least one second detection frame corresponding to the second image.
[0059] The focal length corresponding to the first image is smaller than the focal length corresponding to the second image; and there is an image target in each detection frame.
[0060] The image target matching method provided in the present application is mainly performed by a target matching device. In some embodiments, the target matching device can be a multi-eye camera itself. In some embodiments, the target matching device can be a device connected to the multi-eye camera for communication. For example, the device can be a device for monitoring images, a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, and an autonomous vehicle, a robot, a security system, and any one or more products of glasses and helmets for augmented reality or virtual reality. In some possible implementations, the image target matching method can be implemented by a processor calling a computer-readable instruction stored in a memory.
[0061] The multi-eye camera mentioned in the embodiment of the present application has a camera number greater than or equal to 2. Exemplarily, the multi-eye camera can be a binocular camera or a trinocular camera. In some possible embodiments, at least one of the first eye camera and the second eye camera has a zoom function.
[0062] It can be understood that the multi-eye camera includes at least a first eye camera and a second eye camera, and the resolution of the first eye camera is the same as the resolution of the second eye camera.
[0063] In this embodiment, the field of view of the first camera at least partially overlaps with the field of view of the second camera.
[0064] Specifically, the target matching device obtains a first image taken by a first camera and a second image taken by a second camera at the same time. Figure 2 , see the second image Figure 3 , Figure 2 The box area in the Figure 3 It can be seen that since the focal length corresponding to the first image is smaller than the focal length corresponding to the second image, the actual area captured by the first image is larger than the actual area captured by the second image.
[0065] Furthermore, the target matching device performs target detection on the first image to obtain at least one first detection frame corresponding to the first image. Similarly, the target matching device performs target detection on the second image to obtain at least one second detection frame corresponding to the second image.
[0066] In some possible embodiments, the target matching device performs time synchronization processing on the first eyepiece camera and the second eyepiece camera to ensure that the first image and the second image are acquired at the same time.
[0067] In some possible embodiments, the object matching device uses an object detection method based on template matching to perform object detection on the first image and the second image respectively to obtain a first detection frame and a second detection frame.
[0068] In some possible embodiments, the target matching device inputs the first image into the target detection network to obtain a first detection frame. Specifically, the target detection network can detect image targets in the image, such as one or more types of objects such as vehicles, pedestrians, road signs or poles. In some embodiments, the output result of the target detection network may include a 2D detection frame and a category of each image target, wherein the detection frame output is in the form of the coordinates of the upper left corner and the lower right corner, and the category label is an integer, and different numbers represent different labels.
[0069] Exemplarily, at least one first detection frame corresponding to the first image is as follows: Figure 4 As shown, at least one second detection frame corresponding to the second image is as follows Figure 5 As shown. Understandably, Figure 4 and Figure 5 Although it only contains detection frames of (light) poles and (sign) signs, those skilled in the art can modify the output categories of the target detection network according to actual conditions to output detection frames corresponding to other different categories of targets. Figure 4 and Figure 5 The white frame in the figure is the detection frame. It should be noted that since the focal length of the first image is smaller than that of the second image, Figure 4 and Figure 5 They are in the same area, so Figure 4 Comparison Figure 5 Vague.
[0070] In some possible embodiments, the target matching device performs semantic segmentation on the first image to obtain a semantic segmentation result containing the image target, and further, the target matching device obtains a first detection frame corresponding to the image target based on the semantic segmentation result. Similarly, the target matching device can perform semantic segmentation on the second image in the above manner to obtain a second detection frame corresponding to the image target.
[0071] S2. Project the at least one second detection frame into the first image using a homography matrix to obtain at least one third detection frame of the at least one second detection frame in the first image.
[0072] The homography matrix is determined based on a first intrinsic parameter of the first eyepiece camera and a second intrinsic parameter of the second eyepiece camera.
[0073] In some embodiments, the homography matrix is determined based only on the first intrinsic parameter and the second intrinsic parameter.
[0074] In some embodiments, the homography matrix is determined without any camera extrinsic parameters.
[0075] Specifically, the target matching device projects at least one second detection frame from the second image to the first image according to the homography matrix to obtain at least one third detection frame of the at least one second detection frame in the first image.
[0076] In some embodiments, the number of the third detection frames is the same as the number of the second detection frames.
[0077] In some embodiments, the number of first detection frames in the first image is M, and the number of second detection frames in the second image is N. In some possible embodiments, the number of first detection frames is greater than or equal to the number of second detection frames.
[0078] S3, determining a first matching relationship between each first detection frame and each third detection frame according to a positional relationship between each first detection frame and each third detection frame.
[0079] Exemplarily, the target matching device obtains an intersection-and-union ratio between each first detection frame and each third detection frame, and determines the first detection frame and the third detection frame whose intersection-and-union ratio is greater than a certain threshold to form a first matching relationship.
[0080] It can be understood that there may be a first detection frame that does not match all the third detection frames.
[0081] It should be understood that, in the first matching relationship, a certain first detection frame matches less than or equal to one third detection frame.
[0082] S4: Based on the first matching relationship, determine a second matching relationship between at least one first detection frame and at least one second detection frame.
[0083] It can be understood that according to the homography matrix, a second detection frame corresponding to the third detection frame can be determined. Therefore, based on the first matching relationship, the target matching device can determine a second matching relationship between at least one first detection frame and at least one second detection frame.
[0084] The above scheme uses the first intrinsic parameter of the first eyepiece camera and the second intrinsic parameter of the second eyepiece camera to determine the homography matrix; uses the homography matrix to project at least one second detection frame from the second image to the first image to obtain at least one third detection frame; according to the positional relationship between each third detection frame and each first detection frame, the first matching relationship between each third detection frame and each first detection frame is determined, and then the second matching relationship between at least one first detection frame and at least one second detection frame is determined. The target matching method provided by the present application does not need to use camera extrinsic parameters to determine the homography matrix, reduces mismatching caused by inaccurate camera extrinsic parameters, and thus improves the accuracy and stability of target matching. Furthermore, it helps to improve the stability of subsequent multi-eye perception fusion algorithms.
[0085] Another embodiment of the image target matching method provided by the present application may specifically include the following steps:
[0086] S11, obtaining a first image taken by a first camera in a multi-eye camera at the same time, and obtaining a second image taken by a second camera in the multi-eye camera, and obtaining at least one first detection frame corresponding to the first image, and at least one second detection frame corresponding to the second image.
[0087] The focal length corresponding to the first image is smaller than the focal length corresponding to the second image; and there is an image target in each detection frame.
[0088] S12: Project the at least one second detection frame into the first image using a homography matrix to obtain at least one third detection frame of the at least one second detection frame in the first image.
[0089] The homography matrix is determined based on a first intrinsic parameter of the first eyepiece camera and a second intrinsic parameter of the second eyepiece camera.
[0090] S13, determining an offset matrix between a second center point of each second detection frame and a center point of each first detection frame.
[0091] Exemplarily, the target matching device calculates the offset between each second position and each first position based on the second position of each second center point in the second image and the first position of each first center point in the first image to determine the offset matrix.
[0092] S14, using the offset matrix, adjusting each first detection frame to obtain a fourth detection frame corresponding to each first detection frame.
[0093] In some embodiments, the target matching device uses an offset matrix to adjust the size and / or position of each first detection frame to obtain a fourth detection frame corresponding to each first detection frame.
[0094] S15, in response to the existence of an intersection-and-union ratio between any fourth detection frame and any third detection frame being greater than a preset intersection-and-union ratio, determining that any first detection frame matches any third detection frame successfully.
[0095] Specifically, the target matching device calculates the intersection-over-union ratio between each fourth detection frame and each third detection frame.
[0096] For a fourth detection frame, in response to an intersection-and-union ratio between the fourth detection frame and a third detection frame being greater than a preset intersection-and-union ratio, the target matching device determines that the fourth detection frame is successfully matched with the third detection frame.
[0097] Furthermore, the target matching device determines that the first detection frame corresponding to the fourth detection frame successfully matches the third detection frame.
[0098] In some possible embodiments, for a certain fourth detection frame, there may be multiple third detection frames whose intersection-and-union ratio is greater than a preset intersection-and-union ratio, and the target matching device selects the third detection frame with the largest intersection-and-union ratio as the third detection frame that matches the fourth detection frame.
[0099] S16: Determine a second matching relationship between at least one first detection frame and at least one second detection frame based on the first matching relationship.
[0100] The above scheme uses the first intrinsic parameter of the first eyepiece camera and the second intrinsic parameter of the second eyepiece camera to determine the homography matrix; uses the homography matrix to project at least one second detection frame from the second image to the first image to obtain at least one third detection frame; according to the positional relationship between each third detection frame and each first detection frame, the first matching relationship between each third detection frame and each first detection frame is determined, and then the second matching relationship between at least one first detection frame and at least one second detection frame is determined. The target matching method provided by the present application does not need to use camera extrinsic parameters to determine the homography matrix, reduces mismatching caused by inaccurate camera extrinsic parameters, and thus improves the accuracy and stability of target matching. Furthermore, it helps to improve the stability of subsequent multi-eye perception fusion algorithms.
[0101] Another embodiment of the image target matching method provided by the present application may specifically include the following steps:
[0102] S21, obtaining a first image taken by a first camera in the multi-eye camera at the same time, and obtaining a second image taken by a second camera in the multi-eye camera, and obtaining at least one first detection frame corresponding to the first image, and at least one second detection frame corresponding to the second image.
[0103] The focal length corresponding to the first image is smaller than the focal length corresponding to the second image; and there is an image target in each detection frame.
[0104] S22: Project the at least one second detection frame into the first image using a homography matrix to obtain at least one third detection frame of the at least one second detection frame in the first image.
[0105] The homography matrix is determined based on a first intrinsic parameter of the first eyepiece camera and a second intrinsic parameter of the second eyepiece camera.
[0106] S23, determining an offset matrix between a second center point of each second detection frame and a center point of each first detection frame.
[0107] S24, using the offset matrix, adjusting each first detection frame to obtain a fourth detection frame corresponding to each first detection frame.
[0108] S25, obtaining the first target category corresponding to each fourth detection frame and the second target category corresponding to each third detection frame.
[0109] In some embodiments, the target category corresponding to the fourth detection frame is the same as the target category corresponding to the first detection frame. Similarly, the target category corresponding to the third detection frame is the same as the target category corresponding to the second detection frame.
[0110] The target category corresponds to the category of the above-mentioned image target. For example, the target category may include but is not limited to a pole and a sign.
[0111] S26, in response to the existence of a first target category corresponding to the first number of fourth detection frames, which is the same as the second target category corresponding to the second number of third detection frames, obtaining an intersection-over-union ratio between the first number of fourth detection frames and the second number of third detection frames.
[0112] Specifically, in response to the existence of a first target category corresponding to a first number of fourth detection frames, which is the same as a second target category corresponding to a second number of third detection frames, the target matching device obtains the intersection-and-union ratio between the first number of fourth detection frames and the second number of third detection frames.
[0113] In some embodiments, the first amount may be equal to the second amount.
[0114] It can be understood that if the first target category corresponding to a fourth detection frame is different from the second target category corresponding to a third detection frame, it means that the fourth detection frame does not match the third detection frame.
[0115] Through the above methods, the probability of mismatching of detection frames is reduced.
[0116] S27, in response to the existence of an intersection-and-union ratio between any fourth detection frame and any third detection frame being greater than a preset intersection-and-union ratio, determining that any first detection frame is successfully matched with any third detection frame.
[0117] S28: Determine a second matching relationship between at least one first detection frame and at least one second detection frame based on the first matching relationship.
[0118] The above scheme uses the first intrinsic parameter of the first eyepiece camera and the second intrinsic parameter of the second eyepiece camera to determine the homography matrix; uses the homography matrix to project at least one second detection frame from the second image to the first image to obtain at least one third detection frame; according to the positional relationship between each third detection frame and each first detection frame, the first matching relationship between each third detection frame and each first detection frame is determined, and then the second matching relationship between at least one first detection frame and at least one second detection frame is determined. The target matching method provided by the present application does not need to use camera extrinsic parameters to determine the homography matrix, reduces mismatching caused by inaccurate camera extrinsic parameters, and thus improves the accuracy and stability of target matching. Furthermore, it helps to improve the stability of subsequent multi-eye perception fusion algorithms.
[0119] Furthermore, by calculating the intersection-over-union ratio between the fourth detection frame and the third detection frame, which have the same first target category as the second target category, mismatching between the fourth detection frame and the third detection frame of different categories can be avoided.
[0120] Another embodiment of the image target matching method provided by the present application may specifically include the following steps:
[0121] S31, obtaining a first image taken by a first camera in the multi-eye camera at the same time, and obtaining a second image taken by a second camera in the multi-eye camera, and obtaining at least one first detection frame corresponding to the first image, and at least one second detection frame corresponding to the second image.
[0122] The focal length corresponding to the first image is smaller than the focal length corresponding to the second image; and there is an image target in each detection frame.
[0123] S32: Project the at least one second detection frame into the first image using a homography matrix to obtain at least one third detection frame of the at least one second detection frame in the first image.
[0124] The homography matrix is determined based on a first intrinsic parameter of the first eyepiece camera and a second intrinsic parameter of the second eyepiece camera.
[0125] S33, determining an offset matrix between a second center point of each second detection frame and a center point of each first detection frame.
[0126] S34, using the offset matrix, adjusting each first detection frame to obtain a fourth detection frame corresponding to each first detection frame.
[0127] S35, obtaining the Euclidean distance between the fourth center point of each fourth detection frame and the third center point of each third detection frame.
[0128] In some embodiments, the object matching device obtains the fourth center point in each fourth detection frame and the third center point in each third detection frame, and calculates the Euclidean distance between each fourth center point and each third center point.
[0129] S36, obtaining an intersection-over-union ratio between each fourth detection frame and each third detection frame having the smallest Euclidean distance.
[0130] Specifically, the target matching device obtains the intersection-and-union ratio between each fourth detection frame and each third detection frame with the smallest Euclidean distance.
[0131] It can be understood that the smaller the Euclidean distance is, the closer the positions of the third detection frame and the fourth detection frame are, and the third detection frame is likely to match the fourth detection frame, that is, the fourth detection frame with the smallest Euclidean distance and the third detection frame are candidate matches.
[0132] Through the above methods, the probability of mismatching of detection frames is reduced.
[0133] S37, in response to the existence of an intersection-and-union ratio between any fourth detection frame and any third detection frame being greater than a preset intersection-and-union ratio, determining that any first detection frame matches any third detection frame successfully.
[0134] Furthermore, the target matching device calculates the intersection-and-union ratio between any fourth detection frame and any third detection frame constituting a candidate match, and if the intersection-and-union ratio is less than a preset intersection-and-union ratio, it is determined that the fourth detection frame and the third detection frame do not constitute a match. That is, the first detection frame and the third detection frame do not constitute a match (matching failure).
[0135] S38: Determine a second matching relationship between at least one first detection frame and at least one second detection frame based on the first matching relationship.
[0136] The above scheme uses the first intrinsic parameter of the first eyepiece camera and the second intrinsic parameter of the second eyepiece camera to determine the homography matrix; uses the homography matrix to project at least one second detection frame from the second image to the first image to obtain at least one third detection frame; according to the positional relationship between each third detection frame and each first detection frame, the first matching relationship between each third detection frame and each first detection frame is determined, and then the second matching relationship between at least one first detection frame and at least one second detection frame is determined. The target matching method provided by the present application does not need to use camera extrinsic parameters to determine the homography matrix, reduces mismatching caused by inaccurate camera extrinsic parameters, and thus improves the accuracy and stability of target matching. Furthermore, it helps to improve the stability of subsequent multi-eye perception fusion algorithms.
[0137] Furthermore, by calculating the intersection-over-union ratio between the fourth detection frame and the third detection frame with the smallest Euclidean distance, the probability of mismatching is effectively reduced.
[0138] Another embodiment of the image target matching method provided by the present application may specifically include the following steps:
[0139] S41, obtaining a first image taken by a first camera in the multi-eye camera at the same time, and obtaining a second image taken by a second camera in the multi-eye camera, and obtaining at least one first detection frame corresponding to the first image, and at least one second detection frame corresponding to the second image.
[0140] The focal length corresponding to the first image is smaller than the focal length corresponding to the second image; and there is an image target in each detection frame.
[0141] S42: Project the at least one second detection frame into the first image using a homography matrix to obtain at least one third detection frame of the at least one second detection frame in the first image.
[0142] The homography matrix is determined based on a first intrinsic parameter of the first eyepiece camera and a second intrinsic parameter of the second eyepiece camera.
[0143] S43, obtaining the lateral offset between each second center point and each first center point to determine the lateral offset matrix, and the longitudinal deviation between each second center point and each first center point to determine the longitudinal offset matrix.
[0144] It should be noted that, since the number of first detection frames is M and the number of second detection frames is N, the dimension of the horizontal offset matrix is M*N, and the dimension of the vertical offset matrix is M*N.
[0145] In some embodiments, the first center point can be expressed as (U short , V short )The second center point can be expressed as (U long , V long ).
[0146] In some embodiments, each element in the lateral offset matrix corresponds to a difference between a second horizontal coordinate of each second center point and a first horizontal coordinate of each first center point.
[0147] Exemplarily, for the i-th first detection frame and the j-th second detection frame, the elements of the corresponding lateral offset matrix satisfy the following relationship:
[0148] U Shift_mat [i, j] = U long_j -U short_i
[0149] Among them, UShift_mat is the lateral offset matrix, U long_j is the second horizontal coordinate of the jth second detection box, U short_i is the first horizontal coordinate of the i-th first detection box.
[0150] In some embodiments, each element in the longitudinal offset matrix corresponds to a difference between a second longitudinal coordinate of each second center point and a first longitudinal coordinate of each first center point.
[0151] Exemplarily, for the i-th first detection frame and the j-th second detection frame, the elements of the corresponding longitudinal offset matrix satisfy the following relationship:
[0152] V Shift_mat [i, j] = V long_j -V short_i
[0153] V Shift_mat is the longitudinal offset matrix, V long_j is the second ordinate of the jth second detection box, V short_i is the first vertical coordinate of the i-th first detection box.
[0154] S44, using the horizontal offset matrix and the vertical offset matrix, translate each first detection frame to obtain a fourth detection frame corresponding to each first detection frame.
[0155] In some embodiments, the target matching device uses the first element corresponding to the horizontal offset matrix to perform horizontal translation on the upper left corner point and the lower right corner point in each first detection frame to determine each detection frame after the horizontal translation. Furthermore, the target matching device uses the second element corresponding to the vertical offset matrix to perform vertical translation on the upper left corner point and the lower right corner point of each detection frame after the horizontal translation to obtain a fourth detection frame corresponding to each first detection frame.
[0156] In some embodiments, the horizontal coordinate of each fourth detection frame is the sum of the horizontal coordinate of each first detection frame and the first element corresponding to the horizontal offset matrix.
[0157] Exemplarily, the horizontal coordinate of the i-th fourth detection box satisfies the following relationship:
[0158]
[0159] in, is the horizontal coordinate of the fourth detection box of the ith short_i is the horizontal coordinate of the i-th first detection box, U Shift_mat [i, j] is the first element.
[0160] In some embodiments, the vertical coordinate of each fourth detection frame is the sum of the vertical coordinate of each first detection frame and the second element corresponding to the vertical offset matrix.
[0161] Exemplarily, the vertical coordinate of the i-th fourth detection frame satisfies the following relationship:
[0162] V short_shift_i =V short_i +V Shift_mat [i, j]
[0163] Among them, V short_shift_i The vertical coordinate of the fourth detection box of the i-th frame, V short_i is the vertical coordinate of the i-th first detection box, V Shift_mat [i, j] is the second element.
[0164] S45, in response to the existence of an intersection-and-union ratio between any fourth detection frame and any third detection frame being greater than a preset intersection-and-union ratio, determining that any first detection frame is successfully matched with any third detection frame.
[0165] S46: Determine a second matching relationship between at least one first detection frame and at least one second detection frame based on the first matching relationship.
[0166] The above scheme uses the first intrinsic parameter of the first eyepiece camera and the second intrinsic parameter of the second eyepiece camera to determine the homography matrix; uses the homography matrix to project at least one second detection frame from the second image to the first image to obtain at least one third detection frame; according to the positional relationship between each third detection frame and each first detection frame, the first matching relationship between each third detection frame and each first detection frame is determined, and then the second matching relationship between at least one first detection frame and at least one second detection frame is determined. The target matching method provided by the present application does not need to use camera extrinsic parameters to determine the homography matrix, reduces mismatching caused by inaccurate camera extrinsic parameters, and thus improves the accuracy and stability of target matching. Furthermore, it helps to improve the stability of subsequent multi-eye perception fusion algorithms.
[0167] Another embodiment of the image target matching method provided by the present application may specifically include the following steps:
[0168] S51, obtaining a first image taken by a first camera in the multi-eye camera at the same time, and obtaining a second image taken by a second camera in the multi-eye camera, and obtaining at least one first detection frame corresponding to the first image, and at least one second detection frame corresponding to the second image.
[0169] The focal length corresponding to the first image is smaller than the focal length corresponding to the second image; and there is an image target in each detection frame.
[0170] S52: Project the at least one second detection frame into the first image using a homography matrix to obtain at least one third detection frame of the at least one second detection frame in the first image.
[0171] The homography matrix is determined based on a first intrinsic parameter of the first eyepiece camera and a second intrinsic parameter of the second eyepiece camera.
[0172] S53, determining an offset matrix between a second center point of each second detection frame and a center point of each first detection frame.
[0173] S54, using the offset matrix, adjusting each first detection frame to obtain a fourth detection frame corresponding to each first detection frame.
[0174] S55, in response to the existence of an intersection-and-union ratio between any fourth detection frame and any third detection frame being greater than a preset intersection-and-union ratio, obtain the first detection frame perimeter and the first detection frame area corresponding to any fourth detection frame, and the second detection frame perimeter and the second detection frame area corresponding to any third detection frame.
[0175] In some embodiments, in response to the intersection-and-union ratio of any fourth detection frame to any third detection frame being greater than or equal to a preset intersection-and-union ratio, the target matching device obtains the first detection frame perimeter and the first detection frame area corresponding to any fourth detection frame, and the second detection frame perimeter and the second detection frame area corresponding to any third detection frame.
[0176] In some possible embodiments, the perimeter of the first detection frame can be expressed as C box1 , the perimeter of the second detection box can be expressed as C box2 , the area of the first detection box can be expressed as S box1 , the area of the second detection box can be expressed as S box2 .
[0177] In some possible embodiments, the perimeter of the first detection frame can be expressed as C box2 , the perimeter of the second detection box can be expressed as C box1 , the area of the first detection box can be expressed as S box2 , the area of the second detection box can be expressed as S box1 .
[0178] S56, determining the detection frame difference between any fourth detection frame and any third detection frame by using the first detection frame perimeter, the second detection frame perimeter, the first detection frame area, and the second detection frame area.
[0179] In some embodiments, the target matching device determines the detection frame difference between any fourth detection frame and any third detection frame using the first detection frame perimeter, the second detection frame perimeter, the first detection frame area, and the second detection frame area.
[0180] Specifically, the detection frame difference is used to indicate the size difference between the fourth detection frame and the third detection frame. The larger the detection frame difference, the larger the size difference between the fourth detection frame and the third detection frame, indicating that the fourth detection frame and the third detection frame may not be matched.
[0181] In some embodiments, S56 may specifically include the following steps:
[0182] S561, obtaining a difference between the perimeter of the first detection frame and the perimeter of the second detection frame to obtain the perimeter difference, and a sum of the perimeter of the first detection frame and the perimeter of the second detection frame to obtain the perimeter sum.
[0183] Specifically, the target matching device obtains the absolute value of the difference between the perimeter of the first detection frame and the perimeter of the second detection frame to obtain the perimeter difference, and obtains the sum of the perimeter of the first detection frame and the perimeter of the second detection frame to obtain the perimeter sum.
[0184] S562, obtaining a difference between the area of the first detection frame and the area of the second detection frame to obtain an area difference, and a sum of the area of the first detection frame and the area of the second detection frame to obtain an area sum.
[0185] Specifically, the target matching device obtains the absolute value of the difference between the area of the first detection frame and the area of the second detection frame to obtain the area difference, and obtains the sum of the area of the first detection frame and the area of the second detection frame to obtain the area sum.
[0186] S563, calculating the quotient of the perimeter difference and the perimeter sum to obtain a first parameter, and calculating the quotient of the area difference and the area sum to obtain a second parameter.
[0187] Specifically, the first parameter satisfies the following relationship:
[0188] P1=|C box1 -C box2 | / (C box1 +C box2 )
[0189] Among them, P1 is the first parameter, |C box1 -C box2 | is the circumference difference, (C box1 +C box2 ) is the sum of the circumference.
[0190] Specifically, the second parameter satisfies the following relationship:
[0191] P2=|S box1 -S box2 | / (S box1 +S box2 )
[0192] Among them, P2 is the second parameter, |S box1 -Sbox2 | is the area difference, (S box1 +S box2 ) is the area and .
[0193] S564: Calculate a weighted sum of the first parameter and the second parameter to obtain a detection box difference.
[0194] Specifically, the difference of the detection box satisfies the following relationship:
[0195] Diff=w1*P1+w2*P2
[0196] Wherein, Diff is the difference of the detection frame, w1 is the first weight corresponding to the first parameter, and w2 is the second weight corresponding to the second parameter. In some embodiments, w1+w2=1 is satisfied.
[0197] For example, w1=w2=0.5, then the difference between the detection frames satisfies the following relationship:
[0198] Diff=0.55*P1+0.55*P2=
[0199] 0.5*|C box1 -C box2 | / (C box1 +C box2 )+0.5*|S box1 -S box2 | / (S box1 +S box2 )
[0200] S57: In response to the detection frame difference being less than or equal to the difference threshold, determining that any first detection frame matches any third detection frame successfully.
[0201] It can be understood that when the condition that the intersection-and-union ratio between a third detection frame and a fourth detection frame is greater than the preset intersection-and-union ratio is met, the smaller the detection frame difference between the third detection frame and the fourth detection frame, the closer the size of the third detection frame is to the fourth detection frame.
[0202] In some embodiments, in response to a detection frame difference being less than or equal to a difference threshold, the target matching device determines that any fourth detection frame successfully matches any third detection frame. Further, the target matching device determines that the first detection frame corresponding to the fourth detection frame successfully matches the third detection frame.
[0203] In some embodiments, in response to the difference between the detection frames being less than or equal to a difference threshold, the target matching device determines that the fourth detection frame and the third detection frame are matched as a group. Further, the target matching device determines that the first detection frame corresponding to the fourth detection frame and the third detection frame are matched as a group.
[0204] In some possible embodiments, in response to a detection frame difference being greater than a difference threshold, the target matching device determines that the fourth detection frame fails to match the third detection frame, and further, the target matching device determines that the first detection frame corresponding to the fourth detection frame fails to match the third detection frame.
[0205] S58: Determine a second matching relationship between at least one first detection frame and at least one second detection frame based on the first matching relationship.
[0206] The above scheme uses the first intrinsic parameter of the first eyepiece camera and the second intrinsic parameter of the second eyepiece camera to determine the homography matrix; uses the homography matrix to project at least one second detection frame from the second image to the first image to obtain at least one third detection frame; according to the positional relationship between each third detection frame and each first detection frame, the first matching relationship between each third detection frame and each first detection frame is determined, and then the second matching relationship between at least one first detection frame and at least one second detection frame is determined. The target matching method provided by the present application does not need to use camera extrinsic parameters to determine the homography matrix, reduces mismatching caused by inaccurate camera extrinsic parameters, and thus improves the accuracy and stability of target matching. Furthermore, it helps to improve the stability of subsequent multi-eye perception fusion algorithms.
[0207] Furthermore, by calculating the detection frame difference between the fourth detection frame that may form a match and the third detection frame, the probability of a false match is further reduced.
[0208] Another embodiment of the image target matching method provided by the present application may specifically include the following steps:
[0209] S61, obtaining a first image taken by a first camera in the multi-eye camera at the same time, and obtaining a second image taken by a second camera in the multi-eye camera, and obtaining at least one first detection frame corresponding to the first image, and at least one second detection frame corresponding to the second image.
[0210] The focal length corresponding to the first image is smaller than the focal length corresponding to the second image; and there is an image target in each detection frame.
[0211] S62, obtaining an inverse matrix of a second intrinsic parameter and at least one initial corner point of a second detection box in the second image.
[0212] Specifically, the target matching device obtains the internal reference of the second camera, that is, the second internal reference K long It can be understood that the second internal reference can be determined by calibrating the second eyepiece camera in advance. Similarly, the target matching device obtains the internal reference of the first eyepiece camera, that is, the first internal reference K short .
[0213] In some embodiments, the first intrinsic parameter and the second intrinsic parameter may be determined by performing a calibration operation on the multi-eye camera.
[0214] Furthermore, the target matching device obtains the inverse matrix of the second internal parameter
[0215] In some embodiments, the initial corner point may be represented as (u1, v1).
[0216] S63, calculating the matrix product of the first inner parameter and the inverse matrix to obtain a homography matrix.
[0217] Specifically, the homography matrix satisfies the following relationship:
[0218]
[0219] Among them, H short_long is the homography matrix, K short is the first internal reference, is the inverse matrix.
[0220] S64, using the homography matrix, projecting at least one initial corner point into the first image to obtain at least one projected corner point.
[0221] It should be noted that the initial corner points here include at least the upper left corner point and the lower right corner point of the second detection frame. Here, an initial corner point can be understood as a group of corner points.
[0222] In some embodiments, each projection corner point may be determined by a matrix product of a homography matrix and each initial corner point.
[0223] In some embodiments, the projection corners satisfy the following relationship:
[0224] [u2,v2,1] T =H short_long *[u1,v1,1] T
[0225] Among them, the projection corner point can be expressed as (u2,v2).
[0226] S65: Obtain at least one third detection frame according to the at least one projection corner point.
[0227] In some embodiments, each group of initial corner points includes the upper left corner point and the lower right corner point of the first detection frame. Then each group of projected corner points includes two projected corner points, and the target matching device determines a third detection frame based on these two points, and further determines all third detection frames.
[0228] In some embodiments, each group of initial corner points includes the upper left corner point, the upper right corner point, the lower left corner point, and the lower right corner point of the first detection frame. Then each group of projected corner points includes four projected corner points, and the target matching device determines a third detection frame based on these four points, and further determines all third detection frames.
[0229] S66: Determine a first matching relationship between each first detection frame and each third detection frame according to the positional relationship between each first detection frame and each third detection frame.
[0230] S67: Determine a second matching relationship between at least one first detection frame and at least one second detection frame based on the first matching relationship.
[0231] The above scheme uses the first intrinsic parameter of the first eyepiece camera and the second intrinsic parameter of the second eyepiece camera to determine the homography matrix; uses the homography matrix to project at least one second detection frame from the second image to the first image to obtain at least one third detection frame; according to the positional relationship between each third detection frame and each first detection frame, the first matching relationship between each third detection frame and each first detection frame is determined, and then the second matching relationship between at least one first detection frame and at least one second detection frame is determined. The target matching method provided by the present application does not need to use camera extrinsic parameters to determine the homography matrix, reduces mismatching caused by inaccurate camera extrinsic parameters, and thus improves the accuracy and stability of target matching. Furthermore, it helps to improve the stability of subsequent multi-eye perception fusion algorithms.
[0232] See also Figure 6 , Figure 6 It is a flowchart of another embodiment of the image target matching method provided by the present application.
[0233] like Figure 6 As shown, another embodiment of the image target matching method provided by the present application may specifically include the following steps:
[0234] S101, image acquisition.
[0235] In this embodiment, the multi-eye camera is a binocular camera.
[0236] Specifically, the target matching device collects binocular images taken by a binocular camera. It is understandable that the binocular camera needs time synchronization to ensure that the two images are collected at the same time. Among them, the binocular camera includes a first binocular camera and a second binocular camera. Further, the target matching device collects a first image taken by the first binocular camera and a second image taken by the second binocular camera at the same time, and the focal length corresponding to the first image is smaller than the focal length of the second image.
[0237] In some embodiments, the focal length of the first camera is smaller than the focal length of the second camera.
[0238] The first and second eye images can be found in Figure 2 and Figure 3 It can be found that Figure 3 The actual area captured is approx. Figure 2 The white frame part in the figure, but the resolution of the two sets of images is the same, so the telephoto camera can observe distant targets more clearly.
[0239] S102, target detection.
[0240] Specifically, the target matching device uses a neural network to perform target detection on the first image and the second image respectively. Exemplarily, target detection can be performed on objects such as vehicles, pedestrians, road signs, and poles.
[0241] The output of the neural network is a 2D detection box of each object and the category to which the detection box belongs. In some embodiments, the detection box output is in the form of two point coordinates of the upper left corner and the lower right corner, and the category label is an integer, and different numbers represent different labels.
[0242] The detection results corresponding to the first image and the second image are as follows Figure 4 , Figure 5 As shown in the figure, the two images select parts of the same actual space area size. The white box is the detection box output by the neural network. Although the figure only contains the pole detection box, other types of detection boxes also meet the requirements.
[0243] S103, calculating a transverse offset matrix and a longitudinal offset matrix.
[0244] In some embodiments, it is assumed that the short-focus image detection box Boxs short There are M telephoto image detection boxes long There are N. Among them, the short-focus image detection frame corresponds to the first detection frame above, and the long-focus image detection frame corresponds to the second detection frame above.
[0245] The target matching device initializes two matrices U_Shift_Mat and V_Shift_Mat of dimension M*N, and transforms the telephoto image detection box Boxs long Use the homography matrix to project onto the first image and get the telephoto projection box Boxs on the first image project_long , where the telephoto projection frame corresponds to the third detection frame mentioned above, U_Shift_Mat corresponds to the horizontal offset matrix mentioned above, and V_Shift_Mat corresponds to the vertical offset matrix mentioned above.
[0246] In some embodiments, the target matching device obtains the homography matrix H from the long-focus camera pixel to the short-focus camera image short_long , where the homography matrix satisfies the following relationship:
[0247]
[0248] Among them, K short is the internal parameter matrix of the short-focus camera, K long is the internal parameter matrix of the telephoto camera. The short-focus camera here corresponds to the first camera above, and the telephoto camera corresponds to the second camera above.
[0249] Furthermore, the target matching device uses the homography matrix to project the upper left corner coordinates and the lower right corner coordinates of the telephoto image detection frame onto the short-focus image to obtain a new image frame, namely, the telephoto image projection frame. The telephoto image corresponds to the second image above, the short-focus image corresponds to the first image above, and the telephoto image projection frame corresponds to the third projection frame above.
[0250] Furthermore, the coordinates of the telephoto image projection frame satisfy the following relationship:
[0251] [u2, v2, 1] T =H wide_narrow *[u1,v1,1] T
[0252] Among them, [u1,v1,1] T is the coordinate of the telephoto image detection frame, [u2,v2,1] is the coordinate of the telephoto image projection frame
[0253] Furthermore, the target matching device double loops through the M short-focus image frames and the N long-focus image frames, and for the i-th short-focus image frame Boxs[i] short and the jth telephoto image frame Boxs[j] long , calculate the horizontal coordinate offset U_Shift and the vertical coordinate offset V_Shift of the center points of the two boxes, and assign values to the U_Shift_Mat and V_Shift_Mat matrices.
[0254] Among them, the i-th row and j-th column element in the U_Shift_Mat matrix and the i-th row and j-th column element in the V_Shift_Mat matrix respectively satisfy the following relationship:
[0255] U Shift_mat [i, j] = U long -U short
[0256] V Shift_mat [i, j] = V long -V short
[0257] Among them, U long is the horizontal coordinate of the center point of the telephoto image frame, V longis the ordinate of the center point of the telephoto image frame, U short is the horizontal coordinate of the center point of the short-focus image frame, V short It is the ordinate of the center point of the short-focus image frame.
[0258] S104, calculating the optimal match.
[0259] Specifically, the target matching device defines an M*N Score_Mat matrix, double loops through M*N times, and Boxes short All Box horizontal coordinates plus U Shift_mat [i, j] ordinate plus V Shift_mat [i, j] get Boxes short_shift Among them, Boxes short_shift Corresponding to the fourth detection box above.
[0260] Furthermore, the target matching device will Boxes project_long and Boxes short_shift Score is calculated as input and assigned to the Score_Mat matrix. For the i-th row and j-th column element in the Score_Mat matrix, the following relationship is satisfied:
[0261] Score mat [ij]=Score
[0262] In some embodiments, for Boxes short_shift Each detection box, if it can be in Boxs project_long Find the detection boxes belonging to the same target category, and the target matching device calculates the IOU (IOU is the maximum intersection-over-union ratio) between the two, and uses the IOU as the corresponding Score value. project_long The Score value corresponding to the detection box that does not belong to the same target category is set to 0. Furthermore, the target matching device averages all elements in Score_Mat, that is, Score is equal to Mean_IOU;
[0263] Furthermore, the target matching device determines the Score mat [i, j] The maximum value of U shift_mat [i, j] and V shift_mat [i, j], for Boxes short_shift For each image frame, the target matching device traverses Boxs short_shift Each image box, if it can be in Boxs project_long Find the same type of detection box, and further find the same type of detection box with the closest center point (smallest Euclidean distance). If the IOU of two detection boxes is greater than 0.9, the two detection boxes are determined as a set of candidate matches.
[0264] Furthermore, the target matching device calculates the difference Diff between two detection frames for each set of candidate matches found, and the calculation method is as follows:
[0265]
[0266] Among them, S represents the area of the detection box, and C represents the perimeter of the detection box. If the difference Diff is greater than a certain threshold, it is determined to be a false match and the group of candidate matches is removed. Among them, the difference corresponds to the detection box difference mentioned above.
[0267] S105: Output the matching result.
[0268] Understandably, Boxes short_shift 、Boxes project_long The order of the detection boxes and the original Boxes short , Boxes long Consistent, so if Boxes short_shift The ath detection box and Boxes project_long The bth detection box in is a set of matches, Boxes short The ath detection box and Boxes l0ng The bth detection box in is also a set of matches, so the target matching device outputs all matching detection boxes.
[0269] The above scheme uses the first intrinsic parameter of the first eyepiece camera and the second intrinsic parameter of the second eyepiece camera to determine the homography matrix; uses the homography matrix to project at least one second detection frame from the second image to the first image to obtain at least one third detection frame; according to the positional relationship between each third detection frame and each first detection frame, the first matching relationship between each third detection frame and each first detection frame is determined, and then the second matching relationship between at least one first detection frame and at least one second detection frame is determined. The target matching method provided by the present application does not need to use camera extrinsic parameters to determine the homography matrix, reduces mismatching caused by inaccurate camera extrinsic parameters, and thus improves the accuracy and stability of target matching. Furthermore, it helps to improve the stability of subsequent multi-eye perception fusion algorithms.
[0270] Please continue to see Figure 7 , Figure 7 The terminal device 500 of the embodiment of the present application includes a processor 51 and a memory 52 .
[0271] The processor 51 and the memory 52 are connected to the bus. The memory 52 stores program data. The processor 51 is used to execute the program data to implement the image target matching method described in the above embodiment.
[0272] In the embodiment of the present application, the processor 51 may also be referred to as a CPU (Central Processing Unit). The processor 51 may be an integrated circuit chip having the ability to process signals. The processor 51 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or the processor 51 may also be any conventional processor, etc.
[0273] This application also provides a computer storage medium, please continue to refer to Figure 8 , Figure 8 It is a schematic diagram of the structure of an embodiment of a computer storage medium provided in the present application. The computer storage medium 600 stores program data 61. When the program data 61 is executed by a processor, it is used to implement the image target matching method of the above embodiment.
[0274] When the embodiments of the present application are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk.
[0275] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Equivalent structures or equivalent process changes made by utilizing the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.
Claims
1. A method for matching an image target, characterized in that: The method comprises: Acquire a first image taken by a first camera in a multi-camera at the same time, and acquire a second image taken by a second camera in the multi-camera, and obtain at least one first detection frame corresponding to the first image, and at least one second detection frame corresponding to the second image; wherein the focal length corresponding to the first image is smaller than the focal length corresponding to the second image; and there is an image target in each detection frame; Using a homography matrix, projecting at least one of the second detection frames into the first image, to obtain at least one third detection frame of the at least one of the second detection frames in the first image; wherein the homography matrix is determined based on a first intrinsic parameter of the first eyepiece camera and a second intrinsic parameter of the second eyepiece camera; According to the positional relationship between each of the first detection frames and each of the third detection frames, a first matching relationship between each of the first detection frames and each of the third detection frames is determined; and based on the first matching relationship, a second matching relationship between at least one of the first detection frames and at least one of the second detection frames is determined.
2. The method according to claim 1, characterized in that The determining, according to the positional relationship between each of the first detection frames and each of the third detection frames, a first matching relationship between each of the first detection frames and each of the third detection frames comprises: Determine an offset matrix between a second center point of each of the second detection frames and a first center point of each of the first detection frames; Using the offset matrix, adjusting each of the first detection frames to obtain a fourth detection frame corresponding to each of the first detection frames; In response to the existence of an intersection-and-union ratio between any fourth detection frame and any third detection frame being greater than a preset intersection-and-union ratio, it is determined that any first detection frame is successfully matched with any third detection frame.
3. The method according to claim 2, characterized in that Before the step of determining that any first detection frame successfully matches any third detection frame in response to the existence of an intersection-and-union ratio between any fourth detection frame and any third detection frame being greater than a preset intersection-and-union ratio, the step further includes: Acquire a first target category corresponding to each of the fourth detection frames, and a second target category corresponding to each of the third detection frames; In response to the existence of a first number of fourth detection frames corresponding to the first target category, which is the same as the second target category corresponding to a second number of third detection frames, an intersection-over-union ratio between the first number of fourth detection frames and the second number of third detection frames is obtained.
4. The method according to claim 2, characterized in that: Before the step of determining that any first detection frame successfully matches any third detection frame in response to the existence of an intersection-and-union ratio between any fourth detection frame and any third detection frame being greater than a preset intersection-and-union ratio, the step further includes: Obtaining the Euclidean distance between the fourth center point of each of the fourth detection frames and the third center point of each of the third detection frames; Obtain an intersection-and-union ratio between each of the fourth detection frames and each of the third detection frames having the smallest Euclidean distance.
5. The method according to claim 2, characterized in that: The determining of an offset matrix between a second center point of each second detection frame and a first center point of each first detection frame comprises: Acquire the lateral offset between each of the second center points and each of the first center points to determine a lateral offset matrix, and the longitudinal deviation between each of the second center points and each of the first center points to determine a longitudinal offset matrix; The using the offset matrix to adjust each of the first detection frames to obtain a fourth detection frame corresponding to each of the first detection frames includes: Each of the first detection frames is translated using the horizontal offset matrix and the vertical offset matrix to obtain the fourth detection frame corresponding to each of the first detection frames.
6. The method according to claim 2, characterized in that In response to the existence of an intersection-and-union ratio between any fourth detection frame and any third detection frame being greater than a preset intersection-and-union ratio, determining that any first detection frame successfully matches any third detection frame includes: In response to the existence of an intersection-and-union ratio between any of the fourth detection frames and any of the third detection frames being greater than a preset intersection-and-union ratio, obtaining a first detection frame perimeter and a first detection frame area corresponding to any of the fourth detection frames, and a second detection frame perimeter and a second detection frame area corresponding to any of the third detection frames; Determine a detection frame difference between any fourth detection frame and any third detection frame by using the first detection frame perimeter, the second detection frame perimeter, the first detection frame area, and the second detection frame area; In response to the detection frame difference being less than or equal to a difference threshold, it is determined that any one of the first detection frames is successfully matched with any one of the third detection frames.
7. The method according to claim 6, characterized in that The determining the detection frame difference between the fourth detection frame and the third detection frame by using the first detection frame perimeter, the second detection frame perimeter, the first detection frame area, and the second detection frame area includes: Obtaining a difference between the perimeter of the first detection frame and the perimeter of the second detection frame to obtain a perimeter difference, and a sum of the perimeter of the first detection frame and the perimeter of the second detection frame to obtain a perimeter sum; Obtaining a difference between an area of the first detection frame and an area of the second detection frame to obtain an area difference, and a sum of an area of the first detection frame and an area of the second detection frame to obtain an area sum; Calculate the quotient of the perimeter difference and the perimeter sum to obtain a first parameter, and calculate the quotient of the area difference and the area sum to obtain a second parameter; A weighted sum of the first parameter and the second parameter is calculated to obtain the detection box difference.
8. The method according to claim 1, characterized in that The using the homography matrix to project at least one of the second detection frames into the first image to obtain at least one third detection frame of the at least one of the second detection frames in the first image includes: Obtaining an inverse matrix of the second intrinsic parameter and at least one initial corner point of the second detection box in the second image; Calculate the matrix product of the first inner matrix and the inverse matrix to obtain the homography matrix; Using the homography matrix, projecting at least one of the initial corner points into the first image to obtain a projected corner point; At least one of the third detection frames is obtained according to at least one of the projection corner points.
9. A terminal device, characterized in that: The terminal device includes a processor and a memory connected to the processor, wherein: The memory stores program instructions; The processor is configured to execute program instructions stored in the memory to implement the method according to any one of claims 1 to 8.
10. A computer-readable storage medium, characterized in that: The storage medium stores program instructions, and when the program instructions are executed, the method according to any one of claims 1 to 8 is implemented.
