Multi-camera fusion ranging method and apparatus

Abstract

The application provides a multi-camera fusion ranging method and device, which comprises the following steps: acquiring target ranging results of each camera; performing cache processing on the target ranging results of each camera to obtain cached ranging results; reading the latest time stamp of each camera from the cached ranging results; selecting a first time stamp according to the latest time stamp, and determining a second time stamp according to the first time stamp; realizing time synchronization between different cameras according to the second time stamp; fusing the target ranging results of each camera to obtain fused ranging results; tracking the fused ranging results to obtain tracking results; acquiring multiple frames of information according to the tracking results, and determining a final ranging result by using the multiple frames of information; and the fusion of multi-camera target ranging and the time synchronization between different cameras are realized.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to a ranging method and apparatus using multi-camera fusion. Background Technology

[0002] Currently, in autonomous driving, vision-based ranging methods primarily rely on individual cameras independently perceiving and measuring distances, then fusing these results to output the final ranging result. This approach is prone to errors when linking different cameras, leading to target fragmentation and preventing the fusion of ranging results. Furthermore, the perception timestamps of different cameras cannot be unified, hindering time synchronization. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to provide a ranging method and apparatus for multi-camera fusion, which realizes the fusion of ranging of targets from multiple cameras and time synchronization between different cameras.

[0004] In a first aspect, embodiments of the present invention provide a ranging method using multi-camera fusion, the method comprising:

[0005] Obtain the target ranging results for each camera;

[0006] The target ranging results of each camera are cached to obtain cached ranging results.

[0007] Read the latest timestamp of each camera from the cached ranging results;

[0008] A first timestamp is selected based on the latest timestamp, and a second timestamp is determined based on the first timestamp;

[0009] Time synchronization between different cameras is achieved based on the second timestamp;

[0010] The target ranging results of each camera are fused to obtain the fused ranging result.

[0011] The fused ranging results are then tracked to obtain the tracking results;

[0012] Based on the tracking results, obtain multi-frame information and use the multi-frame information to determine the final ranging result.

[0013] Furthermore, the target ranging result includes the three-dimensional coordinates of each camera in the vehicle coordinate system, the perception timestamp corresponding to each camera, and the camera number corresponding to each camera.

[0014] Furthermore, the target ranging results of each camera are cached to obtain cached ranging results, including:

[0015] Each camera is arranged according to its corresponding camera number to obtain a plurality of arranged cameras, wherein the plurality of arranged cameras include: a forward mid-range camera, a forward long-range camera, a forward close-range camera, a rear camera, and a side camera.

[0016] The perception results of each camera after arrangement are cached sequentially according to the corresponding camera number and preset frame number to obtain the cached ranging results.

[0017] Furthermore, selecting a first timestamp based on the latest timestamp, and determining a second timestamp based on the first timestamp, includes:

[0018] Select the earliest arriving timestamp from the latest timestamp;

[0019] The earliest arriving timestamp is taken as the first timestamp;

[0020] The forward-facing mid-range camera is used as the main sensor, and the most recent timestamp of the main sensor is obtained based on the first timestamp;

[0021] The most recent timestamp of the main sensor is used as the second timestamp.

[0022] Furthermore, time synchronization between different cameras is achieved based on the second timestamp, including:

[0023] The most recent timestamp is found from the cache of each of the cameras and matched according to the second timestamp;

[0024] If the most recent timestamp is within the preset timestamp range, then the time synchronization between the different cameras is successful.

[0025] Furthermore, the target ranging results of each camera are fused to obtain a fused ranging result, including:

[0026] Using the 3D results of the target, calculate the lateral projection distance and longitudinal distance of the target;

[0027] The matching score between the two targets is output based on the lateral projection distance and the longitudinal distance of the target.

[0028] The matching scores of the two targets are fused using the Hungarian matching algorithm to obtain the fused ranging result.

[0029] Furthermore, the fused ranging results are tracked to obtain tracking results, including:

[0030] If the target is a newly emerging target, then the newly emerging target is placed in the tracking sequence;

[0031] If the target is successfully associated, then update the tracking results;

[0032] If the target association fails, then a prediction is made for the target;

[0033] If the target exceeds a preset number of frames, the target tracking will disappear.

[0034] Secondly, embodiments of the present invention provide a multi-camera fusion ranging device, the device comprising:

[0035] The acquisition module is used to acquire the target ranging results for each camera.

[0036] The cache processing module is used to cache the target ranging results of each camera to obtain the cached ranging results.

[0037] The reading module is used to read the latest timestamp of each camera from the cached ranging results;

[0038] The selection module is used to select a first timestamp based on the latest timestamp, and to determine a second timestamp based on the first timestamp;

[0039] The time synchronization module is used to achieve time synchronization between different cameras based on the second timestamp;

[0040] The spatial synchronization module is used to fuse the target ranging results of each camera to obtain the fused ranging result;

[0041] The tracking module is used to track the fused ranging results and obtain tracking results;

[0042] The determination module is used to obtain multi-frame information based on the tracking results and use the multi-frame information to determine the final ranging result.

[0043] Thirdly, embodiments of the present invention provide an electronic device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the computer program to implement the method described above.

[0044] Fourthly, embodiments of the present invention provide a computer-readable medium having processor-executable non-volatile program code that causes the processor to perform the method described above.

[0045] This invention provides a multi-camera fusion ranging method and apparatus, comprising: acquiring target ranging results from each camera; caching the target ranging results from each camera to obtain cached ranging results; reading the latest timestamp of each camera from the cached ranging results; selecting a first timestamp based on the latest timestamp, and determining a second timestamp based on the first timestamp; achieving time synchronization between different cameras based on the second timestamp; fusing the target ranging results from each camera to obtain a fused ranging result; tracking the fused ranging result to obtain a tracking result; acquiring multi-frame information based on the tracking result, and using the multi-frame information to determine the final ranging result; thereby achieving multi-camera target ranging fusion and time synchronization between different cameras.

[0046] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0047] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0048] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0049] Figure 1 This is a flowchart of the multi-camera fusion ranging method provided in Embodiment 1 of the present invention;

[0050] Figure 2 This is a schematic diagram of a multi-camera fusion ranging device provided in Embodiment 2 of the present invention.

[0051] icon:

[0052] 1-Acquisition module; 2-Cache processing module; 3-Reading module; 4-Selection module; 5-Time synchronization module; 6-Spatial synchronization module; 7-Tracking module; 8-Confirmation module. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] To facilitate understanding of this embodiment, the embodiments of the present invention will be described in detail below.

[0055] Example 1:

[0056] Figure 1 This is a flowchart of a multi-camera fusion ranging method provided in Embodiment 1 of the present invention.

[0057] Reference Figure 1 The method includes the following steps:

[0058] Step S101: Obtain the target ranging results for each camera;

[0059] Here, the target ranging results of each camera include, but are not limited to, monocular ranging methods and binocular ranging methods. The final target ranging result is output through either monocular or binocular ranging methods. A single camera performs target ranging on the input image; camera types include, but are not limited to, pinhole cameras and fisheye cameras, and other types of cameras are also possible.

[0060] Step S102: Cache the target ranging results of each camera to obtain the cached ranging results;

[0061] Step S103: Read the latest timestamp of each camera from the cached ranging results;

[0062] Step S104: Select the first timestamp based on the latest timestamp, and determine the second timestamp based on the first timestamp;

[0063] Step S105: Time synchronization between different cameras is achieved based on the second timestamp;

[0064] Step S106: Fuse the target ranging results of each camera to obtain the fused ranging result;

[0065] Here, once time synchronization is complete, each camera has independent sensing results under the same timestamp. Therefore, spatial synchronization is required to fuse and output the target ranging results of each camera.

[0066] Step S107: Track the fused ranging results to obtain the tracking results;

[0067] Step S108: Obtain multi-frame information based on the tracking results, and use the multi-frame information to determine the final ranging result.

[0068] Furthermore, the target ranging results include the three-dimensional coordinates of each camera in the vehicle coordinate system, the perception timestamp corresponding to each camera, and the camera number corresponding to each camera.

[0069] Furthermore, step S102 includes the following steps:

[0070] Step S201: Arrange each camera according to its corresponding camera number to obtain multiple cameras after arrangement. The multiple cameras after arrangement include: forward mid-range camera, forward long-range camera, forward close-range camera, rear camera and side camera.

[0071] Step S202: The perception results of each camera after arrangement are cached sequentially according to the corresponding camera number and the preset number of frames to obtain the cached ranging results. The preset number of frames can be 5 frames.

[0072] Specifically, the perception results of each camera are cached for a preset number of frames. When the preset number of frames is exceeded, the cache is automatically updated, but it will not exceed the preset number of frames. During the caching process, the data is arranged according to the camera ID. That is, the cached content of camera 1 includes: camera 1 ID and the perception results for the preset number of frames; the cached content of camera 2 includes: camera 2 ID and the perception results for the preset number of frames; other cameras are cached in the same way as camera 1.

[0073] Furthermore, step S104 includes the following steps:

[0074] Step S301: Select the earliest arriving timestamp from the latest timestamps;

[0075] Step S302: Take the earliest arriving timestamp as the first timestamp;

[0076] Step S303: Use the forward-facing mid-range camera as the main sensor, and obtain the most recent timestamp of the main sensor based on the first timestamp;

[0077] Step S304: Use the most recent timestamp of the main sensor as the second timestamp.

[0078] Specifically, the multi-camera fusion process first requires determining the master sensor. Among all cameras, the position of the forward mid-range camera is generally the most critical. Therefore, the forward mid-range camera is determined as the master camera, i.e., the master sensor. Then, the most recent timestamp of the master sensor is obtained based on the first timestamp as the second timestamp, and this second timestamp is used as the base timestamp. At the same time, the mid-range ranging result is obtained.

[0079] Furthermore, step S105 includes the following steps:

[0080] Step S401: Find the most recent timestamp from the cache of each camera and match it based on the second timestamp;

[0081] Here, the ranging results of each camera are obtained based on the second timestamp.

[0082] Step S402: If the most recent timestamp is within the preset timestamp range, then the time synchronization between different cameras is successful.

[0083] Furthermore, step S106 includes the following steps:

[0084] Step S501: Using the 3D results of the target, calculate the target's lateral projection distance and longitudinal distance;

[0085] Step S502: Output the matching score between the two targets based on the horizontal projection distance and the vertical distance between the targets;

[0086] Step S503: The matching scores of the two targets are fused using the Hungarian matching algorithm to obtain the fused ranging result.

[0087] Here, after obtaining the fused ranging results, the spatial synchronization of the target is completed.

[0088] Furthermore, step S107 includes the following steps:

[0089] Step S601: If the target is a newly emerging target, then place the newly emerging target in the tracking sequence;

[0090] Step S602: If the target association is successful, update the tracking results;

[0091] Step S603: If target association fails, then predict the target.

[0092] Step S604: If the target exceeds a preset number of frames, target tracking disappears. The preset number of frames is 3 frames.

[0093] This invention provides a multi-camera fusion ranging method, comprising: acquiring target ranging results from each camera; caching the target ranging results from each camera to obtain cached ranging results; reading the latest timestamp of each camera from the cached ranging results; selecting a first timestamp based on the latest timestamp, and determining a second timestamp based on the first timestamp; achieving time synchronization between different cameras based on the second timestamp; fusing the target ranging results from each camera to obtain a fused ranging result; tracking the fused ranging result to obtain a tracking result; acquiring multi-frame information based on the tracking result, and using the multi-frame information to determine the final ranging result; effectively solving the time synchronization problem between different cameras and the ranging fusion problem of multi-camera targets, and realizing the ranging fusion of multi-camera targets and time synchronization between different cameras.

[0094] Example 2:

[0095] Figure 2 This is a schematic diagram of a multi-camera fusion ranging device provided in Embodiment 2 of the present invention.

[0096] Reference Figure 2 The device includes:

[0097] Module 1 is used to acquire the target ranging results of each camera;

[0098] Cache processing module 2 is used to cache the target ranging results of each camera to obtain the cached ranging results;

[0099] Read module 3 is used to read the latest timestamp of each camera from the cached ranging results;

[0100] Select module 4, which is used to select the first timestamp based on the latest timestamp, and determine the second timestamp based on the first timestamp;

[0101] Time synchronization module 5 is used to achieve time synchronization between different cameras based on the second timestamp;

[0102] The spatial synchronization module 6 is used to fuse the target ranging results of each camera to obtain the fused ranging result;

[0103] Tracking module 7 is used to track the fused ranging results and obtain the tracking results;

[0104] The determination module 8 is used to obtain multi-frame information based on the tracking results and to determine the final ranging result using the multi-frame information.

[0105] This invention provides a multi-camera fusion ranging device, comprising: acquiring target ranging results from each camera; caching the target ranging results from each camera to obtain cached ranging results; reading the latest timestamp of each camera from the cached ranging results; selecting a first timestamp based on the latest timestamp, and determining a second timestamp based on the first timestamp; achieving time synchronization between different cameras based on the second timestamp; fusing the target ranging results from each camera to obtain a fused ranging result; tracking the fused ranging result to obtain a tracking result; acquiring multi-frame information based on the tracking result, and using the multi-frame information to determine the final ranging result; thereby achieving the fusion of target ranging from multiple cameras and time synchronization between different cameras.

[0106] This invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the multi-camera fusion ranging method provided in the above embodiments.

[0107] This invention also provides a computer-readable medium having processor-executable non-volatile program code, on which a computer program is stored, and which, when run by a processor, executes the steps of the multi-camera fusion ranging method described above.

[0108] The computer program product provided in this embodiment of the invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the preceding method embodiments. For specific implementation details, please refer to the method embodiments, which will not be repeated here.

[0109] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system and apparatus described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0110] Furthermore, in the description of the embodiments of the present invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in the present invention based on the specific circumstances.

[0111] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0112] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0113] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multi-camera fusion ranging method, characterized in that, The method comprises: acquiring target ranging results of each camera; buffering the target ranging results of each camera to obtain buffered ranging results; reading the latest time stamp of each camera from the buffered ranging results; selecting a first time stamp according to the latest time stamp, and determining a second time stamp according to the first time stamp; synchronizing the time between different cameras according to the second time stamp; fusing the target ranging results of each camera to obtain fused ranging results; tracking the fused ranging results to obtain tracking results; acquiring multiple frames of information according to the tracking results, and determining a final ranging result by using the multiple frames of information; fusing the target ranging results of each camera to obtain fused ranging results, comprising: calculating the lateral projection distance and the longitudinal distance of the target by using the 3D result of the target; outputting the matching scores of the two targets according to the lateral projection distance and the longitudinal distance of the target; fusing the matching scores of the two targets by using the Hungarian matching algorithm to obtain the fused ranging results.

2. The multi-camera fusion ranging method of claim 1, wherein, The target ranging results comprise three-dimensional coordinates of each camera in a vehicle coordinate system, a perception time stamp corresponding to each camera, and a camera number corresponding to each camera.

3. The multi-camera fusion ranging method of claim 1, wherein, Buffering the target ranging results of each camera to obtain buffered ranging results, comprising: arranging each camera according to the corresponding camera number to obtain arranged multiple cameras, wherein the arranged multiple cameras comprise a forward middle-distance camera, a forward long-distance camera, a forward short-distance camera, a rear camera, and a side camera; buffering each arranged camera according to the corresponding camera number and the perception result of a preset frame number in sequence to obtain the buffered ranging results.

4. The multi-camera fusion ranging method of claim 1, wherein, Selecting a first time stamp according to the latest time stamp, and determining a second time stamp according to the first time stamp, comprising: selecting the first-arriving time stamp from the latest time stamp; taking the first-arriving time stamp as the first time stamp; taking the forward middle-distance camera as a master sensor, and acquiring the latest time stamp of the master sensor according to the first time stamp; taking the latest time stamp of the master sensor as the second time stamp.

5. The multi-camera fusion ranging method of claim 1, wherein, Synchronizing the time between different cameras according to the second time stamp, comprising: finding the latest time stamp in the buffer of each camera according to the second time stamp to perform matching; if the latest time stamp is within a preset time stamp range, the time synchronization between the different cameras is successful.

6. The multi-camera fusion ranging method of claim 1, wherein, Tracking the fused ranging results to obtain tracking results, comprising: if the target is a newly appearing target, placing the newly appearing target in a tracking sequence; if the target association is successful, updating the tracking results; if the target association fails, predicting the target; if the target exceeds a preset frame number, the target tracking disappears.

7. A multi-camera fusion ranging device, comprising: The device comprises: an acquisition module configured to acquire target ranging results of each camera; The cache processing module is configured to perform cache processing on the target ranging result of each camera to obtain a cached ranging result. The reading module is configured to read a latest timestamp of each camera from the cached ranging result. The selecting module is configured to select a first timestamp according to the latest timestamp, and determine a second timestamp according to the first timestamp. The time synchronization module is configured to realize time synchronization between different cameras according to the second timestamp. The space synchronization module is configured to fuse the target ranging result of each camera to obtain a fused ranging result. The tracking module is configured to track the fused ranging result to obtain a tracking result. The determining module is configured to acquire multi-frame information according to the tracking result, and determine a final ranging result by using the multi-frame information. The space synchronization module is specifically configured to: calculate a lateral projection distance and a longitudinal distance of the target by using the 3D result of the target; output a matching score of two targets according to the lateral projection distance and the longitudinal distance of the target; fuse the matching score of the two targets by using a Hungarian matching algorithm to obtain the fused ranging result.

8. An electronic device comprising a memory, a processor, the memory having stored thereon a computer program executable on the processor, characterized in that, The processor executes the computer program to implement the method in any one of claims 1 to 6.

9. A computer readable medium having a non-transitory program code executable by a processor, the program code comprising instructions for: The program code causes the processor to execute the method in any one of claims 1 to 6.

Images