Image data fusion method, device and equipment and computer readable storage medium

Abstract

This application provides an image data fusion method, apparatus, device, and computer-readable storage medium. The image data fusion method includes: acquiring raw road image data collected by multiple image acquisition devices; determining objects in each raw road image data; matching objects in a target raw road image data to obtain a matching result; fusing the first matching object in the target raw road image data to obtain fused target road image data; updating the fused target road image data with any non-target raw road image data to the target raw road image data; returning to matching objects in the target raw road image data; and continuing until all objects in all raw road image data have been matched. According to this application embodiment, objects repeatedly acquired by different image acquisition devices can be fused to obtain fused road image data with high realism and accuracy.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, and in particular relates to an image data fusion method, apparatus, device and computer-readable storage medium. Background Technology

[0002] The realization of autonomous driving relies primarily on vehicle-side intelligence and roadside intelligence. Vehicle-side intelligence can acquire and analyze data through the vehicle's image acquisition devices, but due to the limitations of the vehicle itself, these devices cannot identify targets over a wide area. Therefore, roadside intelligence is needed to supplement this.

[0003] In roadside intelligence, image acquisition devices installed near roads collect data and transmit it to nearby vehicles so they can analyze it. However, in the implementation of roadside intelligence, to prevent missed target detection, the acquisition areas of different image acquisition devices may overlap. Targets in the overlapping areas may be detected simultaneously by different image acquisition devices, resulting in duplicate data collection for that target. If this duplicated data is not processed, it may affect the accuracy of vehicle data analysis. Summary of the Invention

[0004] This application provides an image data fusion method, apparatus, device, and computer-readable storage medium, which can fuse objects repeatedly acquired by different image acquisition devices to obtain fused road image data with high authenticity and accuracy.

[0005] In a first aspect, embodiments of this application provide an image data fusion method, which includes: acquiring original road image data acquired by multiple image acquisition devices, wherein the acquisition areas of at least two image acquisition devices overlap; performing object recognition on each original road image data to determine the objects in each original road image data; matching the objects in a target original road image data to obtain a matching result, wherein the target original road image data is the original road image data acquired by any two image acquisition devices among the multiple image acquisition devices; when the matching result indicates that there is a matching first object in the target original road image data, fusing the first object in the target original road image data to obtain fused target road image data; updating the fused target road image data with any non-target original road image data to the target original road image data, and returning to matching the objects in the target original road image data to obtain a matching result, until all objects in all original road image data have completed the matching operation to obtain the final road image data; wherein the non-target original road image data is road image data other than the target original road image data among the multiple original road image data.

[0006] According to an embodiment of the first aspect of the present application, the target original road image data includes the first original road image data collected by the first image acquisition device and the second original road image data collected by the second image acquisition device. The first original road image data includes M objects, and the second original road image data includes N objects. Both M and N are positive integers. Matching the objects in the target original road image data to obtain a matching result includes: constructing a matrix with M rows and N columns; determining the relative distance parameter between any one of the M objects and any one of the N objects according to the characteristic parameters of the M objects in the first original road image data and the characteristic parameters of the N objects in the second original road image data; using the relative distance parameter between the m-th object in the first original road image data and the n-th object in the second original road image data as the initial value of the element in the m-th row and n-th column of the matrix to obtain an initial matrix, where 0 < m ≤ M, 0 < n ≤ N, and both m and n are positive integers; inputting the initial matrix into a preset matching algorithm to obtain the matching result of the M objects in the first original road image data and the N objects in the second original road image data.

[0007] According to any one of the foregoing embodiments of the first aspect of the present application, the characteristic parameter includes position information. Determining the relative distance parameter between any one of the M objects and any one of the N objects according to the characteristic parameters of the M objects in the first original road image data and the characteristic parameters of the N objects in the second original road image data includes: calculating the relative distance between any one of the M objects and any one of the N objects according to the position information of the M objects in the first original road image data and the position information of the N objects in the second original road image data to obtain the relative distance parameter.

[0008] According to any one of the foregoing embodiments of the first aspect of the present application, the characteristic parameter further includes direction information. Before inputting the initial matrix into the preset matching algorithm, the image data fusion method further includes: calculating the included angle between the direction of any one of the M objects and the direction of any one of the N objects according to the direction information of the M objects in the first original road image data and the direction information of the N objects in the second original road image data; correcting the element values in the initial matrix according to the included angle to obtain a corrected matrix; inputting the initial matrix into the preset matching algorithm includes: inputting the corrected matrix into the preset matching algorithm.

[0009] According to any of the foregoing embodiments of the first aspect of this application, the element values ​​in the initial matrix are corrected according to the included angle to obtain a corrected matrix, including: when the included angle between the direction of the m-th object in the first original road image data and the direction of the n-th object in the second original road image data is greater than a preset angle threshold, the order of magnitude of the initial value of the element in the m-th row and n-th column of the initial matrix is ​​increased to obtain the corrected matrix.

[0010] According to any of the foregoing embodiments of the first aspect of this application, the feature parameters further include object type. Before inputting the initial matrix into the preset matching algorithm, the image data fusion method further includes: if the type of the m-th object in the first original road image data is inconsistent with the type of the n-th object in the second original road image data, increasing the order of magnitude of the initial value of the element in the m-th row and n-th column of the initial matrix to obtain a modified matrix; inputting the initial matrix into the preset matching algorithm includes: inputting the modified matrix into the preset matching algorithm.

[0011] According to any of the foregoing embodiments of the first aspect of this application, before acquiring the original road image data collected by multiple image acquisition devices, the image data fusion method further includes: calibrating the internal and external parameters of multiple image acquisition devices based on the same world coordinate system, and performing time synchronization operation on multiple image acquisition devices based on a preset time synchronization protocol.

[0012] Secondly, embodiments of this application provide an image data fusion apparatus, comprising: an acquisition module for acquiring original road image data acquired by multiple image acquisition devices, wherein the acquisition areas of at least two image acquisition devices overlap; an identification module for performing object identification on each original road image data to determine the objects in each original road image data; a matching module for matching the objects in the target original road image data to obtain a matching result, wherein the target original road image data is the original road image data acquired by any two of the multiple image acquisition devices; a fusion module for fusing the first object in the target original road image data when the matching result indicates that there is a matching first object in the target original road image data to obtain fused target road image data; and an update module for updating the fused target road image data and any non-target original road image data to the target original road image data, and returning to match the objects in the target original road image data to obtain a matching result, until all objects in all original road image data have completed the matching operation to obtain the final road image data.

[0013] Thirdly, embodiments of this application provide an electronic device, which includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of the image data fusion method provided in the first aspect.

[0014] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the image data fusion method provided in the first aspect.

[0015] The image data fusion method, apparatus, device, and computer-readable storage medium of this application embodiment identify objects in each original road image data by performing object recognition on original road image data acquired by multiple image acquisition devices. Any two image acquisition devices are selected, and their acquired target original road image data are matched. Based on the first matching object in the target original road image data indicated by the matching result, the objects repeatedly acquired by the two image acquisition devices can be determined. Thus, by fusing the first object, a more accurate fused target road image data is obtained. The fused target road image data is then updated with any non-target original road image data to become the target original road image data. The above steps are repeated until all objects in all original road image data have been matched, thus achieving the fusion of objects repeatedly acquired by all image acquisition devices. The resulting fused road image data can more accurately reflect the actual road conditions, thereby facilitating accurate analysis of the actual road conditions by vehicles based on highly realistic and accurate road image data. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating an image data fusion method provided in an embodiment of this application;

[0018] Figure 2 This is a flowchart illustrating another image data fusion method provided in an embodiment of this application;

[0019] Figure 3 This is a schematic diagram of the structure of an image data fusion device provided in an embodiment of this application;

[0020] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0021] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0022] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0023] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0024] Various modifications and variations can be made to this application without departing from its spirit or scope, which will be apparent to those skilled in the art. Therefore, this application is intended to cover modifications and variations falling within the scope of the corresponding claims (the claimed technical solutions) and their equivalents. It should be noted that the embodiments provided in this application can be combined with each other without contradiction.

[0025] Before describing the technical solutions provided in the embodiments of this application, in order to facilitate understanding of the embodiments of this application, this application first specifically explains the problems existing in the prior art:

[0026] In the implementation of roadside intelligence, at least one image acquisition device is typically deployed in each lane of each intersection to detect vehicles, pedestrians, and other targets passing through the intersection. The acquisition areas of multiple image acquisition devices may overlap to some extent in the middle of the road to prevent missed detections. However, targets in the overlapping area will be detected simultaneously by at least two different image acquisition devices. Therefore, when different image acquisition devices report their data to the vehicle, the same target may be reported repeatedly, causing the vehicle to be unable to accurately reconstruct the true road conditions based on the data reported by the image acquisition devices.

[0027] In related technologies, to avoid the same target being repeatedly reported by different image acquisition devices, a fixed distance threshold is usually set. If the distance between multiple targets perceived by different image acquisition devices is less than the set distance threshold, these targets are judged to be the same target. However, this method is only suitable for scenarios with a small number of targets and a large distance between them, and is not suitable for scenarios with a large number of targets, diverse target types, and close proximity between them. In the second scenario, if the distance threshold is set too small, the same target may be split into multiple targets; if the distance threshold is set too large, multiple targets that are actually very close may be regarded as the same target. This leads to a discrepancy between the perceived number of targets and the actual number, with a large error, thus affecting the vehicle's analysis of the true road conditions. Therefore, related technologies cannot effectively solve the above-mentioned technical problems.

[0028] To address the problems of the prior art, embodiments of this application provide an image data fusion method, apparatus, device, and computer-readable storage medium.

[0029] The image data fusion method provided in the embodiments of this application will be introduced first below.

[0030] Figure 1 This is a schematic flowchart of an image data fusion method provided in an embodiment of this application. Figure 1 As shown, the image data fusion method may include the following steps S101 to S105.

[0031] S101. Obtain raw road image data collected by multiple image acquisition devices, with at least two image acquisition devices having overlapping acquisition areas.

[0032] S102. Perform object recognition on each original road image data to determine the objects in each original road image data.

[0033] S103. Match the objects in the original road image data of the target to obtain the matching results.

[0034] S104. If the matching result indicates that there is a matching first object in the target original road image data, the first object in the target original road image data is fused to obtain the fused target road image data.

[0035] S105. Update the fused target road image data and any non-target original road image data to the target original road image data, and return to step S103 until all objects in the original road image data have completed the matching operation to obtain the final road image data.

[0036] The specific implementation methods of the above steps will be described in detail below.

[0037] The image data fusion method of this application embodiment identifies objects in each original road image data by performing object recognition on the original road image data collected by multiple image acquisition devices. It then selects any two image acquisition devices and matches the objects in their collected target original road image data. Based on the first matching object in the target original road image data indicated by the matching result, it determines the objects repeatedly acquired by the two image acquisition devices. Thus, by fusing the first object, a more accurate fused target road image data is obtained. The fused target road image data is then updated with any non-target original road image data to become the target original road image data. This process is repeated until all objects in all original road image data have been matched, thus achieving the fusion of objects repeatedly acquired by all image acquisition devices. The resulting fused road image data accurately reflects the true road conditions, which is beneficial for vehicles to accurately analyze the true road conditions based on highly realistic and accurate road image data.

[0038] The specific implementation methods for each of the above steps are described below.

[0039] In step S101, raw road image data collected by multiple image acquisition devices is acquired. For example, the image acquisition devices can be camera devices deployed at different intersections or lanes, such as cameras or webcams. By using multiple image acquisition devices, the actual road conditions at each intersection and lane can be captured from different angles, thereby acquiring raw road image data collected by each of the multiple image acquisition devices. At least two image acquisition devices have overlapping acquisition areas, so even if one image acquisition device malfunctions, objects in the overlapping area can still be detected by the other image acquisition devices, which helps ensure the stability of the raw road image data acquisition.

[0040] To facilitate the direct identification of objects simultaneously detected by different image acquisition devices based on the original road image data, it is necessary to perform a unified calibration operation on all image acquisition devices beforehand. This ensures that the information for the same object remains consistent across the original road image data acquired by all devices. Therefore, as another implementation of the image data fusion method in this application, such as... Figure 2 As shown, before S101, the image data fusion method may also include the following step S201.

[0041] S201. Calibrate the internal and external parameters of multiple image acquisition devices based on the same world coordinate system, and perform time synchronization operation on multiple image acquisition devices based on a preset time synchronization protocol.

[0042] In S201, the intrinsic parameters of multiple image acquisition devices are first calibrated; then, a unified world coordinate system is established based on a pre-defined world coordinate origin and coordinate axis directions; finally, the extrinsic parameters of the multiple image acquisition devices are calibrated based on this unified world coordinate system. In this way, the raw road image data acquired by all image acquisition devices can be unified under the same world coordinate system, thereby ensuring that the information of the same object remains consistent across all the raw road image data acquired by all image acquisition devices.

[0043] Furthermore, it is necessary to perform time synchronization operations on multiple image acquisition devices based on a preset time synchronization protocol. This ensures that the raw road image data acquired by all image acquisition devices has an accurate and consistent timestamp, thereby guaranteeing the temporal consistency of the raw road image data acquired by all image acquisition devices, and in other words, ensuring the temporal consistency of information about the same object in the raw road image data acquired by all image acquisition devices. For example, the preset time synchronization protocol can be a Precision Time Protocol (PTP) or a Network Time Protocol (NTP).

[0044] In S102, each original road image data includes information about each object within the acquisition area of ​​the corresponding image acquisition device. By performing object recognition on each original road image data, the objects in each original road image data can be identified.

[0045] For example, if the original road image data consists of specific images captured by a camera or other imaging device, then image recognition technology can be used to identify objects within each piece of the original road image data. For instance, when an image acquisition device acquires original road image data, it can identify various vehicles or pedestrians appearing within the acquisition area.

[0046] In S103, the raw road image data collected by any two of the multiple image acquisition devices is used as the target raw road image data, and the objects in the different raw road image data in the target raw road image data are matched to obtain a matching result.

[0047] As an implementation manner of S103, S103 may specifically include: constructing a matrix of M rows and N columns; determining the relative distance parameter between any one of the M objects in the first raw road image data and any one of the N objects in the second raw road image data according to the characteristic parameters of the M objects in the first raw road image data and the characteristic parameters of the N objects in the second raw road image data; using the relative distance parameter between the m-th object in the first raw road image data and the n-th object in the second raw road image data as the initial value of the element in the m-th row and n-th column of the matrix to obtain an initial matrix, where 0 < m ≤ M, 0 < n ≤ N, and both m and n are positive integers; inputting the initial matrix into a preset matching algorithm to obtain the matching result of the M objects in the first raw road image data and the N objects in the second raw road image data.

[0048] Exemplarily, the target raw road image data may include the first raw road image data collected by the first image acquisition device and the second raw road image data collected by the second image acquisition device, where the first raw road image data may include M objects and the second raw road image data may include N objects, and both M and N are positive integers.

[0049] When matching the M objects in the first raw road image data and the N objects in the second raw road image data based on a preset matching algorithm, it is necessary to pre-construct a matrix of M rows and N columns, and the element value in the m-th row and n-th column of the matrix can represent the cost of matching the m-th object in the first raw road image data and the n-th object in the second raw road image data, where 0 < m ≤ M, 0 < n ≤ N, and both m and n are positive integers.

[0050] According to the respective characteristic parameters of the M objects and the N objects, the relative distance parameter between any one of the M objects and any one of the N objects can be determined. Using the relative distance parameter between the m-th object in the first raw road image data and the n-th object in the second raw road image data as the initial value of the element in the m-th row and n-th column of the matrix, the initial values of each element in the matrix are thus obtained, and the initial matrix is determined.

[0051] The initial matrix is ​​input into a preset matching algorithm. Based on the initial value of each element in the initial matrix, the preset matching algorithm calculates the minimum cost matching scenario between M objects in the first original road image data and N objects in the second original road image data, thus obtaining the matching result. For example, the preset matching algorithm can be a weighted bipartite graph optimal matching algorithm (Kuhn-Munkres, KM algorithm).

[0052] Thus, by constructing an initial matrix and inputting it into a preset matching algorithm, the matching results of M objects in the first original road image data and N objects in the second original road image data can be obtained, avoiding the problem of large matching error caused by inaccurate distance threshold settings. Therefore, through the embodiments of this application, even in scenarios with a large number of targets, diverse target types, and close distances between targets, a relatively accurate matching result can be obtained directly and quickly.

[0053] As another implementation of S103, based on the feature parameters of M objects in the first original road image data and the feature parameters of N objects in the second original road image data, the relative distance parameter between any one of the M objects and any one of the N objects is determined. Specifically, this may include: calculating the relative distance between any one of the M objects and any one of the N objects based on the position information of the M objects in the first original road image data and the position information of the N objects in the second original road image data, and obtaining the relative distance parameter.

[0054] For example, the feature parameters may include location information, such as the specific coordinates of vehicles or pedestrians in the same world coordinate system. Based on the specific coordinates of M objects in the first original road image data and the specific coordinates of N objects in the second original road image data in the same world coordinate system, the relative distance between any one of the M objects and any one of the N objects can be calculated, and each calculated relative distance is used as the initial value of the corresponding element in the initial matrix.

[0055] In this way, an initial matrix that can be used for the preset matching algorithm can be determined, so as to obtain the specific matching results of M objects in the first original road image data and N objects in the second original road image data according to the preset matching algorithm.

[0056] As another implementation of the image data fusion method of this application, before inputting the initial matrix into the preset matching algorithm, the image data fusion method may further include: calculating the angle between the direction of any one of the M objects and the direction of any one of the N objects based on the direction information of M objects in the first original road image data and the direction information of N objects in the second original road image data; and correcting the element values ​​in the initial matrix based on the angle to obtain the corrected matrix.

[0057] For example, the feature parameters may also include direction information. For instance, the horizontal angle between the north direction line from the world coordinate origin and the direction line where the object is located, moving clockwise, can be used as the direction information of this object. Based on the direction information of M objects in the first original road image data and the direction information of N objects in the second original road image data, the angle between the direction of any one of the M objects and the direction of any one of the N objects can be calculated. Based on the angle between the direction of the m-th object in the first original road image data and the direction of the n-th object in the second original road image data, the initial value of the element in the m-th row and n-th column of the aforementioned initial matrix can be corrected to obtain a corrected matrix. This corrected matrix is ​​then input into a preset matching algorithm to obtain the matching result.

[0058] Therefore, considering the influence of the orientation information of different objects on the matching results, this application embodiment further combines the orientation information on the basis of the position information to modify the initial matrix. In this way, when the modified matrix is ​​input into the preset matching algorithm, the influence of the orientation information on the matching results can be eliminated, thereby further improving the accuracy of the matching results.

[0059] As another implementation of the image data fusion method of this application, the element values ​​in the initial matrix are corrected according to the included angle to obtain the corrected matrix. Specifically, it may include: when the included angle between the direction of the m-th object in the first original road image data and the direction of the n-th object in the second original road image data is greater than a preset angle threshold, increasing the order of magnitude of the initial value of the element in the m-th row and n-th column of the initial matrix to obtain the corrected matrix.

[0060] For example, the preset angle threshold can be set to 60° according to actual needs, but this application embodiment does not limit it.

[0061] If the angle between the direction of the m-th object in the first original road image data and the direction of the n-th object in the second original road image data is less than or equal to a preset angle threshold of 60°, then there is no need to correct the initial value of the element in the m-th row and n-th column of the initial matrix.

[0062] If the angle between the direction of the m-th object in the first original road image data and the direction of the n-th object in the second original road image data is greater than a preset angle threshold of 60°, then the initial value of the element in the m-th row and n-th column of the initial matrix is ​​increased by an order of magnitude. For example, the initial value of the element in the m-th row and n-th column can be increased to a value two orders of magnitude higher than the largest initial value in the initial matrix. This increases the cost of matching the m-th object in the first original road image data with the n-th object in the second original road image data, thereby reducing the probability that these two objects will match in subsequent matching based on the preset matching algorithm. Thus, the corrected matrix can be obtained.

[0063] Therefore, considering the influence of the orientation information of different objects on the matching results, this application embodiment further combines the orientation information on the basis of the position information to modify the initial matrix. In this way, when the modified matrix is ​​input into the preset matching algorithm, the influence of the orientation information on the matching results can be eliminated, thereby further improving the accuracy of the matching results.

[0064] As another implementation of the image data fusion method of this application, before inputting the initial matrix into the preset matching algorithm, the image data fusion method may further include: if the type of the m-th object in the first original road image data is inconsistent with the type of the n-th object in the second original road image data, increasing the order of magnitude of the initial value of the element in the m-th row and n-th column of the initial matrix to obtain the corrected matrix.

[0065] For example, the feature parameters may also include object types, such as pedestrians, bicycles, motorcycles, tricycles, cars, trucks, buses, etc.

[0066] If the type of the m-th object in the first original road image data is the same as the type of the n-th object in the second original road image data, for example, if the type of the m-th object and the type of the n-th object are both cars, then there is no need to modify the initial value of the element in the m-th row and n-th column of the initial matrix.

[0067] If the type of the m-th object in the first original road image data is inconsistent with the type of the n-th object in the second original road image data—for example, the m-th object is a pedestrian and the n-th object is a car—then the initial value of the element in the m-th row and n-th column of the initial matrix is ​​increased by an order of magnitude. For instance, the initial value of the element in the m-th row and n-th column can be increased to a value two orders of magnitude higher than the largest initial value in the initial matrix. This increases the cost of matching the m-th object in the first original road image data with the n-th object in the second original road image data, thereby reducing the probability that these two objects will match in subsequent matching based on a preset matching algorithm. Thus, the corrected matrix is ​​obtained.

[0068] Therefore, considering the impact of different object types on the matching results, this embodiment of the application further combines object type with position and direction information to modify the initial matrix. When the modified matrix is ​​input into the preset matching algorithm, the influence of object type on the matching results can be eliminated, thereby further improving the accuracy of the matching results.

[0069] In S104, the matching result can also be presented as an M-row N-column matrix. In this case, the element value in the m-th row and n-th column of the matrix represents the matching result between the m-th object in the first original road image data and the n-th object in the second original road image data. If the m-th object and the n-th object are a matching first object, then the element value in the m-th row and n-th column is 0; if the m-th object and the n-th object do not match, then the element value in the m-th row and n-th column is -1.

[0070] For example, if there are elements with a value of 0 in the matrix, it indicates that there is a matching first object in the target original road image data. That is, an object in the first original road image data and an object in the second original road image data are the same object and were repeatedly captured by the two image acquisition devices. In this case, it is necessary to fuse the repeatedly captured first object. For example, the average value of the feature parameters such as the position information and direction information of the two matching objects can be calculated as the fused target road image data. Alternatively, based on the weights of the two image acquisition devices, the weighted average value of the feature parameters such as the position information and direction information of the two matching objects can be calculated as the fused target road image data. This application does not limit this aspect.

[0071] In S105, the fused target road image data is updated with any one of the road image data other than the target original road image data from the multiple acquired original road image data, and the process returns to step S103 until all objects in the original road image data have completed the matching operation, and the final road image data is obtained.

[0072] For example, if four original road image data sets are acquired from four image acquisition devices, the original road image data sets acquired by the first and second image acquisition devices are first used as target original road image data. Objects in the target original road image data are matched, and the first object in the target original road image data is fused to obtain fused target road image data. Then, the fused target road image data and the original road image data acquired by the third image acquisition device are used as new target original road image data, and the above steps are repeated to complete the matching and fusion of objects in the three original road image data sets acquired by the three image acquisition devices. Then, the fused target road image data and the original road image data acquired by the fourth image acquisition device are used as new target original road image data, and the above steps are repeated until all objects in the four original road image data sets have been matched, resulting in the final road image data.

[0073] Based on the image data fusion method provided in the above embodiments, this application also provides specific implementations of an image data fusion apparatus. Please refer to the following embodiments.

[0074] First see Figure 3 The image data fusion apparatus 300 provided in this application embodiment includes the following modules:

[0075] The acquisition module 301 is used to acquire raw road image data collected by multiple image acquisition devices, wherein the acquisition areas of at least two image acquisition devices overlap.

[0076] The recognition module 302 is used to perform object recognition on each original road image data and determine the objects in each original road image data.

[0077] The matching module 303 is used to match objects in the target original road image data and obtain matching results. The target original road image data is the original road image data collected by any two image acquisition devices among multiple image acquisition devices.

[0078] The fusion module 304 is used to fuse the first object in the target original road image data when the matching result indicates that there is a matching first object in the target original road image data, so as to obtain the fused target road image data.

[0079] The update module 305 is used to update the fused target road image data with any non-target original road image data to the target original road image data, and return the matching results of objects in the target original road image data until all objects in the original road image data have been matched to obtain the final road image data.

[0080] The image data fusion device according to the embodiment of the present application can determine the objects in each original road image data by performing object recognition on the original road image data collected by multiple image acquisition devices; select any two image acquisition devices and match the objects in the target original road image data collected by them; according to the first objects that match in the target original road image data indicated by the matching result, the objects repeatedly collected by these two image acquisition devices can be determined. In this way, through the fusion of the first objects, the more accurate target road image data after fusion can be obtained. Then, the target road image data after fusion and any non-target original road image data are updated to the target original road image data, and the above steps are repeated until the matching operations of all the objects in all the original road image data are completed, and the fusion of the objects repeatedly collected by all the image acquisition devices is realized. The road image data after fusion obtained in this way can more accurately reflect the real situation of the road, which is beneficial for the vehicle to accurately analyze the real situation of the road based on the road image data with high authenticity and accuracy.

[0081] In some embodiments, the above-mentioned matching module 303 is specifically configured to construct a matrix with M rows and N columns; determine the relative distance parameters between any one of the M objects in the first original road image data and any one of the N objects in the second original road image data according to the characteristic parameters of the M objects in the first original road image data and the characteristic parameters of the N objects in the second original road image data; use the relative distance parameter between the m-th object in the first original road image data and the n-th object in the second original road image data as the initial value of the element in the m-th row and n-th column of the matrix to obtain an initial matrix, where 0 < m ≤ M, 0 < n ≤ N, and both m and n are positive integers; input the initial matrix into a preset matching algorithm to obtain the matching result of the M objects in the first original road image data and the N objects in the second original road image data.

[0082] In some embodiments, the above-mentioned matching module 303 can also be used to calculate the relative distance between any one of the M objects and any one of the N objects according to the position information of the M objects in the first original road image data and the position information of the N objects in the second original road image data, so as to obtain the relative distance parameter.

[0083] In some embodiments, the image data fusion apparatus 300 may further include a first correction module, configured to calculate the angle between the direction of any one of the M objects and the direction of any one of the N objects based on the direction information of M objects in the first original road image data and the direction information of N objects in the second original road image data; correct the element values ​​in the initial matrix based on the angle to obtain a corrected matrix; and input the initial matrix into a preset matching algorithm, including: inputting the corrected matrix into the preset matching algorithm.

[0084] In some embodiments, the first correction module described above can also be used to increase the order of magnitude of the initial value of the element in the m-th row and n-th column of the initial matrix to obtain a corrected matrix when the angle between the direction of the m-th object in the first original road image data and the direction of the n-th object in the second original road image data is greater than a preset angle threshold.

[0085] In some embodiments, the image data fusion apparatus 300 may further include a second correction module, used to increase the order of magnitude of the initial value of the element in the m-th row and n-th column of the initial matrix to obtain a corrected matrix when the type of the m-th object in the first original road image data is inconsistent with the type of the n-th object in the second original road image data; and to input the initial matrix into a preset matching algorithm, including: inputting the corrected matrix into the preset matching algorithm.

[0086] In some embodiments, the image data fusion device 300 may further include a calibration module for calibrating the intrinsic and extrinsic parameters of multiple image acquisition devices based on the same world coordinate system, and for performing time synchronization operations on multiple image acquisition devices based on a preset time synchronization protocol.

[0087] Figure 3 Each module in the illustrated device has the ability to implement Figure 1 The functions of each step in the process and their corresponding technical effects are described in detail here for the sake of brevity.

[0088] Based on the image data fusion method provided in the above embodiments, this application also provides specific implementation methods for electronic devices. Please refer to the following embodiments.

[0089] Figure 4 A schematic diagram of the hardware structure of the electronic device provided in an embodiment of this application is shown.

[0090] The electronic device may include a processor 401 and a memory 402 storing computer program instructions.

[0091] Specifically, the processor 401 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0092] Memory 402 may include mass storage for data or instructions. For example, and not limitingly, memory 402 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. In one example, memory 402 may include removable or non-removable (or fixed) media, or memory 402 may be non-volatile solid-state memory. Memory 402 may be internal or external to the integrated gateway disaster recovery device.

[0093] In one example, memory 402 may be read-only memory (ROM). In one example, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0094] Memory 402 may include read-only memory (ROM), random access memory (RAM), disk storage media device, optical storage media device, flash memory device, electrical, optical, or other physical / tangible memory storage device. Therefore, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to one aspect of this application.

[0095] The processor 401 reads and executes computer program instructions stored in the memory 402 to achieve... Figure 1 The method / steps S101 to S105 in the illustrated embodiment achieve the following: Figure 1 The technical effects achieved by executing the methods / steps shown in the examples are not elaborated here for the sake of brevity.

[0096] In one example, the electronic device may also include a communication interface 403 and a bus 410. For example, Figure 4As shown, the processor 401, memory 402, and communication interface 403 are connected through bus 410 and complete communication with each other.

[0097] The communication interface 403 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0098] Bus 410 includes hardware, software, or both, that couples components of an electronic device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 410 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, this application contemplates any suitable bus or interconnect.

[0099] Furthermore, in conjunction with the image data fusion methods in the above embodiments, this application embodiment can provide a computer-readable storage medium for implementation. This computer-readable storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the image data fusion methods in the above embodiments. Examples of computer-readable storage media include non-transitory computer-readable storage media, such as electronic circuits, semiconductor memory devices, ROM, random access memory, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, and hard disks.

[0100] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0101] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0102] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0103] The aspects of this application have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by dedicated hardware performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0104] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. An image data fusion method, characterized in that, The method includes: Obtaining original road image data collected by multiple image acquisition devices, where the acquisition areas of at least two of the image acquisition devices overlap; Performing object recognition on each of the original road image data to determine the objects in each of the original road image data; Matching the objects in the target original road image data to obtain a matching result, where the target original road image data is the original road image data collected by any two of the multiple image acquisition devices; When the matching result indicates that there is a first object that matches the object in the target original road image data, fusing the first object in the target original road image data to obtain the fused target road image data; Updating the fused target road image data and any non-target original road image data as the target original road image data, and returning to the step of matching the objects in the target original road image data to obtain a matching result until the objects in all the original road image data have completed the matching operation to obtain the final road image data; The non-target original road image data is the road image data other than the target original road image data among the multiple original road image data; Among them, the target original road image data includes the first original road image data collected by the first image acquisition device and the second original road image data collected by the second image acquisition device. The first original road image data includes M objects, and the second original road image data includes N objects, where M and N are both positive integers. The step of matching the objects in the target original road image data to obtain a matching result includes: Constructing a matrix with M rows and N columns; Determining the relative distance parameter between any one of the M objects and any one of the N objects according to the characteristic parameters of the M objects in the first original road image data and the characteristic parameters of the N objects in the second original road image data; Taking the relative distance parameter between the m-th object in the first original road image data and the n-th object in the second original road image data as the initial value of the element in the m-th row and n-th column of the matrix to obtain an initial matrix, where 0 < m ≤ M, 0 < n ≤ N, and m and n are both positive integers; Inputting the initial matrix into a preset matching algorithm to obtain the matching result of the M objects in the first original road image data and the N objects in the second original road image data.

2. The method according to claim 1, characterized in that, The characteristic parameters include position information. The step of determining the relative distance parameter between any one of the M objects and any one of the N objects according to the characteristic parameters of the M objects in the first original road image data and the characteristic parameters of the N objects in the second original road image data includes: Calculating the relative distance between any one of the M objects and any one of the N objects according to the position information of the M objects in the first original road image data and the position information of the N objects in the second original road image data to obtain the relative distance parameter.

3. The method according to claim 2, characterized in that, The feature parameters also include direction information. Before inputting the initial matrix into the preset matching algorithm, the method further includes: Based on the orientation information of M objects in the first original road image data and the orientation information of N objects in the second original road image data, calculate the angle between the orientation of any one of the M objects and the orientation of any one of the N objects. Based on the included angle, the element values ​​in the initial matrix are corrected to obtain the corrected matrix; The step of inputting the initial matrix into the preset matching algorithm includes: Input the corrected matrix into the preset matching algorithm.

4. The method according to claim 3, characterized in that, The step of correcting the element values ​​in the initial matrix according to the included angle to obtain the corrected matrix includes: If the angle between the direction of the m-th object in the first original road image data and the direction of the n-th object in the second original road image data is greater than a preset angle threshold, the order of magnitude of the initial value of the element in the m-th row and n-th column of the initial matrix is ​​increased to obtain the corrected matrix.

5. The method according to any one of claims 2-4, characterized in that, The feature parameters also include object type, and the method further includes, before inputting the initial matrix into the preset matching algorithm: If the type of the m-th object in the first original road image data is inconsistent with the type of the n-th object in the second original road image data, the order of magnitude of the initial value of the element in the m-th row and n-th column of the initial matrix is ​​increased to obtain the corrected matrix; The step of inputting the initial matrix into the preset matching algorithm includes: Input the corrected matrix into the preset matching algorithm.

6. The method according to claim 1, characterized in that, Before acquiring the raw road image data collected by multiple image acquisition devices, the method further includes: The internal and external parameters of the multiple image acquisition devices are calibrated based on the same world coordinate system, and the multiple image acquisition devices are synchronized in time based on a preset time synchronization protocol.

7. An image data fusion device, characterized in that, The device includes: The acquisition module is used to acquire raw road image data collected by multiple image acquisition devices, wherein the acquisition areas of at least two of the image acquisition devices overlap. The recognition module is used to perform object recognition on each of the original road image data and determine the objects in each of the original road image data. The matching module is used to match objects in the target original road image data to obtain matching results. The target original road image data is the original road image data acquired by any two of the multiple image acquisition devices. A fusion module is used to fuse the first object in the target original road image data when the matching result indicates that there is a matching first object in the target original road image data, so as to obtain fused target road image data. The update module is used to update the fused target road image data with any non-target original road image data to the target original road image data, and return the matching results of the objects in the target original road image data until all objects in the original road image data have been matched to obtain the final road image data. Among them, the target original road image data includes the first original road image data collected by the first image acquisition device and the second original road image data collected by the second image acquisition device. The first original road image data includes M objects, and the second original road image data includes N objects. Both M and N are positive integers. The matching module is specifically configured to: Construct a matrix with M rows and N columns; Determine the relative distance parameter between any one of the M objects and any one of the N objects according to the feature parameters of the M objects in the first original road image data and the feature parameters of the N objects in the second original road image data; Use the relative distance parameter between the m-th object in the first original road image data and the n-th object in the second original road image data as the initial value of the element in the m-th row and n-th column of the matrix to obtain an initial matrix, where 0 < m ≤ M, 0 < n ≤ N, and both m and n are positive integers; Input the initial matrix into a preset matching algorithm to obtain the matching result of the M objects in the first original road image data and the N objects in the second original road image data.

8. An electronic device, characterized in that, The electronic device includes: a processor, a memory, and a computer program stored on the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of the image data fusion method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium. When the computer program is executed by the processor, it implements the steps of the image data fusion method according to any one of claims 1 to 6.

Images