An image association method and device, electronic equipment and storage medium

Abstract

The present disclosure relates to an image association method, device, electronic equipment and storage medium. The image association method comprises: acquiring a plurality of to-be-associated images, each to-be-associated image in the plurality of to-be-associated images has corresponding image information, if it is determined according to the image information corresponding to the plurality of to-be-associated images that the plurality of to-be-associated images do not belong to the same driving data, determining at least one spatial data according to the image information corresponding to the plurality of to-be-associated images, wherein the driving data comprises a plurality of images generated by a vehicle and image information corresponding to each image in the plurality of images, and associating two to-be-associated images corresponding to target spatial data satisfying a preset condition in the at least one spatial data. The method provided by the present disclosure can quickly and accurately automatically associate images, further reducing the waste of manpower and material resources.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to an image association method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the development of computer technology, images are widely used in the automotive field. In many car navigation and positioning applications, it is necessary to comprehensively analyze a large number of images taken by the car to obtain a large number of image features. A common method is to associate the images and then extract the features of the associated images for application. However, traditional techniques often involve manually labeling images and associating images with the same labels. However, for massive amounts of image data, this requires a huge investment of manpower and time and is not easy to implement. Summary of the Invention

[0003] To address the aforementioned technical problems, this disclosure provides an image association method, apparatus, electronic device, and storage medium that can quickly and accurately associate images automatically, reducing the waste of manpower and resources.

[0004] In a first aspect, embodiments of this disclosure provide an image association method, the method comprising:

[0005] Multiple images to be associated are obtained, and each of the multiple images to be associated has corresponding image information;

[0006] If it is determined from the image information corresponding to the multiple images to be associated that the multiple images to be associated do not belong to the same driving data, then at least one spatial data is determined from the image information corresponding to the multiple images to be associated, wherein the driving data includes multiple frames of images generated by the vehicle and image information corresponding to each frame of the multiple frames of images;

[0007] Associate the images to be associated with the target spatial data that meet the preset conditions in the at least one spatial data.

[0008] Optionally, the method further includes:

[0009] If it is determined from the image information corresponding to the multiple images to be associated that the multiple images to be associated belong to the same driving data, then it is determined whether the multiple images to be associated are adjacent frame images in the driving data;

[0010] If yes, then the images that are adjacent frames among the multiple images to be associated are associated; if not, then at least one spatial data is determined based on the image information corresponding to the multiple images to be associated.

[0011] Optionally, the spatial data includes distance data and angle data.

[0012] Optionally, associating the image to be associated with the target spatial data that meets the preset conditions in the at least one spatial data includes:

[0013] The images to be associated are associated with the target spatial data whose distance data is less than a first preset threshold and whose angle data is less than a second preset threshold, wherein one spatial data has two corresponding images to be associated.

[0014] Optionally, after associating the target spatial data corresponding to the at least one spatial data that meets preset conditions with the images to be associated, the method further includes:

[0015] A first judgment result is generated based on the image information corresponding to the two associated images;

[0016] A second judgment result is generated based on the extracted feature information of the two associated images;

[0017] If either the first judgment result or the second judgment result indicates that the two images after association are mistakenly associated, then the association relationship between the two images after association is terminated.

[0018] The judgment result is either that the two images after association are incorrectly associated or that the two images after association are not incorrectly associated.

[0019] Optionally, the two associated images include a first image and a second image; generating a first judgment result based on the image information corresponding to the two associated images includes:

[0020] After determining the first coordinates of the first vehicle that generated the first image from the image information corresponding to the first image, the first vehicle is projected into the pixel coordinate system of the second image based on the first coordinates, and a third judgment result is generated based on the pixel coordinate system of the projected second image.

[0021] After determining the second coordinates of the second vehicle that generated the second image from the image information corresponding to the second image, the second vehicle is projected onto the pixel coordinate system of the first image based on the second coordinates, and a fourth judgment result is generated based on the pixel coordinate system of the projected first image.

[0022] A first judgment result is generated based on the third and fourth judgment results.

[0023] Optionally, generating a third judgment result based on the pixel coordinate system of the projected second image includes:

[0024] In the pixel coordinate system of the projected second image, determine whether the first vehicle is in the drivable area of ​​the second vehicle that generated the second image; if yes, the generated two associated images are not a third judgment result of misassociation; if no, the generated two associated images are a third judgment result of misassociation.

[0025] The drivable area of ​​the second vehicle is determined based on the segmented image of the second image.

[0026] Optionally, generating a second judgment result based on the extracted feature information of the two associated images includes:

[0027] Extract the feature information of the two associated images to obtain the feature information of the first image and the feature information of the second image in the two associated images;

[0028] The feature information of the first image and the feature information of the second image are matched to determine the number of successfully matched feature points;

[0029] A second judgment result is generated based on the number of successfully matched feature points and a third preset threshold.

[0030] Secondly, embodiments of this disclosure provide an image association apparatus, the apparatus comprising:

[0031] The acquisition unit is used to acquire multiple images to be associated, and each of the multiple images to be associated has corresponding image information;

[0032] The determining unit is configured to determine at least one spatial data based on the image information corresponding to the plurality of images to be associated if it is determined that the plurality of images to be associated do not belong to the same driving data. The driving data includes multiple frames of images generated by the vehicle and image information corresponding to each frame of the multiple frames.

[0033] The association unit is used to associate the target spatial data that meets the preset conditions in the at least one spatial data with the corresponding image to be associated.

[0034] Thirdly, embodiments of this disclosure provide an electronic device, including:

[0035] Memory;

[0036] Processor; and

[0037] Computer programs;

[0038] The computer program is stored in memory and configured to be executed by a processor to implement the image association method described above.

[0039] Fourthly, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the image association method described above.

[0040] This disclosure provides an image association method, apparatus, electronic device, and storage medium. The image association method includes: acquiring multiple images to be associated, each of which has corresponding image information; if it is determined based on the image information that the multiple images do not belong to the same driving data, then determining at least one spatial data based on the image information, wherein the driving data includes multiple frames of images generated by the vehicle and image information corresponding to each frame; and associating the images to be associated corresponding to the target spatial data that meets preset conditions from the at least one spatial data. The method provided in this disclosure performs a preliminary division of the images to be associated by determining the driving data to which the images belong, which can accelerate the image association process and reduce computational load. For images that do not belong to the same driving data, i.e., images not generated by the same vehicle at the same time, the spatial data between the two images is further calculated. Spatial data can be understood as the data between the position and orientation of the two images at the time of generation. Subsequently, the spatial data between the two images is used to determine whether the two images are associated. This method can quickly and accurately automatically associate images, further reducing the waste of manpower and resources, and also facilitating subsequent vehicle positioning based on the associated images. Attached Figure Description

[0041] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0042] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 A flowchart illustrating an image association method provided in an embodiment of this disclosure;

[0044] Figure 2 A flowchart illustrating an image association method provided in an embodiment of this disclosure;

[0045] Figure 3 A flowchart illustrating an image association method provided in an embodiment of this disclosure;

[0046] Figure 4 A schematic diagram of an image to be associated provided in an embodiment of this disclosure;

[0047] Figure 5 This is a schematic diagram of the structure of an image association device provided in an embodiment of the present disclosure;

[0048] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0049] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0050] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0051] To address the aforementioned technical problems, this disclosure provides an image association method, which is specifically illustrated through one or more of the following embodiments.

[0052] Figure 1 This is a flowchart illustrating an image association method provided in an embodiment of this disclosure. The image association method provided in this disclosure can be executed by a terminal or a server, and specifically includes, for example: Figure 1 The following steps S110 to S130 are shown:

[0053] S110. Obtain multiple images to be associated, each of which has corresponding image information.

[0054] Understandably, multiple images to be associated are obtained. These images can be obtained from driving data, which is data generated by the vehicle with a unique identifier. Driving data is generated whether the vehicle is in a moving state or a stationary state. The driving data involved in this disclosure includes multiple frames of images with timestamps generated by the vehicle and the image information corresponding to each frame. The multiple frames of images are generated by the camera device configured on the vehicle, and the image information corresponding to each frame is collected by the data acquisition device configured on the vehicle. The image information includes spatial coordinates (GPS coordinates) with timestamps, segmented images of the image, and relevant parameters of the camera device. Each frame of image has a corresponding spatial coordinate, that is, the spatial coordinates with timestamps and the time of the image with timestamps are aligned. Time alignment means that a spatial coordinate corresponding to the image is obtained during the time period when the image is generated. The imaging device can be a camera, and its relevant parameters include the camera's intrinsic parameters, distortion coefficients, and extrinsic parameters between the camera and the vehicle, as well as calibration results. If the camera and the GPS coordinate acquisition device are not installed at the same location on the vehicle, and the distance between the camera and the positioning device is known to be fixed, the acquired GPS coordinates of the positioning device can be interpreted as the GPS coordinates of the camera, and thus as the GPS coordinates of the image. Specifically, multiple images to be associated can be acquired within a defined area. The defined area can be several streets, including main roads and auxiliary roads, roads on and under bridges, etc. In other words, multiple images generated by the vehicle can be acquired within the defined area of ​​several streets. In this case, the multiple frames of images can be generated by at least one vehicle. Understandably, the identification of driving data can consist of the vehicle's vehicle identification number and the time of the driving data, or the device identification number of the data acquisition device and the time of generation of the driving data. The vehicle identification number can be the vehicle identification number (VIN) of the vehicle, and the device identification number of the driving data acquisition device can be the unique identification number of the driving data acquisition device at the time of manufacture. The device identification number of the data acquisition device and the vehicle identification number can also have a one-to-one correspondence. The composition method of the driving data identification number is not limited and can be set by the user according to their needs.

[0055] The following example illustrates how to obtain two images to be associated within the same defined area.

[0056] S120. If it is determined from the image information corresponding to the plurality of images to be associated that the plurality of images to be associated do not belong to the same driving data, then at least one spatial data is determined from the image information corresponding to the plurality of images to be associated, wherein the driving data includes multiple frames of images generated by the vehicle and image information corresponding to each frame of the multiple frames of images.

[0057] Understandably, based on the above S110, after obtaining two images to be associated, it is determined whether the two images to be associated belong to the same driving data according to the image information corresponding to the two images to be associated, and the images to be associated are initially divided to improve the calculation efficiency. Here, each driving data includes multiple frames of images, and the multiple frames of images can have the same identifier as the driving data. The image identifier is also stored in the image information. For example, vehicle A generates driving data A1, which includes 5 frames of images, which are denoted as A1-1 to A1-5 respectively. If the two images to be associated obtained are A1-1 and A1-5, it is determined that the two images to be associated belong to the same driving data. If the two images to be associated obtained are A1-1 and A1-5, it is determined that the two images to be associated do not belong to the same driving data. If it is determined from the image information corresponding to two images that they do not belong to the same driving data, then a spatial data is determined from the image information corresponding to the two images. The spatial data includes distance data and angle data. The distance data includes horizontal distance and vertical distance, and the angle data can be the direction angle or the orientation angle. Specifically, a spatial data is calculated based on the GPS coordinates corresponding to each of the two images. A spatial data can be calculated for every two images to be associated. Accordingly, if three images to be associated are obtained, up to three spatial data can be calculated. The driving data includes multiple frames of images generated by the vehicle and the image information corresponding to each frame of the multiple frames.

[0058] Optionally, the image association method provided in this disclosure further includes the following steps:

[0059] If it is determined from the image information corresponding to the multiple images to be associated that the multiple images to be associated belong to the same driving data, then it is determined whether the multiple images to be associated are adjacent frame images in the driving data; if so, then the images that are adjacent frames in the multiple images to be associated are associated; if not, then at least one spatial data is determined from the image information corresponding to the multiple images to be associated.

[0060] Understandably, if it is determined from the image information corresponding to the two images to be associated that the two images belong to the same driving data, then it is further determined whether the two images to be associated are adjacent frame images in the driving data. For example, in the example above, driving data A1 includes images A1-1 and A1-2 which are adjacent frame images. After determining that the two images to be associated are adjacent frame images in the driving data, the two images to be associated that are adjacent frames are associated. If it is determined that the two images to be associated are not adjacent frame images in the driving data, then it is necessary to calculate the spatial data of the two images to be associated. For example, in the example above, driving data A1 includes images A1-1 and A1-5 which are not adjacent frame images.

[0061] S130. Associate the image to be associated with the target spatial data that meets the preset conditions in the at least one spatial data.

[0062] Understandably, based on the above S120, after calculating at least one spatial data according to the image information corresponding to the multiple images to be associated, spatial data that meets the preset conditions is determined in the at least one spatial data, and the spatial data that meets the preset conditions is recorded as the target spatial data. Two images to be associated can be calculated to obtain one spatial data. It is determined whether the spatial data meets the preset conditions. If the spatial data meets the preset conditions, the two images to be associated are associated. If the spatial data does not meet the preset conditions, it is determined that there is no association relationship between the two images to be associated.

[0063] Optionally, S130 specifically includes the following steps: associating the target spatial data corresponding to the at least one spatial data whose distance data is less than a first preset threshold and whose angle data is less than a second preset threshold with the image to be associated, wherein one spatial data has two corresponding images to be associated.

[0064] Understandably, in at least one calculated spatial data, spatial data whose distance is less than a first preset threshold and whose angle is less than a second preset threshold are determined as target spatial data. The distance data includes horizontal distance and vertical distance. The first preset threshold includes a horizontal threshold and a vertical threshold. For example, the horizontal threshold can be 20 meters and the vertical threshold can be 5 meters. The second preset threshold is an angle threshold, which can be 30 degrees. If the horizontal distance is less than 20 meters, the vertical distance is less than 5 meters, and the azimuth angle is less than 5 degrees, based on the GPS coordinates of two images to be associated, it indicates that the two images to be associated are related, and the two images to be associated are associated.

[0065] Understandably, the image association method provided in this disclosure can also be applied to scenarios where the vehicle's current location is determined during navigation, further determining whether the vehicle is on a main road or a secondary road, or on or under a bridge. Reference images are pre-stored on road sections prone to positioning errors, and these reference images are used as one of the images to be associated. When the vehicle travels to this road section, the vehicle-generated image and the reference image are compared for image association. If a correlation is found between the vehicle-generated image and the reference image, the road corresponding to the reference image can be determined as the road where the vehicle is currently located, thereby improving the accuracy of vehicle positioning.

[0066] This disclosure provides an image association method that acquires multiple images to be associated, each with corresponding image information. If the multiple images to be associated are determined not to belong to the same driving data based on the corresponding image information, then at least one spatial data is determined based on the corresponding image information. The driving data includes multiple frames of images generated by the vehicle and the image information corresponding to each frame. Two images to be associated that meet preset conditions within the at least one spatial data are then associated. This method preliminarily divides the images to be associated by determining the driving data to which they belong, reducing computation and accelerating the determination of image association. For two images that do not belong to the same driving data (i.e., two images generated by different vehicles at the same time), the spatial data between the two images is further calculated. Spatial data can be understood as data obtained based on the position and orientation of the two images at the time of generation. Subsequently, the spatial data between the two images is used to determine whether they are associated. This method enables fast and accurate automatic image association, further reducing the waste of manpower and resources, and facilitating subsequent vehicle positioning based on the associated images.

[0067] Based on the above embodiments, Figure 2 An image association method provided in this disclosure, optionally, after associating the images to be associated corresponding to target spatial data that meet preset conditions in at least one spatial data set, further includes, as follows: Figure 2 The following steps S210 to S230 are shown:

[0068] Understandably, after associating two images to be associated, it is necessary to further determine whether there is a false association between the two images. Specifically, a false association judgment can be performed after each pair of images to be associated is determined, or the association relationship of multiple images to be associated can be determined at once, and then the false association between the multiple images to be associated can be checked to improve the accuracy of determining the image association relationship.

[0069] Optionally, the judgment result is either that the two images after association are incorrectly associated or that the two images after association are not incorrectly associated.

[0070] S210. Generate a first judgment result based on the image information corresponding to the two associated images.

[0071] Understandably, the two images after association are denoted as the first image and the second image. A first judgment result is generated based on the image information corresponding to the first image and the image information corresponding to the second image. The judgment result can be either that the two images after association are mistakenly associated or that the two images after association are not mistakenly associated.

[0072] S220. Generate a second judgment result based on the extracted feature information of the two associated images.

[0073] Understandably, feature information from the first and second images is extracted separately, and a second judgment result is generated based on the two feature information. Understandably, the execution order of S210 and S220 is not limited; they can be executed simultaneously or sequentially.

[0074] Optionally, the second judgment result generated in S220 above based on the extracted feature information of the two associated images specifically includes the following steps S221 to S223:

[0075] S221. Extract the feature information of the two associated images to obtain the feature information of the first image and the feature information of the second image in the two associated images.

[0076] S222. Match the feature points in the feature information of the first image and the feature information of the second image to determine the number of successfully matched feature points.

[0077] S223. Generate a second judgment result based on the number of successfully matched feature points and the third preset threshold.

[0078] Understandably, extracting feature information from the first and second images separately allows us to remove dynamic information and retain static information, reducing the number of feature points that need to be matched and speeding up the calculation process. Dynamic information refers to information about the vehicle itself, such as the vehicle body, while static information refers to road information, road signs, and fences. Subsequently, the remaining feature points from the first and second image feature information (after removing dynamic information) are matched to determine the number of successfully matched feature points. Each feature point has a corresponding descriptor; a successful feature point match means that the descriptor corresponding to the feature point is... If the similarity between the descriptors corresponding to two feature points is greater than a preset value (which can be 90%), then the two feature points are considered to be successfully matched. The number of successfully matched feature points is counted, and a second judgment result is generated based on the number of successfully matched feature points and a third preset threshold. The third preset threshold can be 20% or 30%, and the specific value can be determined according to the user's needs. If the number of successfully matched feature points is less than or equal to the third preset threshold, it means that the first image and the second image are misassociated. If the number of successfully matched feature points is greater than the third preset threshold, it means that the first image and the second image are not misassociated.

[0079] S230. If either the first judgment result or the second judgment result indicates that the two images after association are mistakenly associated, then the association relationship between the two images after association is terminated.

[0080] Understandably, based on the above S210 and S220, the first judgment result and the second judgment result are obtained through two methods. If either the first judgment result or the second judgment result indicates that the two images are falsely associated, then the association between the two images is terminated. In other words, if one of the judgment results obtained through the two methods indicates that the two images are falsely associated, then the association between the two images is terminated. Only if both judgment results indicate that the two images are not falsely associated, then the association between the two images is terminated.

[0081] This disclosure provides an image association method. After determining the association relationship between two images, it further detects whether the two images are falsely associated. Specifically, a first judgment result is generated based on the image information corresponding to the two associated images, and a second judgment result is generated based on the extracted feature information of the two associated images. If either the first judgment result or the second judgment result indicates that the two associated images are falsely associated, then the association relationship between the two associated images is terminated. By detecting whether two images with an association relationship are falsely associated, the method can effectively improve the accuracy of image association and has strong feasibility.

[0082] Based on the above embodiments, Figure 3 This is a flowchart illustrating an image association method provided in an embodiment of the present disclosure. Optionally, the step of generating a first judgment result based on the image information corresponding to the two associated images specifically includes, as follows: Figure 3 The following steps S310 to S330 are shown:

[0083] Optionally, the two associated images include a first image and a second image.

[0084] S310. After determining the first coordinates of the first vehicle that generated the first image from the image information corresponding to the first image, the first vehicle is projected onto the pixel coordinate system of the second image according to the first coordinates, and a third judgment result is generated according to the pixel coordinate system of the projected second image.

[0085] Understandably, the first coordinates of the first vehicle that generated the first image are determined from the image information corresponding to the first image. Specifically, the vehicle's position can be determined based on the GPS coordinates in the image information corresponding to the first image. The vehicle's position can be the midpoint of the line connecting the rear wheels of the first vehicle, and the coordinates of the midpoint of the line connecting the rear wheels of the first vehicle are also the GPS coordinates of the image. Then, the midpoint of the line connecting the rear wheels of the first vehicle is projected onto the pixel coordinate system of the second image based on the first coordinates. The midpoint of the line connecting the rear wheels of the first vehicle is projected onto the pixel coordinate system of the second image as a pixel. Specifically, the GPS coordinates of the midpoint of the line connecting the rear wheels of the first vehicle can be converted into the coordinates of a pixel in the pixel coordinate system of the second image based on the relevant camera parameters in the image information corresponding to the second image. After the projection is completed, a third judgment result is generated based on the pixel coordinate system of the projected second image.

[0086] Optionally, a third judgment result is generated based on the pixel coordinate system of the projected second image, including: determining whether the first vehicle is within the drivable area of ​​the second vehicle in the generated second image in the pixel coordinate system of the projected second image; if yes, then a third judgment result is generated that the two associated images are not mistakenly associated; if no, then a third judgment result is generated that the two associated images are mistakenly associated; wherein, the drivable area of ​​the second vehicle is determined based on the segmented image of the second image.

[0087] Understandably, in the pixel coordinate system of the projected second image, determining whether the first vehicle is within the drivable area of ​​the second vehicle in the generated second image involves determining whether the pixel in the pixel coordinate system of the second image, projected from the midpoint of the line connecting the rear wheels of the first vehicle, is within the drivable area of ​​the second vehicle. The drivable area refers to the region composed of multiple pixels in the pixel coordinate system of the second image. The drivable area of ​​the second vehicle is determined based on the segmented image of the second image. The road segmented in the segmented image can be considered as the drivable area of ​​the second vehicle. For example, if the drivable area of ​​the second image consists of 100 pixels, and the pixel in the pixel coordinate system of the second image, projected from the midpoint of the line connecting the rear wheels of the first vehicle, is one of those 100 pixels, then it can be determined that the first vehicle is within the drivable area of ​​the second vehicle in the generated second image, and the two images generated after association are not a result of a false association. Otherwise, the two images generated after association are a result of a false association.

[0088] For example, see Figure 4 , Figure 4 This is a schematic diagram of an associated image provided in this disclosure. Figure 4 The first image 410 generated from the first vehicle includes the body 411 of the second vehicle and multiple white lines of the road 412. Figure 4It also includes a second image 420 generated by the second vehicle. The second image 420 includes a road 412. In the first image 410, the drivable area of ​​the first vehicle is the road 412 in front. It can be seen from the first image 410 that the second vehicle is driving in the drivable area of ​​the first vehicle. The second image 420 includes the drivable area of ​​the second vehicle, the road 412. It can be seen from the first image 410 and the second image 420 that the first vehicle and the second vehicle are driving on the road 412 at the same time. The second vehicle is directly in front of the first vehicle. After projecting the midpoint of the rear wheel of the first vehicle onto the pixel coordinate system of the second image, it can be determined from the pixel coordinate system of the projected second image that the midpoint of the rear wheel of the first vehicle is not in the drivable area of ​​the second vehicle, that is, not in the part of the road 412 shown in the second image 420. The first vehicle is behind the second vehicle, and the road behind the second vehicle does not belong to the drivable area of ​​the second vehicle. In this case, a third judgment result can be generated that the first image and the second image are misassociated.

[0089] S320. After determining the second coordinates of the second vehicle that generated the second image from the image information corresponding to the second image, the second vehicle is projected onto the pixel coordinate system of the first image according to the second coordinates, and a fourth judgment result is generated according to the pixel coordinate system of the first image after projection.

[0090] Understandably, the method for projecting the rear wheel center of the second image into the pixel coordinate system of the first image is the same as the method for projecting the rear wheel center of the first image into the pixel coordinate system of the second image, and will not be repeated here. The execution order of S310 and S320 is not limited.

[0091] Optionally, a fourth judgment result is generated based on the pixel coordinate system of the first image after projection, including: determining whether the second vehicle is in the drivable area of ​​the first vehicle that generated the first image in the pixel coordinate system of the first image after projection; if yes, then a fourth judgment result is generated that the two associated images are not mistakenly associated; if no, then a fourth judgment result is generated that the two associated images are mistakenly associated.

[0092] For example, see Figure 4 As can be seen from the first image 410 and the second image 420, the second vehicle is driving in the drivable area of ​​the first vehicle. After projecting the center of the rear wheel of the second vehicle onto the pixel coordinate system of the first image, the pixel corresponding to the center of the rear wheel of the second vehicle is in the drivable area of ​​the first vehicle in the pixel coordinate system of the first image. In this case, a fourth judgment result can be generated that the first image 410 and the second image 420 are not misassociated.

[0093] S330. Generate a first judgment result based on the third judgment result and the fourth judgment result.

[0094] Understandably, based on the above S310 and S320, if either the third or fourth judgment result indicates that the two associated images are not mistakenly associated, then the first judgment result that the two associated images are not mistakenly associated is also considered. (See [link to relevant documentation]). Figure 4 The first image 410 and the second image 420 shown can generate a first judgment result that the first image 410 and the second image 420 are not misassociated.

[0095] This disclosure provides an image association method. Based on the GPS coordinates in the image information corresponding to the first image, the rear axle center of the first vehicle is projected onto the pixel coordinate system of the second image. Based on the GPS coordinates in the image information corresponding to the second image, the rear axle center of the second vehicle is projected onto the pixel coordinate system of the first image. The pixel coordinate system obtained by projecting the two images onto each other generates a judgment result on whether the association is incorrect, which further improves the accuracy of determining image association.

[0096] Figure 5 This is a schematic diagram of the image association apparatus provided in an embodiment of this disclosure. The image association apparatus provided in this embodiment can execute the processing flow provided in the above-described image association method embodiments, such as... Figure 5 As shown, the image association device 500 includes:

[0097] The acquisition unit 510 is used to acquire multiple images to be associated, each of the multiple images to be associated having corresponding image information;

[0098] The determining unit 520 is configured to determine at least one spatial data based on the image information corresponding to the plurality of images to be associated if it is determined that the plurality of images to be associated do not belong to the same driving data. The driving data includes multiple frames of images generated by the vehicle and image information corresponding to each frame of the multiple frames.

[0099] The association unit 530 is used to associate the image to be associated with the target spatial data that meets the preset conditions in the at least one spatial data.

[0100] Optionally, the device 500 further includes a first determination unit, which is specifically used for:

[0101] If it is determined from the image information corresponding to the multiple images to be associated that the multiple images to be associated belong to the same driving data, then it is determined whether the multiple images to be associated are adjacent frame images in the driving data;

[0102] If yes, then the images that are adjacent frames among the multiple images to be associated are associated; if not, then at least one spatial data is determined based on the image information corresponding to the multiple images to be associated.

[0103] Optionally, the spatial data in device 500 includes distance data and angle data.

[0104] Optionally, the association unit 530 associates the target spatial data corresponding to the at least one spatial data that meets preset conditions with the image to be associated, specifically for:

[0105] The images to be associated are associated with the target spatial data whose distance data is less than a first preset threshold and whose angle data is less than a second preset threshold, wherein one spatial data has two corresponding images to be associated.

[0106] Optionally, the device 500 further includes a second determining unit, which is used to associate the target spatial data corresponding to the at least one spatial data that meets preset conditions with the image to be associated, specifically for:

[0107] A first judgment result is generated based on the image information corresponding to the two associated images;

[0108] A second judgment result is generated based on the extracted feature information of the two associated images;

[0109] If either the first judgment result or the second judgment result indicates that the two images after association are mistakenly associated, then the association relationship between the two images after association is terminated.

[0110] The judgment result is either that the two images after association are incorrectly associated or that the two images after association are not incorrectly associated.

[0111] Optionally, the two associated images in the second judgment unit include a first image and a second image; the step of generating a first judgment result based on the image information corresponding to the two associated images is specifically used for:

[0112] After determining the first coordinates of the first vehicle that generated the first image from the image information corresponding to the first image, the first vehicle is projected into the pixel coordinate system of the second image based on the first coordinates, and a third judgment result is generated based on the pixel coordinate system of the projected second image.

[0113] After determining the second coordinates of the second vehicle that generated the second image from the image information corresponding to the second image, the second vehicle is projected onto the pixel coordinate system of the first image based on the second coordinates, and a fourth judgment result is generated based on the pixel coordinate system of the first image after projection.

[0114] A first judgment result is generated based on the third and fourth judgment results.

[0115] Optionally, the generation of the third judgment result based on the pixel coordinate system of the projected second image in the second judgment unit is specifically used for:

[0116] In the pixel coordinate system of the projected second image, determine whether the first vehicle is in the drivable area of ​​the second vehicle that generated the second image; if yes, the generated two associated images are not a third judgment result of misassociation; if no, the generated two associated images are a third judgment result of misassociation.

[0117] The drivable area of ​​the second vehicle is determined based on the segmented image of the second image.

[0118] Optionally, the second judgment unit's step of generating a second judgment result based on the extracted feature information of the two associated images is specifically used for:

[0119] Extract the feature information of the two associated images to obtain the feature information of the first image and the feature information of the second image in the two associated images;

[0120] The feature information of the first image and the feature information of the second image are matched to determine the number of successfully matched feature points;

[0121] A second judgment result is generated based on the number of successfully matched feature points and a third preset threshold.

[0122] Figure 5 The image association device shown in the embodiment can be used to execute the technical solution of the above method embodiment. Its implementation principle and technical effect are similar, and will not be repeated here.

[0123] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. See below for details. Figure 6 The diagram illustrates a structural schematic suitable for implementing the electronic device 600 in the embodiments of this disclosure. The electronic device 600 in the embodiments of this disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), wearable electronic devices, etc., as well as fixed terminals such as digital TVs, desktop computers, smart home devices, etc. Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0124] like Figure 6As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603 to implement the multimedia information processing method as described in the embodiments of this disclosure. The RAM 603 also stores various programs and data required for the operation of the electronic device 600. The processing device 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0125] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 An electronic device 600 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0126] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts, thereby implementing the multimedia information processing method as described above. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, it performs the functions defined in the methods of embodiments of this disclosure.

[0127] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0128] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0129] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0130] Optionally, when one or more of the above-described procedures are executed by the electronic device, the electronic device may also perform other steps described in the above embodiments.

[0131] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0132] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0133] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0134] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0135] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0136] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or gateway that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or gateway. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or gateway that includes said element.

[0137] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image association method, characterized in that, The method includes: Multiple images to be associated are obtained, and each of the multiple images to be associated has corresponding image information, wherein the image information includes positioning coordinates; If it is determined from the image information corresponding to the multiple images to be associated that the multiple images to be associated do not belong to the same driving data, then at least one spatial data is determined from the positioning coordinates in the image information corresponding to the multiple images to be associated. The driving data includes multiple frames of images generated by the vehicle and the image information corresponding to each frame of the multiple frames. Associate the image to be associated with the target spatial data that meets the preset conditions in the at least one spatial data; A first judgment result is generated based on the image information corresponding to the two associated images. This first judgment result is used to determine whether there is a misassociation between the two associated images. The two associated images include a first image and a second image. After determining the first coordinates of the first vehicle that generated the first image from the image information corresponding to the first image, the first vehicle is projected into the pixel coordinate system of the second image based on the first coordinates, and a third judgment result is generated based on the pixel coordinate system of the projected second image. After determining the second coordinates of the second vehicle that generated the second image from the image information corresponding to the second image, the second vehicle is projected onto the pixel coordinate system of the first image based on the second coordinates, and a fourth judgment result is generated based on the pixel coordinate system of the projected first image. A first judgment result is generated based on the third and fourth judgment results.

2. The method according to claim 1, characterized in that, The method further includes: If it is determined from the image information corresponding to the multiple images to be associated that the multiple images to be associated belong to the same driving data, then it is determined whether the multiple images to be associated are adjacent frame images in the driving data; If yes, then the images that are adjacent frames among the multiple images to be associated are associated; if not, then at least one spatial data is determined based on the image information corresponding to the multiple images to be associated.

3. The method according to claim 1, characterized in that, The spatial data includes distance data and angle data; The step of associating the target spatial data corresponding to the at least one spatial data that meets the preset conditions with the image to be associated includes: The images to be associated are associated with the target spatial data whose distance data is less than a first preset threshold and whose angle data is less than a second preset threshold, wherein one spatial data has two corresponding images to be associated.

4. The method according to claim 1, characterized in that, After associating the images to be associated with the target spatial data that meet the preset conditions in the at least one spatial data, the method further includes: A second judgment result is generated based on the extracted feature information of the two associated images; If either the first judgment result or the second judgment result indicates that the two images after association are mistakenly associated, then the association relationship between the two images after association is terminated. The judgment result is either that the two images after association are incorrectly associated or that the two images after association are not incorrectly associated.

5. The method according to claim 1, characterized in that, The step of generating a third judgment result based on the pixel coordinate system of the projected second image includes: In the pixel coordinate system of the projected second image, determine whether the first vehicle is in the drivable area of ​​the second vehicle that generated the second image; if yes, the generated two associated images are not a third judgment result of misassociation; if no, the generated two associated images are a third judgment result of misassociation. The drivable area of ​​the second vehicle is determined based on the segmented image of the second image.

6. The method according to claim 4, characterized in that, The step of generating a second judgment result based on the extracted feature information of the two associated images includes: Extract the feature information of the two associated images to obtain the feature information of the first image and the feature information of the second image in the two associated images; The feature information of the first image and the feature information of the second image are matched to determine the number of successfully matched feature points; A second judgment result is generated based on the number of successfully matched feature points and a third preset threshold.

7. An image association device, characterized in that, The device includes: An acquisition unit is used to acquire multiple images to be associated, each of which has corresponding image information, including positioning coordinates. The determining unit is configured to determine at least one spatial data based on the positioning coordinates in the image information corresponding to the multiple images to be associated if it is determined that the multiple images to be associated do not belong to the same driving data. The driving data includes multiple frames of images generated by the vehicle and image information corresponding to each frame of the multiple frames. The association unit is used to associate the image to be associated with the target spatial data that meets the preset conditions in the at least one spatial data. The device is also used for: A first judgment result is generated based on the image information corresponding to the two associated images. The first judgment result is used to determine whether there is a misassociation between the two associated images. The two associated images include a first image and a second image. The process includes: determining the first coordinates of the first vehicle that generated the first image from the image information corresponding to the first image; projecting the first vehicle onto the pixel coordinate system of the second image based on the first coordinates; and generating a third judgment result based on the pixel coordinate system of the projected second image. Then, determining the second coordinates of the second vehicle that generated the second image from the image information corresponding to the second image; projecting the second vehicle onto the pixel coordinate system of the first image based on the second coordinates; and generating a fourth judgment result based on the pixel coordinate system of the projected first image. Finally, a first judgment result is generated based on the third judgment result and the fourth judgment result.

8. An electronic device, characterized in that, include: Memory; processor; as well as Computer programs; The computer program is stored in the memory and configured to be executed by the processor to implement the image association method as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the image association method as described in any one of claims 1 to 6.

Images