Data acquisition method, model training method, and vehicle trajectory fusion method and device

Abstract

The embodiment of the invention discloses a vehicle annotation data acquisition method, a model training method, a vehicle track fusion method and related devices. According to the main technical scheme, the method comprises the steps of obtaining an image sequence collected by air collection equipment for a target area, wherein the image sequence comprises a first image of a first vehicle; acquiring a second image acquired by the ground acquisition equipment for the target area, wherein the second image comprises a second vehicle; constructing a three-dimensional map of the target area according to the image sequence, and generating an orthographic projection drawing according to the three-dimensional map; determining a spatial mapping relation among the orthographic projection drawing, the first image and the second image; extracting first annotation data of the first vehicle in the first image; and identifying that the first vehicle and the second vehicle are the same vehicle according to the spatial mapping relationship, and determining second annotation data of the second vehicle in the second image according to the first annotation data. The accuracy and efficiency of vehicle marking can be improved, the difficulty of vehicle track fusion is reduced, and the fusion effect is improved.

Description

Technical Field

[0001] The present application relates to the field of vehicle-road collaboration technology, and in particular to a method for acquiring vehicle annotation data, a method for training a vehicle annotation model, a vehicle trajectory fusion method and related devices. Background Art

[0002] In modern intelligent transportation systems, the perception ability of vehicles is limited by the sensors and computing power they carry. They can usually only monitor targets within a few hundred meters around them, and cannot effectively detect targets beyond this range. In order to overcome this limitation and achieve target monitoring at a longer distance, with higher accuracy and a more comprehensive field of view, vehicle-road collaboration technology came into being.

[0003] At present, vehicle-road collaboration technology mainly relies on 3D target detection to achieve vehicle trajectory fusion. This method first uses LiDAR to obtain the 3D annotation of the vehicle, then learns the 3D information of the target from the image through a deep learning model, and finally performs trajectory fusion through the 3D information of the target under multiple cameras. However, this method has significant limitations, such as the need for high consistency of camera internal parameters, the high cost of obtaining 3D annotation data, and the difficulty in ensuring consistency in mass production between different intersections or different cameras at the same intersection. Summary of the invention

[0004] The present application provides a method for acquiring vehicle annotation data, a method for training a vehicle annotation model, a vehicle trajectory fusion method and related devices, so as to improve the accuracy and efficiency of vehicle annotation, thereby reducing the difficulty of vehicle trajectory fusion and improving the fusion effect.

[0005] This application provides the following solutions:

[0006] According to a first aspect, a method for acquiring vehicle annotation data is provided, comprising:

[0007] Acquire an image sequence acquired by an aerial acquisition device for a target area, wherein the image sequence includes a first image of a first vehicle;

[0008] Acquire a second image acquired by a ground acquisition device for the target area, wherein the second image includes a second vehicle; wherein the first image and the second image are synchronized in time;

[0009] constructing a three-dimensional map of the target area according to the image sequence, and generating an orthographic projection image of the target area according to the three-dimensional map;

[0010] Determining a spatial mapping relationship between the orthographic projection image, the first image, and the second image;

[0011] Extracting first annotation data of the first vehicle in the first image;

[0012] When the first vehicle and the second vehicle are identified as the same vehicle according to the spatial mapping relationship, second annotation data of the second vehicle in the second image is determined according to the first annotation data.

[0013] According to one embodiment of the present invention, extracting first annotation data of the first vehicle in the first image specifically includes:

[0014] Performing target detection on the first image through a first target detection model, generating a first vehicle position frame of the first vehicle in the first image and a first heading angle of the first vehicle relative to the aerial acquisition device; the first vehicle position frame includes: the center point coordinates, length and width of the first vehicle in the first image;

[0015] The first vehicle position frame and the first heading angle are used as the first annotation data.

[0016] According to one embodiment of the present invention, the spatial mapping relationship includes:

[0017] a first spatial mapping relationship between the first image and the orthographic projection image, and a second spatial mapping relationship between the first image and the second image;

[0018] When the first vehicle and the second vehicle are identified as the same vehicle according to the spatial mapping relationship, determining second annotation data of the second vehicle in the second image according to the first annotation data specifically includes:

[0019] identifying whether the first vehicle and the second vehicle are the same vehicle according to the first spatial mapping relationship and the second spatial mapping relationship;

[0020] When the first vehicle and the second vehicle are identified as the same vehicle, second annotation data of the second vehicle in the second image is determined according to the first annotation data.

[0021] According to an embodiment of the present invention, based on the first spatial mapping relationship and the second spatial mapping relationship, identifying whether the first vehicle and the second vehicle are the same vehicle specifically includes:

[0022] Performing target detection on the second image by using a second target detection model to generate a second vehicle position frame of the second vehicle in the second image, wherein the second vehicle position frame includes the center point coordinates, length, and width of the second vehicle in the second image;

[0023] Based on the first spatial mapping relationship and the second spatial mapping relationship, projecting the first vehicle position frame into the second image to obtain a quadrilateral frame, wherein the quadrilateral frame includes the coordinates of the projection points of the first vehicle in the second image and the coordinates of the center point of the first vehicle in the second image;

[0024] The second vehicle position frame is matched with the quadrilateral frame, and when the matching is successful, the first vehicle and the second vehicle are identified as the same vehicle.

[0025] According to an embodiment of the present invention, determining second annotation data of the second vehicle in the second image according to the first annotation data specifically includes:

[0026] determining a physical length and a physical width of the first vehicle based on the length and the width of the first vehicle in the first image;

[0027] Converting a first heading angle of the first vehicle relative to the aerial acquisition device into a second heading angle of the first vehicle relative to the ground acquisition device;

[0028] The physical length, physical width, second heading angle, projection point coordinates of the first vehicle in the second image, and center point coordinates of the first vehicle in the second image are determined as the second annotation data.

[0029] According to one embodiment of the present invention, converting the first heading angle of the first vehicle relative to the aerial acquisition device into the second heading angle of the first vehicle relative to the ground acquisition device specifically includes:

[0030] The long side of the quadrilateral frame is determined according to the coordinates of the projection point of the first vehicle in the second image, and the orientation of the long side is used as the second heading angle of the first vehicle relative to the ground acquisition device.

[0031] According to one embodiment of the present invention, the first spatial mapping relationship is constructed as follows:

[0032] Selecting an image from the image sequence, and selecting a plurality of first feature points from the selected image;

[0033] Selecting second feature points from the orthographic projection image that correspond to the same physical position as the first feature points;

[0034] Obtaining a mapping matrix according to the first feature point and the second feature point;

[0035] According to the mapping matrix, pixels in the selected image are converted into the orthographic projection image to obtain the first spatial mapping relationship.

[0036] According to one embodiment of the present invention, after acquiring the image sequence, the method further includes:

[0037] The image sequence is subjected to de-jitter processing, wherein the de-jitter processing comprises: taking a starting image in the image sequence as a reference, aligning each image subsequent to the starting image to the starting image, so as to obtain a processed image sequence.

[0038] According to a second aspect, a method for training a vehicle labeling model is provided, comprising:

[0039] Acquire first training data including a plurality of first training samples, wherein the first training samples at least include: a second image of a second vehicle acquired by a ground acquisition device, and second annotation data of the second vehicle in the second image; wherein the second annotation data is acquired by the above-mentioned vehicle annotation data acquisition method;

[0040] The vehicle labeling model is trained based on the first training data, wherein the training includes: taking the second image as the input of the vehicle labeling model, obtaining third labeling data of the second vehicle predicted by the vehicle labeling model, and updating the model parameters of the vehicle labeling model using the loss function value corresponding to the first training objective, wherein the first training objective includes: minimizing the difference between the third labeling data output by the vehicle labeling model and the corresponding second labeling data.

[0041] According to one embodiment of the present invention, the vehicle annotation model includes: a feature extraction network, a bounding box branch network, a classification branch network, a vehicle length and width branch network, a vehicle center point branch network, a projection point branch network and a heading angle branch network, wherein:

[0042] The feature extraction network is used to extract feature representation of the second image;

[0043] The bounding box branch network is used to predict a bounding box of the second vehicle in the second image based on a feature representation of the second image;

[0044] The classification branch network is used to predict the category of the second vehicle in the second image based on the feature representation of the second image;

[0045] The vehicle length and width branch network is used to predict the physical length and physical width of the second vehicle based on the feature representation of the second image;

[0046] The vehicle center point branch network is used to predict the center point coordinates of the second vehicle in the second image based on the feature representation of the second image;

[0047] The vehicle projection point branch network is used to predict the projection point coordinates of the second vehicle in the second image based on the feature representation of the second image;

[0048] The heading angle branch network is used to predict a second heading angle of the second vehicle relative to the ground acquisition device based on a feature representation of the second image.

[0049] According to one embodiment of the present invention, the vehicle annotation model further includes a target deep feature branch network, and the deep feature branch network is used to predict the deep features of the second vehicle based on the feature representation of the second image.

[0050] According to a third aspect, a vehicle trajectory fusion method is provided, comprising:

[0051] Acquire multiple images within the target area based on multiple ground acquisition devices; the images contain vehicles;

[0052] Inputting the multiple images into a vehicle annotation model respectively, and obtaining vehicle annotation information corresponding to each image obtained by the vehicle annotation model according to each image;

[0053] Vehicle track fusion is performed according to the vehicle labeling information corresponding to each image.

[0054] According to an embodiment of the present invention, vehicle track fusion is performed according to the vehicle annotation information corresponding to each image, specifically including:

[0055] Projecting the vehicle annotation information corresponding to each image into a reference coordinate system to obtain a vehicle projection frame corresponding to each image;

[0056] The vehicles corresponding to the multiple vehicle projection frames whose overlap degree is greater than a preset threshold are determined as the same target vehicle;

[0057] The motion track of the target vehicle is determined according to the comprehensive ground projection frames of the same target vehicle at different time points.

[0058] According to one embodiment of the present invention, vehicles corresponding to a plurality of vehicle projection frames whose overlap degree is greater than a preset threshold are determined as the same target vehicle, specifically including:

[0059] Determine the weight of each vehicle projection frame; wherein the weight is determined based on the distance between the vehicle corresponding to the vehicle projection frame and each of the ground acquisition devices;

[0060] Performing weighted fusion on the vehicle projection frames corresponding to each of the images to obtain a fused comprehensive vehicle projection frame;

[0061] The vehicles corresponding to the multiple integrated vehicle projection frames whose overlap degree is greater than a preset first threshold value are determined to be the same target vehicle.

[0062] According to one embodiment of the present invention, the reference coordinate system is a pre-obtained orthographic projection of the target area.

[0063] According to one embodiment of the present invention, the vehicle annotation information includes: the physical length and physical width of the vehicle, the heading angle of the vehicle from the perspective of the ground acquisition device, and the projection point coordinates and center point coordinates of the vehicle in the image frame.

[0064] According to an embodiment of the present invention, the vehicle annotation information may further include: depth feature information of the vehicle;

[0065] According to the vehicle annotation information corresponding to each image, vehicle trajectory fusion is performed, specifically including:

[0066] Determining the cosine similarity between the vehicles corresponding to each image according to the depth feature information of the vehicle corresponding to each image;

[0067] Determine the vehicles whose cosine similarity is greater than a preset second threshold as the same target vehicle;

[0068] The motion track of the target vehicle is determined according to the comprehensive ground projection frames of the same target vehicle at different time points.

[0069] According to a fourth aspect, a device for acquiring vehicle labeling data is provided, comprising:

[0070] A first image acquisition unit is configured to acquire an image sequence acquired by an aerial acquisition device for a target area, wherein the image sequence includes a first image of a first vehicle;

[0071] A second image acquisition unit is configured to acquire a second image acquired by a ground acquisition device for the target area, wherein the second image includes a second vehicle; wherein the first image and the second image are time synchronized;

[0072] an orthographic projection image generating unit, configured to construct a three-dimensional map of the target area according to the image sequence, and generate an orthographic projection image of the target area according to the three-dimensional map;

[0073] a spatial mapping relationship determining unit, configured to determine a spatial mapping relationship between the orthographic projection image, the first image, and the second image;

[0074] A first annotation data extraction unit is configured to extract first annotation data of the first vehicle in the first image;

[0075] The second annotation data determining unit is configured to determine second annotation data of the second vehicle in the second image according to the first annotation data when the first vehicle and the second vehicle are identified as the same vehicle according to the spatial mapping relationship.

[0076] According to a fifth aspect, a training device for a vehicle labeling model is provided, comprising:

[0077] A sample acquisition unit is configured to acquire first training data including multiple first training samples, wherein the first training samples at least include: a second image of a second vehicle acquired by a ground acquisition device, and second annotation data of the second vehicle in the second image; wherein the second annotation data is obtained using the method of the above-mentioned implementation mode; a training unit is configured to train a vehicle annotation model based on the first training data, wherein the training includes: taking the second image as an input of the vehicle annotation model, acquiring third annotation data of the second vehicle predicted by the vehicle annotation model, and updating model parameters of the vehicle annotation model using a loss function value corresponding to a first training objective, wherein the first training objective includes: minimizing the difference between the third annotation data output by the vehicle annotation model and the corresponding second annotation data.

[0078] According to a sixth aspect, a vehicle trajectory fusion device is provided, comprising:

[0079] An image acquisition unit is configured to acquire a plurality of images in a target area based on a plurality of ground acquisition devices; the images include vehicles;

[0080] a labeling information acquisition unit, configured to input the plurality of images into a vehicle labeling model respectively, and acquire vehicle labeling information corresponding to each image obtained by the vehicle labeling model according to each image;

[0081] The track fusion unit is configured to perform vehicle track fusion according to the vehicle labeling information corresponding to each image.

[0082] According to a seventh aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the steps of the method described in any one of the first aspects are implemented.

[0083] According to an eighth aspect, there is provided an electronic device, comprising:

[0084] one or more processors; and

[0085] A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of the first aspects above.

[0086] In a ninth aspect, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the steps of any one of the methods described in the first aspect.

[0087] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0088] The embodiment of the present application uses aerial data collection equipment instead of traditional laser radar to collect data on the target area. When a vehicle is captured by the aerial data collection equipment and the ground data collection equipment at the same time, the vehicle annotation data obtained by the aerial data collection equipment can be used to provide the ground data collection equipment with more accurate vehicle annotation information. This process not only simplifies the data collection process and reduces the difficulty of obtaining vehicle annotation data, but also significantly improves the accuracy and acquisition efficiency of vehicle annotation data, ultimately reducing the difficulty of vehicle trajectory fusion and improving the fusion effect.

[0089] Of course, any product implementing the present application does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0091] Figure 1 is a flow chart of a method for acquiring vehicle annotation data provided in an embodiment of the present application;

[0092] Figure 2 A schematic diagram of feature point selection provided in an embodiment of the present application;

[0093] Figure 3 A schematic diagram of a detection result of a first image provided in an embodiment of the present application;

[0094] Figure 4 A schematic diagram of a quadrilateral frame displayed in a second image provided in an embodiment of the present application;

[0095] Figure 5 A flowchart of a method for training a vehicle annotation model provided in an embodiment of the present application;

[0096] Figure 6 A schematic diagram of a vehicle annotation model provided in an embodiment of the present application;

[0097] Figure 7 A flow chart of a method for vehicle trajectory fusion provided in an embodiment of the present application;

[0098] Figure 8 A projection diagram under the reference coordinate system provided in the embodiment of the present application;

[0099] Fig. 9 A schematic diagram of the camera coverage provided in an embodiment of the present application;

[0100] Fig.10 A schematic block diagram of a device for acquiring vehicle annotation data provided in an embodiment of the present application;

[0101] Fig.11 A schematic block diagram of a training device for a vehicle annotation model provided in an embodiment of the present application;

[0102] Fig.12 A schematic block diagram of a vehicle trajectory fusion device provided in an embodiment of the present application;

[0103] Fig.13 A schematic block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0104] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0105] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0106] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0107] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0108] In the vehicle-road cooperative system, in order to achieve the fusion of vehicle trajectories, one of the most mainstream implementation methods is to use 3D target detection technology. This method first uses lidar to capture the 3D annotation information of the target in the camera, then trains the model to extract the 3D information of the target from the image, and finally uses multiple cameras to perform trajectory fusion of the 3D information of the target. However, this method faces two major challenges. First, the consistency of the camera's internal parameters is extremely high, and in actual applications, the internal parameters of different sections and different cameras in the same section are often inconsistent, which has a significant impact on large-scale production and promotion. Secondly, the high cost of obtaining accurate 3D target detection annotation data also increases the difficulty and complexity of technical implementation. Another method is to use vanishing point and depth estimation algorithms to calculate the pseudo 3D information of the target and fuse it through this information. This method can also achieve a certain degree of three-dimensional target perception without too much annotation data, but it also shows certain limitations in actual applications. For example, each intersection needs to be debugged separately, and terrain changes can easily affect the fusion effect of this method, resulting in a limited scope of application.

[0109] In view of this, the present application provides a new idea and a new method for obtaining vehicle annotation data. Figure 1 The flowchart of the vehicle labeling data acquisition method provided in the embodiment of the present application, the execution subject of the method can be a vehicle labeling data acquisition device, the device can be set in any computer device with data storage and processing capabilities, can be set in the server side, and can also be set in a computer terminal with strong data processing capabilities. The method may include the following steps:

[0110] Step 101: Acquire an image sequence captured by an aerial acquisition device for a target area, where the image sequence includes a first image of a first vehicle;

[0111] Step 102: Acquire a second image captured by a ground acquisition device for the target area, where the second image includes a second vehicle; wherein the first image and the second image are synchronized in time;

[0112] Step 103: constructing a three-dimensional map of the target area according to the image sequence, and generating an orthographic projection map of the target area according to the three-dimensional map;

[0113] Step 104: Determine a spatial mapping relationship between the orthographic projection image, the first image, and the second image;

[0114] Step 105: extracting first annotation data of the first vehicle in the first image;

[0115] Step 106: When the first vehicle and the second vehicle are identified as the same vehicle according to the spatial mapping relationship, second annotation data of the second vehicle in the second image is determined according to the first annotation data.

[0116] It can be seen that the embodiment of the present application uses aerial collection equipment instead of traditional lidar to collect data on the target area. When the vehicle is captured by the aerial collection equipment and the ground collection equipment at the same time, the vehicle annotation data obtained by the aerial collection equipment can be used to provide the ground collection equipment with more accurate vehicle annotation information. This process not only simplifies the data collection process and reduces the difficulty of obtaining vehicle annotation data, but also significantly improves the accuracy and acquisition efficiency of vehicle annotation data, ultimately reducing the difficulty of vehicle trajectory fusion and improving the fusion effect.

[0117] The following is a detailed description of each step in the above process and the effects that can be further produced in conjunction with the embodiments. It should be noted that the limitations such as "first" and "second" involved in the present disclosure do not have limitations in terms of size, order and quantity, and are only used to distinguish between the names. For example, "first vehicle" and "second vehicle" are only used to distinguish between two vehicles in name. "First image" and "second image" are only used to distinguish between two images in name. "First annotation data" and "second annotation data" are only used to distinguish between two annotation data in name. And so on.

[0118] First, the above step 101, namely, "obtaining an image sequence acquired by an aerial acquisition device for a target area, wherein the image sequence includes a first image of a first vehicle" is described in detail in conjunction with the embodiment.

[0119] In the embodiments of the present application, the aerial data collection device refers to a device that can fly in the air and perform data collection tasks. Such devices are broad in scope and can include, but are not limited to, drones, flying robots, aircraft, and high-altitude balloons. They can be equipped with a variety of sensors and cameras to obtain image sequences of the target area. The target area can specifically refer to a certain intersection or a section of road area.

[0120] In practical applications, the aerial data collection equipment is deployed above the target area to determine the flight altitude and position to cover the entire target area. The flight altitude needs to be strictly controlled. If the flight altitude is too low, the accuracy of the vehicle annotation data may be impaired. On the contrary, if the flight altitude is too high, the size of the vehicle in the target area will appear too small, making the annotation work complicated. Taking a drone as an example, in the embodiment of the present application, the drone is used to be deployed above a specific intersection area for overhead photography, and its flight altitude is preferably maintained at about 200 meters.

[0121] In addition, the first image of the first vehicle refers to an image captured in an image sequence captured by an aerial acquisition device in the target area, including the image of the first vehicle, which is a single image in the image sequence, and they can show the position, state and characteristics of the first vehicle at a specific moment. It should be noted that the first vehicle can refer to any one of all vehicles appearing in the image sequence captured by the aerial acquisition device in the target area, rather than a specific vehicle.

[0122] In addition, since the aerial acquisition equipment records at high altitudes, it is affected by weather factors, such as strong winds, which will cause drift and other problems, thereby affecting the spatial mapping relationship between the orthographic projection image, the first image and the second image. Therefore, after acquiring the image sequence, the image sequence can be de-jittered first, and the starting image in the image sequence is used as a reference, and each image subsequent to the starting image is aligned to the starting image to obtain a processed image sequence.

[0123] Specifically, the first frame of the image sequence is selected, and then the next frame of the image is matched with the first frame of the image by SIFT (Scale Invariant Feature Transform) or (Accelerated Robust Feature) key point matching, and then the single projection matrix, that is, the transformation matrix from the starting image to the current image, is calculated based on the matched key point pairs. The single projection matrix can map each subsequent frame of the image to the coordinate system of the starting image to achieve image alignment.

[0124] The above step 102, namely "obtaining a second image captured by a ground acquisition device for the target area, wherein the second image includes a second vehicle; wherein the first image and the second image are time synchronized", is described in detail below in conjunction with an embodiment.

[0125] In the embodiments of the present application, the ground acquisition device refers to a device installed on the ground and capable of collecting data from the target area. Such devices may include fixed cameras, mobile monitoring vehicles, roadside sensors, etc., all of which can be equipped with necessary sensors and cameras to obtain image sequences of the target area. The target area can also be a specific intersection or road section.

[0126] In practical applications, ground acquisition equipment can be arranged around the target area. The deployment position and angle of these devices need to be precisely adjusted to ensure that the best vehicle images can be obtained and time synchronization with the image sequence of the aerial acquisition equipment can be achieved. Taking a fixed camera as an example, in an embodiment of the present application, the camera is installed on an elevated structure or a dedicated monitoring pole near the intersection to obtain the best viewing angle of the intersection area.

[0127] The ground acquisition device may also acquire an image sequence including multiple images for the target area, and the second image is a single frame in the image sequence acquired by the ground acquisition device, which can show the position, state and characteristics of the second vehicle at a specific moment. Similar to the first vehicle, the second vehicle here refers to any one of all vehicles appearing in the second image captured by the ground acquisition device in the target area, rather than a specific vehicle.

[0128] In order to achieve time synchronization between the first image and the second image, the embodiment of the present application may adopt high-precision time synchronization technology, such as GPS (Global Positioning System) timing, to ensure that the images collected in the air and on the ground can accurately correspond in time. Since the aerial acquisition device and the ground acquisition device may have different frame rates, this may cause a difference in the number of vehicle images in the image sequence collected by the aerial acquisition device and the image sequence collected by the ground acquisition device. Therefore, the first image and the second image in the embodiment of the present application refer to any two images in their respective image sequences whose timestamps can match each other. This flexible matching method can accurately identify and compare the vehicle status at the same time in image sequences (or video streams) with different frame rates.

[0129] The above step 103, namely "constructing a three-dimensional map of the target area according to the image sequence, and generating an orthographic projection image of the target area according to the three-dimensional map" is described in detail below in conjunction with an embodiment.

[0130] In the embodiment of the present application, a three-dimensional map of the target area can be constructed based on the image sequence collected by the aerial acquisition device, and an orthographic projection map of the target area can be generated according to the three-dimensional map, that is, based on the three-dimensional map, the three-dimensional map is converted into a two-dimensional orthographic projection map through the orthographic projection technology. Among them, the orthographic projection map is a map representation method that projects objects on the ground in a direction perpendicular to the ground to provide a two-dimensional view from a perspective directly above.

[0131] The above step 104, namely "determining the spatial mapping relationship between the orthographic projection image, the first image and the second image", is described in detail below in conjunction with an embodiment.

[0132] The spatial mapping relationship in the embodiment of the present application may include:

[0133] A first spatial mapping relationship between the first image and the orthographic projection image, and a second spatial mapping relationship between the first image and the second image.

[0134] As one of the achievable methods, the first space mapping relationship can be constructed as follows:

[0135] Selecting an image from the image sequence, and selecting a plurality of first feature points from the selected image;

[0136] Selecting second feature points corresponding to the same physical position as the first feature points from the orthographic projection image;

[0137] According to the first feature point and the second feature point, a mapping matrix is ​​obtained;

[0138] According to the mapping matrix, the pixels in the selected image are converted into the orthographic projection image to obtain a first spatial mapping relationship.

[0139] The images selected from the image sequence should be representative images that can cover key features of the target area, such as the overall view of the intersection and the layout of the road.

[0140] In addition, multiple first feature points can be selected from the selected image by feature point matching or manual point selection. If the feature point matching method is used, a computer vision algorithm can be used to automatically detect key points in the image. If the manual point selection method is used, points that are easy to identify in the image and have obvious corresponding objects in the orthographic projection image can be selected, such as road intersections, corner points of buildings, etc.

[0141] like Figure 2 As shown in (a), four feature points are selected from the selected image and marked as P1, P2, P3 and P4 respectively. Figure 2 As shown in (b), the physical corresponding positions of these feature points are also found on the orthographic projection image, also marked as P1, P2, P3 and P4, and then the selected points are input into the FindHomograhph function in the Opencv library to obtain a 3x3 homography matrix. Using this matrix, the pixel points in the first image can be accurately converted to the orthographic projection image, or the pixel points in the orthographic projection image can be reversely mapped back to the first image, thereby establishing a first spatial mapping relationship between the first image and the orthographic projection image.

[0142] It should be noted that the physical position accuracy of the selected first feature point and the second feature point is crucial to establishing an accurate first spatial mapping relationship. If there is a positioning error between these feature points, the final mapping relationship may be inaccurate. Therefore, after obtaining the first spatial mapping relationship, further verification can be performed.

[0143] Specifically, the calculated homography matrix can be applied to each pixel of the first image. If the image after application completely coincides with the orthographic projection image, it means that the obtained homography matrix is ​​accurate, thereby ensuring the accuracy of the first spatial mapping relationship.

[0144] The process of constructing the second spatial mapping relationship is similar to the process of constructing the first spatial mapping relationship, and will not be described in detail here.

[0145] The above step 105, namely "extracting first annotation data of the first vehicle in the first image", is described in detail below in conjunction with an embodiment.

[0146] As one of the feasible methods, the embodiment of the present application can perform target detection on the first image through a first target detection model, generate a first vehicle position frame of the first vehicle in the first image and a first heading angle of the first vehicle relative to the aerial acquisition device, where the first vehicle position frame includes: the center point coordinates, length and width of the first vehicle in the first image; finally, the first vehicle position frame and the first heading angle are used as the first annotation data.

[0147] The first annotation data here is the 2.5D annotation data of the first vehicle from the perspective of the aerial acquisition device. It is called 2.5D annotation data because it contains the position (2D information) and heading angle (a kind of 2.5D depth information) of the vehicle in the image, but does not have complete three-dimensional spatial coordinates.

[0148] It should be noted that the center point coordinates, length and width of the first vehicle in the first image contained in the first vehicle position frame here are all relative to the pixel size of the first image, that is, the center point coordinates represent the horizontal and vertical positions of the vehicle in the first image, usually in pixels, calculated from the upper left corner of the first image; the length and width are in pixels, indicating the size of the vehicle in the first image.

[0149] In practical applications, the embodiments of the present application can use a high-precision rotating target detection model and a multi-target tracker to perform multi-target detection on the first image in the image sequence. The detection results are as follows: Figure 3 As shown in the figure, the blue box represents the first detected vehicle position box. In addition, each detected target object is assigned a unique tracking identifier, marked as "trkid: number". This tracking mechanism is because the frame rate of the image sequence collected by the aerial acquisition device is usually lower than the frame rate of the image collected by the ground acquisition device. Through tracking technology, the frame rate can be effectively improved, thereby reducing the difference in acquisition time with the ground acquisition device and improving the accuracy of data synchronization.

[0150] The above step 106, namely "determining second annotation data of the second vehicle in the second image according to the first annotation data when the first vehicle and the second vehicle are identified as the same vehicle according to the spatial mapping relationship" is described in detail below in conjunction with an embodiment.

[0151] Specifically, the spatial mapping relationship as described above may include a first spatial mapping relationship and a second spatial mapping relationship. Then, as one of the achievable ways, according to the first spatial mapping relationship and the second spatial mapping relationship, identifying whether the first vehicle and the second vehicle are the same vehicle may be implemented as follows:

[0152] First, target detection is performed on the second image using a second target detection model to generate a second vehicle position frame of the second vehicle in the second image. The second vehicle position frame includes the center point coordinates, length, and width of the second vehicle in the second image.

[0153] Among them, the center point coordinates, length and width of the second vehicle in the second image contained in the second vehicle position frame are all relative to the pixel size of the second image, that is, the center point coordinates represent the horizontal and vertical positions of the vehicle in the second image, usually in pixels, calculated from the upper left corner of the second image; the length and width are in pixels, indicating the size of the vehicle in the second image.

[0154] Secondly, based on the first spatial mapping relationship and the second spatial mapping relationship, the first vehicle position frame is projected into the second image to obtain a quadrilateral frame, which includes the projection point coordinates of the first vehicle in the second image and the center point coordinates of the first vehicle in the second image.

[0155] like Figure 4 As shown, the green box represents the quadrilateral box obtained by projecting the first vehicle position box into the second image, and the green point is the center point of the vehicle in the second image.

[0156] Finally, the second vehicle position frame and the quadrilateral frame are matched, and when the match is successful, the first vehicle and the second vehicle are identified as the same vehicle.

[0157] As a feasible method, the second vehicle position frame and the quadrilateral frame can be matched by IoU (Intersection over Union). Specifically, firstly, the intersection area (the area of ​​the overlapping part of the two frames) and the union area (the area of ​​all areas covered by the two frames) of the second vehicle position frame and the quadrilateral frame are calculated; then a threshold is set, and the IoU value of the second vehicle position frame and the quadrilateral frame is compared with the threshold; if the IoU value is greater than or equal to the threshold, it is considered that the first vehicle and the second vehicle are spatially overlapped, so they are identified as the same vehicle; if the IoU value is lower than the threshold, it is considered that the match fails, and the first vehicle and the second vehicle are not the same vehicle.

[0158] Furthermore, when the first vehicle and the second vehicle are identified as the same vehicle, the first vehicle and the second vehicle may be given the same identification.

[0159] When the first vehicle and the second vehicle are identified as the same vehicle, as a feasible method, the physical length and physical width of the first vehicle can be first determined based on the length and width of the first vehicle in the first image; then the first heading angle of the first vehicle relative to the aerial acquisition device is converted into the second heading angle of the first vehicle relative to the ground acquisition device; finally, the physical length, physical width, second heading angle of the first vehicle, the coordinates of the projection point of the first vehicle in the second image and the coordinates of the center point of the first vehicle in the second image are determined as the second annotation data.

[0160] The second annotation data here refers to the 2.5D annotation data of the second vehicle from the perspective of the ground acquisition device.

[0161] It should be noted that when determining the physical length and physical width of the first vehicle, firstly, based on the length and width (in pixels) of the first vehicle measured in the first image, and combined with the calibration parameters and image resolution of the camera, these pixel sizes can be converted into actual physical sizes (in meters). This conversion is based on the internal and external parameters of the camera, and may involve factors such as focal length, principal point position, and pixel size.

[0162] In addition, when converting the first heading angle and the second heading angle, it is necessary to consider the difference in viewing angles and convert the heading angle of the first vehicle relative to the aerial collection device into the heading angle relative to the ground collection device. As a feasible method, the long side of the quadrilateral box can be determined based on the coordinates of the projection point of the first vehicle in the second image, and the direction of the long side can be used as the second heading angle of the first vehicle relative to the ground collection device.

[0163] The embodiment of the present application uses aerial data collection equipment instead of traditional laser radar to collect data on the target area. When a vehicle is captured by the aerial data collection equipment and the ground data collection equipment at the same time, the vehicle annotation data obtained by the aerial data collection equipment can be used to provide the ground data collection equipment with more accurate vehicle annotation information. This process not only simplifies the data collection process and reduces the difficulty of obtaining vehicle annotation data, but also significantly improves the accuracy and acquisition efficiency of vehicle annotation data, ultimately reducing the difficulty of vehicle trajectory fusion and improving the fusion effect.

[0164] Based on the above-mentioned method for obtaining vehicle annotation information, the embodiment of the present application also provides a method for training a vehicle annotation model, such as Figure 5 As shown, the following steps are included:

[0165] Step 501: Acquire first training data including a plurality of first training samples, wherein the first training sample at least includes: a second image of a second vehicle acquired by a ground acquisition device, and second annotation data of the second vehicle in the second image;

[0166] The second annotation data may be obtained by using the above-mentioned method for obtaining vehicle annotation data.

[0167] Step 502: Training a vehicle labeling model based on the first training data, the training comprising: taking the second image as the input of the vehicle labeling model, obtaining third labeling data of the second vehicle predicted by the vehicle labeling model, and updating model parameters of the vehicle labeling model using the loss function value corresponding to the first training objective, wherein the first training objective comprises: minimizing the difference between the third labeling data output by the vehicle labeling model and the corresponding second labeling data.

[0168] As one of the achievable methods, the vehicle annotation model in the embodiment of the present application may include a feature extraction network, a bounding box branch network, a classification branch network, a vehicle length and width branch network, a vehicle center point branch network, a projection point branch network, and a heading angle branch network, wherein:

[0169] The feature extraction network is used to extract feature representation of the second image;

[0170] The bounding box branch network is used to predict a bounding box of a second vehicle in the second image based on a feature representation of the second image;

[0171] The classification branch network is used to predict the category of the second vehicle in the second image based on the feature representation of the second image;

[0172] The vehicle length and width branch network is used to predict the physical length and physical width of the second vehicle based on the feature representation of the second image;

[0173] The vehicle center point branch network is used to predict the center point coordinates of the second vehicle in the second image based on the feature representation of the second image;

[0174] The vehicle projection point branch network is used to predict the projection point coordinates of the second vehicle in the second image based on the feature representation of the second image;

[0175] The heading angle branch network is used to predict a second heading angle of the second vehicle relative to the ground acquisition device based on the feature representation of the second image.

[0176] Furthermore, the vehicle annotation model may also include a target deep feature branch network, and the deep feature branch network is used to predict the deep features of the second vehicle based on the feature representation of the second image.

[0177] The following is a detailed description of the training process of the vehicle labeling model with examples.

[0178] First, it is necessary to prepare the first training data, which includes multiple first training samples obtained from ground acquisition equipment, each training sample contains at least a second image of the second vehicle, and corresponding second annotation data. These second annotation data can be obtained through the above-mentioned vehicle annotation data acquisition method, including the second vehicle's position frame (center point coordinates, length and width) and heading angle and other information.

[0179] A deep learning model is pre-set. The structure of the deep learning model is as follows: Figure 6 As shown, it includes a feature extraction network composed of a backbone network and an FPN (Feature Pyramid Network), wherein the backbone network is responsible for extracting the features of the second image, and the FPN is used to construct a feature pyramid and integrate feature information of different scales.

[0180] Assuming that the size of the second image input to the vehicle annotation model is W×H, after S-fold downsampling, the size of the output feature map is Subsequent use of w and h respectively replace and And set w×h candidate boxes on the output feature map, and the size of each candidate box is S×S.

[0181] The vehicle labeling model includes multiple output branches, each responsible for a different task:

[0182] 1) Heading angle branch network: used to output the heading angle of the second vehicle in the second image.

[0183] The output dimension is the cosine and sine of the heading angle of the nearest positive sample of each candidate box, that is, cos(α) and sin(α). α is not directly regressed here to facilitate the convergence of model training.

[0184] The loss function is Loss 航向角 =cos(α)+sin(α).

[0185] 2) Vehicle length and width branch network: used to output the physical properties of the second vehicle, namely the physical length and physical width (unit: meter).

[0186] The output dimension is n×h×w×2, where n is the number of output channels.

[0187] The loss function is Loss 长宽 L1 loss can be used.

[0188] 3) Vehicle projection point branch network: used to output the coordinates of the projection point of the second vehicle in the second image, that is, the offset d between the center point of each candidate box and its nearest positive sample chassis projection point x 1.d y1.d x 2.d y 2.d x 3.d y 3.d x 4 and d y 4.

[0189] The output dimension is n×h×w×8.

[0190] The loss function is Loss 底盘投影点 L1 loss can be used.

[0191] 4) Vehicle center point branch network: used to output the center point coordinates of the second vehicle in the second image, that is, the deviation d between the center point of each candidate box and the center point of the chassis of its nearest positive sample x and d y , so the output dimension is n×h×w×2.

[0192] The loss function is Loss 中心点 L1 loss can be used.

[0193] 5) Target deep feature branch network: used to output the deep features of the second vehicle, that is, the 512-dimensional deep features of each candidate box, which are used for target matching and target re-identification of multiple ground acquisition devices (such as cameras). Therefore, the output dimension is n×h×w×512.

[0194] This branch network can be learned using a classification loss function, and only the output of the embedding layer is extracted during inference.

[0195] The loss function is Loss 深度特征 Group loss can be used.

[0196] In addition to the five specific output branches mentioned above, the model also contains a bounding box branch network (such as Figure 6 The BBox branch shown in Figure 2) and the classification branch network (shown in Figure 2) Figure 6 The bounding box branch network is responsible for predicting the bounding box position of the second vehicle in the second image, while the classification branch network is responsible for identifying the category of the second vehicle. These two output branches are common components in traditional vehicle annotation models and belong to the scope of the prior art. Therefore, the specific details of these two branches are not repeated here.

[0197] Therefore, based on the above description, the complete training loss of the entire vehicle labeling model consists of the following parts:

[0198] Total Loss = Lossbbox + Loss classification + Loss heading angle + Loss length and width + Loss projection point + Loss center point + Loss depth feature

[0199] The vehicle labeling model of the embodiment of the present application can provide more accurate vehicle labeling information by utilizing images collected by ground acquisition equipment. This model not only reduces the difficulty of obtaining vehicle labeling data, but also significantly improves the accuracy and acquisition efficiency of vehicle labeling data, ultimately reducing the difficulty of vehicle trajectory fusion and improving the fusion effect.

[0200] Based on the above vehicle labeling model, the embodiment of the present application also provides a method for vehicle trajectory fusion, such as Figure 7 As shown, the following steps are included:

[0201] Step 701: Acquire multiple images in a target area based on multiple ground acquisition devices, where the images contain vehicles;

[0202] Step 702: inputting a plurality of images into a vehicle annotation model respectively, and obtaining vehicle annotation information corresponding to each image obtained by the vehicle annotation model according to each image;

[0203] Step 703: Car track fusion is performed according to the vehicle annotation information corresponding to each image.

[0204] In step 701, first, multiple images in the target area need to be synchronously acquired from multiple ground acquisition devices, and the images contain vehicles. In the embodiment of the present application, high-precision time synchronization technology, such as GPS timing, can be used to ensure time synchronization of multiple ground acquisition devices.

[0205] In step 702, the vehicle annotation information may include: the physical length and width of the vehicle, the heading angle of the vehicle from the perspective of the ground acquisition device, the projection point coordinates of the vehicle in the image, and the center point coordinates.

[0206] The vehicle annotation model may include a feature extraction network, a bounding box branch network, a classification branch network, a vehicle length and width branch network, a vehicle center point branch network, a projection point branch network, and a heading angle branch network, wherein:

[0207] The feature extraction network is used to extract feature representation of the second image;

[0208] The bounding box branch network is used to predict a bounding box of a second vehicle in the second image based on a feature representation of the second image;

[0209] The classification branch network is used to predict the category of the second vehicle in the second image based on the feature representation of the second image;

[0210] The vehicle length and width branch network is used to predict the physical length and physical width of the second vehicle based on the feature representation of the second image;

[0211] The vehicle center point branch network is used to predict the center point coordinates of the second vehicle in the second image based on the feature representation of the second image;

[0212] The vehicle projection point branch network is used to predict the projection point coordinates of the second vehicle in the second image based on the feature representation of the second image;

[0213] The heading angle branch network is used to predict a second heading angle of the second vehicle relative to the ground acquisition device based on the feature representation of the second image.

[0214] Furthermore, the vehicle annotation model may also include a target deep feature branch network, and the deep feature branch network is used to predict the deep features of the second vehicle based on the feature representation of the second image.

[0215] The specific principle of the vehicle labeling model can be found in the relevant records in the previous embodiments, which will not be elaborated here.

[0216] In step 703, vehicle track fusion is performed according to the vehicle annotation information corresponding to each image, which may specifically include:

[0217] First, the vehicle annotation information corresponding to each image is projected into the reference coordinate system to obtain the vehicle projection frame corresponding to each image; then, the vehicles corresponding to multiple vehicle projection frames whose overlap is greater than a preset threshold are determined to be the same target vehicle; finally, the motion trajectory of the target vehicle is determined based on the comprehensive ground projection frames of the same target vehicle at different time points.

[0218] As a feasible method, the reference coordinate system may be a pre-obtained orthographic projection of the target area. Figure 8 As shown, the small picture in the upper right corner is an enlarged view of the dotted part of the picture in the lower left corner. The vehicle projection frames of different colors represent the vehicle annotation information from different ground acquisition devices. It can be seen that when the vehicle annotation information is projected into the orthographic projection map of the target area in the form of vehicle projection frames, the overlap of the vehicle projection frames of the same vehicle is very high.

[0219] In addition, the closer the vehicle is to the ground acquisition device, the more accurate the vehicle projection frame is. Therefore, as a feasible method, the vehicles corresponding to multiple vehicle projection frames with a degree of overlap greater than a preset threshold are determined to be the same target vehicle, which can be achieved as follows:

[0220] Determine the weight of each vehicle projection frame; wherein the weight is determined based on the distance between the vehicle corresponding to the vehicle projection frame and each ground acquisition device;

[0221] Perform weighted fusion on the vehicle projection frames corresponding to each image to obtain a fused comprehensive vehicle projection frame;

[0222] The vehicles corresponding to the multiple integrated vehicle projection frames whose overlap degree is greater than a preset threshold are determined to be the same target vehicle.

[0223] In an embodiment of the present application, the ground projection point of the target vehicle is known, and the longitude and latitude of the ground acquisition device and the target vehicle on the orthographic projection map can be determined through the projection relationship, and then the distance between the two can be calculated, and the weight of each ground acquisition device can be determined based on the distance between the two.

[0224] For example, when a target vehicle is captured by only one ground acquisition device, such as Fig. 9 1 area (green area), then the target vehicle will not be fused; when a target vehicle is captured by multiple ground acquisition devices, such as Fig. 9 2 areas (the overlapping areas of the red area and the green area), the weighted values ​​of the two ground acquisition devices can be used for output. When the sum of multiple weights is less than 1, the remaining weights are compensated by the maximum weight. For example, the weight of camera A is 0.6 and the weight of camera B is 0.3, then the remaining 0.1 is fused with camera A; if the sum of weights is greater than 1, for example, the weight of camera A is 0.8 and the weight of camera B is 0.4, then the higher weight is used first, that is, the excess 0.2 of camera B is discarded.

[0225] In addition, in addition to judging whether the vehicles in different images are the same vehicle based on the overlap of the vehicle projection frames, the embodiment of the present application can also use the depth features of the vehicle output by the vehicle annotation model to judge whether the vehicles in different images are the same vehicle. Specifically, the cosine similarity between the vehicles corresponding to each image can be calculated based on the depth feature information of the vehicles in different images, and the vehicles with cosine similarity greater than the preset second threshold value are determined to be the same target vehicle.

[0226] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0227] According to an embodiment of another aspect, a device for acquiring vehicle labeling data is provided. Fig.10 A schematic block diagram of a device for acquiring vehicle annotation data according to an embodiment is shown, wherein the device is arranged in Fig.10 As shown, the device 1000 includes:

[0228] The first image acquisition unit 1001 is configured to acquire an image sequence acquired by an aerial acquisition device for a target area, wherein the image sequence includes a first image of a first vehicle;

[0229] The second image acquisition unit 1002 is configured to acquire a second image acquired by a ground acquisition device for the target area, wherein the second image includes a second vehicle; wherein the first image and the second image are time synchronized;

[0230] An orthographic projection image generating unit 1003 is configured to construct a three-dimensional map of the target area according to the image sequence, and generate an orthographic projection image of the target area according to the three-dimensional map;

[0231] A spatial mapping relationship determining unit 1004 is configured to determine a spatial mapping relationship between the orthographic projection image, the first image and the second image;

[0232] A first annotation data extraction unit 1005 is configured to extract first annotation data of the first vehicle in the first image;

[0233] The second annotation data determining unit 1006 is configured to determine second annotation data of the second vehicle in the second image according to the first annotation data when the first vehicle and the second vehicle are identified as the same vehicle according to the spatial mapping relationship.

[0234] Optionally, the first annotation data extraction unit 1005 is configured to:

[0235] Performing target detection on the first image through a first target detection model, generating a first vehicle position frame of the first vehicle in the first image and a first heading angle of the first vehicle relative to the aerial acquisition device; the first vehicle position frame includes: the center point coordinates, length and width of the first vehicle in the first image;

[0236] The first vehicle position frame and the first heading angle are used as the first annotation data.

[0237] As an implementable manner, the spatial mapping relationship includes:

[0238] a first spatial mapping relationship between the first image and the orthographic projection image, and a second spatial mapping relationship between the first image and the second image;

[0239] The second label data determining unit 1006 is configured to:

[0240] identifying whether the first vehicle and the second vehicle are the same vehicle according to the first spatial mapping relationship and the second spatial mapping relationship;

[0241] When the first vehicle and the second vehicle are identified as the same vehicle, second annotation data of the second vehicle in the second image is determined according to the first annotation data.

[0242] Further, the second annotation data determining unit 1006 is configured to: identify whether the first vehicle and the second vehicle are the same vehicle based on the first spatial mapping relationship and the second spatial mapping relationship, specifically including:

[0243] Performing target detection on the second image by using a second target detection model to generate a second vehicle position frame of the second vehicle in the second image, wherein the second vehicle position frame includes the center point coordinates, length, and width of the second vehicle in the second image;

[0244] Based on the first spatial mapping relationship and the second spatial mapping relationship, projecting the first vehicle position frame into the second image to obtain a quadrilateral frame, wherein the quadrilateral frame includes the coordinates of the projection points of the first vehicle in the second image and the coordinates of the center point of the first vehicle in the second image;

[0245] The second vehicle position frame is matched with the quadrilateral frame, and when the matching is successful, the first vehicle and the second vehicle are identified as the same vehicle.

[0246] Furthermore, the second annotation data determining unit 1006 is configured to:

[0247] determining a physical length and a physical width of the first vehicle based on the length and the width of the first vehicle in the first image;

[0248] Converting a first heading angle of the first vehicle relative to the aerial acquisition device into a second heading angle of the first vehicle relative to the ground acquisition device;

[0249] The physical length, physical width, second heading angle, projection point coordinates of the first vehicle in the second image, and center point coordinates of the first vehicle in the second image are determined as the second annotation data.

[0250] Furthermore, the second annotation data determining unit 1006 is configured to:

[0251] The long side of the quadrilateral frame is determined according to the coordinates of the projection point of the first vehicle in the second image, and the orientation of the long side is used as the second heading angle of the first vehicle relative to the ground acquisition device.

[0252] Optionally, the spatial mapping relationship determining unit 1004 is configured to:

[0253] Selecting an image from the image sequence, and selecting a plurality of first feature points from the selected image;

[0254] Selecting second feature points from the orthographic projection image that correspond to the same physical position as the first feature points;

[0255] Obtaining a mapping matrix according to the first feature point and the second feature point;

[0256] According to the mapping matrix, pixels in the selected image are converted into the orthographic projection image to obtain the first spatial mapping relationship.

[0257] Optionally, the device 1000 may further include:

[0258] The de-jittering unit 1007 is configured to perform de-jittering processing on the image sequence, wherein the de-jittering processing includes: taking the starting image in the image sequence as a reference, aligning each image subsequent to the starting image to the starting image to obtain a processed image sequence.

[0259] According to an embodiment of another aspect, a training device for a vehicle labeling model is provided. Fig.11 A schematic block diagram of a training device for the vehicle labeling model according to an embodiment is shown, and the device is arranged in any computer device with data storage and processing capabilities. Fig.11 As shown, the device 1100 includes: a sample acquisition unit 1101 and a training unit 1102. The main functions of each component unit are as follows:

[0260] The sample acquisition unit 1101 is configured to acquire first training data including a plurality of first training samples, wherein the first training samples at least include: a second image of a second vehicle acquired by a ground acquisition device, and second annotation data of the second vehicle in the second image; wherein the second annotation data is acquired by the above-mentioned vehicle annotation data acquisition method;

[0261] The training unit 1102 is configured to train a vehicle labeling model based on the first training data, wherein the training includes: taking the second image as the input of the vehicle labeling model, obtaining third labeling data of the second vehicle predicted by the vehicle labeling model, and updating the model parameters of the vehicle labeling model using the loss function value corresponding to the first training objective, wherein the first training objective includes: minimizing the difference between the third labeling data output by the vehicle labeling model and the corresponding second labeling data.

[0262] Furthermore, the vehicle annotation model includes: a feature extraction network, a bounding box branch network, a classification branch network, a vehicle length and width branch network, a vehicle center point branch network, a projection point branch network and a heading angle branch network, wherein:

[0263] The feature extraction network is used to extract feature representation of the second image;

[0264] The bounding box branch network is used to predict a bounding box of the second vehicle in the second image based on a feature representation of the second image;

[0265] The classification branch network is used to predict the category of the second vehicle in the second image based on the feature representation of the second image;

[0266] The vehicle length and width branch network is used to predict the physical length and physical width of the second vehicle based on the feature representation of the second image;

[0267] The vehicle center point branch network is used to predict the center point coordinates of the second vehicle in the second image based on the feature representation of the second image;

[0268] The vehicle projection point branch network is used to predict the projection point coordinates of the second vehicle in the second image based on the feature representation of the second image;

[0269] The heading angle branch network is used to predict a second heading angle of the second vehicle relative to the ground acquisition device based on a feature representation of the second image.

[0270] Furthermore, the vehicle labeling model also includes a target deep feature branch network, and the deep feature branch network is used to predict the deep features of the second vehicle based on the feature representation of the second image.

[0271] According to an embodiment of another aspect, a vehicle trajectory fusion device is provided. Fig.12 A schematic block diagram of a vehicle trajectory fusion device according to an embodiment is shown, and the device is set in any computer device with data storage and processing capabilities. Fig.12 As shown, the device 1200 includes: an image acquisition unit 1201, a label information acquisition unit 1202 and a trajectory fusion unit 1203. The main functions of each component unit are as follows:

[0272] The image acquisition unit 1201 is configured to acquire multiple images in the target area based on multiple ground acquisition devices; the images include vehicles;

[0273] The annotation information acquisition unit 1202 is configured to input the plurality of images into a vehicle annotation model respectively, and acquire vehicle annotation information corresponding to each image obtained by the vehicle annotation model according to each image;

[0274] The track fusion unit 1203 is configured to perform vehicle track fusion according to the vehicle labeling information corresponding to each image.

[0275] Optionally, the trajectory fusion unit 1203 is configured to:

[0276] Projecting the vehicle annotation information corresponding to each image into a reference coordinate system to obtain a vehicle projection frame corresponding to each image;

[0277] The vehicles corresponding to the multiple vehicle projection frames whose overlap degree is greater than a preset threshold are determined as the same target vehicle;

[0278] The motion track of the target vehicle is determined according to the comprehensive ground projection frames of the same target vehicle at different time points.

[0279] Furthermore, the trajectory fusion unit 1203 is configured as follows:

[0280] Determine the weight of each vehicle projection frame; wherein the weight is determined based on the distance between the vehicle corresponding to the vehicle projection frame and each of the ground acquisition devices;

[0281] Performing weighted fusion on the vehicle projection frames corresponding to each of the images to obtain a fused comprehensive vehicle projection frame;

[0282] The vehicles corresponding to the multiple integrated vehicle projection frames whose overlap degree is greater than a preset first threshold value are determined to be the same target vehicle.

[0283] As an implementable manner, the reference coordinate system is a pre-obtained orthographic projection map of the target area.

[0284] Furthermore, the vehicle annotation information includes: the physical length and physical width of the vehicle, the heading angle of the vehicle from the perspective of the ground acquisition device, and the projection point coordinates and center point coordinates of the vehicle in the image frame.

[0285] Furthermore, the vehicle annotation information may also include: depth feature information of the vehicle;

[0286] The trajectory fusion unit 1203 is configured as follows:

[0287] Determining the cosine similarity between the vehicles corresponding to each image according to the depth feature information of the vehicle corresponding to each image;

[0288] Determine the vehicles whose cosine similarity is greater than a preset second threshold as the same target vehicle;

[0289] The motion track of the target vehicle is determined according to the comprehensive ground projection frames of the same target vehicle at different time points.

[0290] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0291] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0292] In addition, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of any one of the methods in the aforementioned method embodiments are implemented.

[0293] And an electronic device, comprising:

[0294] one or more processors; and

[0295] A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of the aforementioned method embodiments.

[0296] The present application also provides a computer program product, including a computer program, which implements the steps of any one of the methods in the aforementioned method embodiments when executed by a processor.

[0297] in, Fig.13The architecture of the electronic device is shown as an example, which may include a processor 1310, a video display adapter 1311, a disk drive 1312, an input / output interface 1313, a network interface 1314, and a memory 1320. The processor 1310, the video display adapter 1311, the disk drive 1312, the input / output interface 1313, the network interface 1314, and the memory 1320 may be communicatively connected via a communication bus 1330.

[0298] Among them, the processor 1310 can be implemented by a general-purpose CPU, a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., to execute relevant programs to implement the technical solutions provided in this application.

[0299] The memory 1320 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1320 can store an operating system 1321 for controlling the operation of the electronic device 1300, and a basic input and output system (BIOS) 1322 for controlling the low-level operation of the electronic device 1300. In addition, a web browser 1323, a data storage management system 1324, a vehicle annotation data acquisition device (or a vehicle annotation model training device or a vehicle trajectory fusion device) 1325, etc. can also be stored. The above-mentioned vehicle annotation data acquisition device (or a vehicle annotation model training device or a vehicle trajectory fusion device) 1325 can be an application program that specifically implements the aforementioned steps in the embodiment of the present application. In short, when the technical solution provided in the present application is implemented by software or firmware, the relevant program code is stored in the memory 1320 and is called and executed by the processor 1310.

[0300] The input / output interface 1313 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0301] The network interface 1314 is used to connect to a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0302] The bus 1330 comprises a pathway for transmitting information between the various components of the device (eg, the processor 1310, the video display adapter 1311, the disk drive 1312, the input / output interface 1313, the network interface 1314, and the memory 1320).

[0303] It should be noted that, although the above device only shows a processor 1310, a video display adapter 1311, a disk drive 1312, an input / output interface 1313, a network interface 1314, a memory 1320, a bus 1330, etc., in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include components necessary for implementing the solution of the present application, and does not necessarily include all the components shown in the figure.

[0304] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a computer program product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0305] The technical solution provided by the present application is described in detail above. The principle and implementation method of the present application are described in detail using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as limiting the present application.

Claims

1. A method for acquiring vehicle annotation data, characterized in that: include: Acquire an image sequence acquired by an aerial acquisition device for a target area, wherein the image sequence includes a first image of a first vehicle; Acquire a second image acquired by a ground acquisition device for the target area, wherein the second image includes a second vehicle; wherein the first image and the second image are synchronized in time; constructing a three-dimensional map of the target area according to the image sequence, and generating an orthographic projection image of the target area according to the three-dimensional map; Determining a spatial mapping relationship between the orthographic projection image, the first image, and the second image; Extracting first annotation data of the first vehicle in the first image; When the first vehicle and the second vehicle are identified as the same vehicle according to the spatial mapping relationship, second annotation data of the second vehicle in the second image is determined according to the first annotation data.

2. The method according to claim 1, characterized in that Extracting first annotation data of the first vehicle in the first image specifically includes: Performing target detection on the first image through a first target detection model, generating a first vehicle position frame of the first vehicle in the first image and a first heading angle of the first vehicle relative to the aerial acquisition device; the first vehicle position frame includes: the center point coordinates, length and width of the first vehicle in the first image; The first vehicle position frame and the first heading angle are used as the first annotation data.

3. The method according to claim 2, characterized in that The spatial mapping relationship includes: a first spatial mapping relationship between the first image and the orthographic projection image, and a second spatial mapping relationship between the first image and the second image; When the first vehicle and the second vehicle are identified as the same vehicle according to the spatial mapping relationship, determining second annotation data of the second vehicle in the second image according to the first annotation data specifically includes: identifying whether the first vehicle and the second vehicle are the same vehicle according to the first spatial mapping relationship and the second spatial mapping relationship; When the first vehicle and the second vehicle are identified as the same vehicle, second annotation data of the second vehicle in the second image is determined according to the first annotation data.

4. The method according to claim 3, characterized in that Based on the first spatial mapping relationship and the second spatial mapping relationship, identifying whether the first vehicle and the second vehicle are the same vehicle specifically includes: Performing target detection on the second image by using a second target detection model to generate a second vehicle position frame of the second vehicle in the second image, wherein the second vehicle position frame includes the center point coordinates, length, and width of the second vehicle in the second image; Based on the first spatial mapping relationship and the second spatial mapping relationship, projecting the first vehicle position frame into the second image to obtain a quadrilateral frame, wherein the quadrilateral frame includes the coordinates of the projection points of the first vehicle in the second image and the coordinates of the center point of the first vehicle in the second image; The second vehicle position frame is matched with the quadrilateral frame, and when the matching is successful, the first vehicle and the second vehicle are identified as the same vehicle.

5. The method according to claim 4, characterized in that Determining second annotation data of the second vehicle in the second image according to the first annotation data specifically includes: determining a physical length and a physical width of the first vehicle based on the length and the width of the first vehicle in the first image; Converting a first heading angle of the first vehicle relative to the aerial acquisition device into a second heading angle of the first vehicle relative to the ground acquisition device; The physical length, physical width, second heading angle, projection point coordinates of the first vehicle in the second image, and center point coordinates of the first vehicle in the second image are determined as the second annotation data.

6. The method according to claim 5, characterized in that Converting the first heading angle of the first vehicle relative to the aerial acquisition device into the second heading angle of the first vehicle relative to the ground acquisition device specifically includes: The long side of the quadrilateral frame is determined according to the coordinates of the projection point of the first vehicle in the second image, and the orientation of the long side is used as the second heading angle of the first vehicle relative to the ground acquisition device.

7. The method according to claim 3, characterized in that The first spatial mapping relationship is constructed as follows: Selecting an image from the image sequence, and selecting a plurality of first feature points from the selected image; Selecting second feature points from the orthographic projection image that correspond to the same physical position as the first feature points; Obtaining a mapping matrix according to the first feature point and the second feature point; According to the mapping matrix, pixels in the selected image are converted into the orthographic projection image to obtain the first spatial mapping relationship.

8. The method according to claim 1, characterized in that After acquiring the image sequence, the method further includes: The image sequence is subjected to de-jitter processing, wherein the de-jitter processing comprises: taking a starting image in the image sequence as a reference, aligning each image subsequent to the starting image to the starting image, so as to obtain a processed image sequence.

9. A method for training a vehicle labeling model, characterized in that: include: Acquire first training data including a plurality of first training samples, wherein the first training samples at least include: a second image of a second vehicle acquired by a ground acquisition device, and second annotation data of the second vehicle in the second image; wherein the second annotation data is acquired by the method according to any one of claims 1 to 8; The vehicle labeling model is trained based on the first training data, wherein the training includes: taking the second image as the input of the vehicle labeling model, obtaining third labeling data of the second vehicle predicted by the vehicle labeling model, and updating the model parameters of the vehicle labeling model using the loss function value corresponding to the first training objective, wherein the first training objective includes: minimizing the difference between the third labeling data output by the vehicle labeling model and the corresponding second labeling data.

10. The method according to claim 9, characterized in that The vehicle annotation model includes: a feature extraction network, a bounding box branch network, a classification branch network, a vehicle length and width branch network, a vehicle center point branch network, a projection point branch network and a heading angle branch network, wherein: The feature extraction network is used to extract feature representation of the second image; The bounding box branch network is used to predict a bounding box of the second vehicle in the second image based on a feature representation of the second image; The classification branch network is used to predict the category of the second vehicle in the second image based on the feature representation of the second image; The vehicle length and width branch network is used to predict the physical length and physical width of the second vehicle based on the feature representation of the second image; The vehicle center point branch network is used to predict the center point coordinates of the second vehicle in the second image based on the feature representation of the second image; The vehicle projection point branch network is used to predict the projection point coordinates of the second vehicle in the second image based on the feature representation of the second image; The heading angle branch network is used to predict a second heading angle of the second vehicle relative to the ground acquisition device based on a feature representation of the second image.

11. The method according to claim 10, characterized in that The vehicle annotation model also includes a target deep feature branch network, and the deep feature branch network is used to predict the deep features of the second vehicle based on the feature representation of the second image.

12. A vehicle trajectory fusion method, characterized in that: include: Acquire multiple images within the target area based on multiple ground acquisition devices; The image includes a vehicle; Inputting the multiple images into a vehicle annotation model respectively, and obtaining vehicle annotation information corresponding to each image obtained by the vehicle annotation model according to each image; Vehicle track fusion is performed according to the vehicle labeling information corresponding to each image.

13. The method according to claim 12, characterized in that According to the vehicle annotation information corresponding to each image, vehicle trajectory fusion is performed, specifically including: Projecting the vehicle annotation information corresponding to each image into a reference coordinate system to obtain a vehicle projection frame corresponding to each image; The vehicles corresponding to the multiple vehicle projection frames whose overlap degree is greater than a preset threshold are determined as the same target vehicle; The motion track of the target vehicle is determined according to the comprehensive ground projection frames of the same target vehicle at different time points.

14. The method according to claim 13, characterized in that The vehicles corresponding to the multiple vehicle projection frames whose overlap degree is greater than a preset threshold are determined as the same target vehicle, specifically including: Determine the weight of each vehicle projection frame; wherein the weight is determined based on the distance between the vehicle corresponding to the vehicle projection frame and each of the ground acquisition devices; Performing weighted fusion on the vehicle projection frames corresponding to each of the images to obtain a fused comprehensive vehicle projection frame; The vehicles corresponding to the multiple integrated vehicle projection frames whose overlap degree is greater than a preset first threshold value are determined to be the same target vehicle.

15. The method according to claim 13, characterized in that The reference coordinate system is a pre-obtained orthographic projection of the target area.

16. The method according to claim 13, characterized in that The vehicle annotation information includes: the physical length and width of the vehicle, the heading angle of the vehicle from the perspective of the ground acquisition device, and the projection point coordinates and center point coordinates of the vehicle in the image.

17. The method according to claim 16, characterized in that The vehicle annotation information may also include: depth feature information of the vehicle; According to the vehicle annotation information corresponding to each image, vehicle trajectory fusion is performed, specifically including: Determining the cosine similarity between the vehicles corresponding to each image according to the depth feature information of the vehicle corresponding to each image; Determine the vehicles whose cosine similarity is greater than a preset second threshold as the same target vehicle; The motion track of the target vehicle is determined according to the comprehensive ground projection frames of the same target vehicle at different time points.

18. A device for acquiring vehicle labeling data, characterized in that: include: A first image acquisition unit is configured to acquire an image sequence acquired by an aerial acquisition device for a target area, wherein the image sequence includes a first image of a first vehicle; A second image acquisition unit is configured to acquire a second image acquired by a ground acquisition device for the target area, wherein the second image includes a second vehicle; wherein the first image and the second image are time synchronized; an orthographic projection image generating unit, configured to construct a three-dimensional map of the target area according to the image sequence, and generate an orthographic projection image of the target area according to the three-dimensional map; a spatial mapping relationship determining unit, configured to determine a spatial mapping relationship between the orthographic projection image, the first image, and the second image; A first annotation data extraction unit is configured to extract first annotation data of the first vehicle in the first image; The second annotation data determining unit is configured to determine second annotation data of the second vehicle in the second image according to the first annotation data when the first vehicle and the second vehicle are identified as the same vehicle according to the spatial mapping relationship.

19. A training device for a vehicle labeling model, characterized in that: include: A sample acquisition unit is configured to acquire first training data including a plurality of first training samples, wherein the first training samples at least include: a second image of a second vehicle acquired by a ground acquisition device, and second annotation data of the second vehicle in the second image; wherein the second annotation data is acquired by the method according to any one of claims 1 to 8; A training unit is configured to train a vehicle labeling model based on the first training data, wherein the training includes: taking the second image as the input of the vehicle labeling model, obtaining third labeling data of the second vehicle predicted by the vehicle labeling model, and updating the model parameters of the vehicle labeling model using the loss function value corresponding to the first training objective, wherein the first training objective includes: minimizing the difference between the third labeling data output by the vehicle labeling model and the corresponding second labeling data.

20. A vehicle trajectory fusion device, characterized in that: include: An image acquisition unit is configured to acquire a plurality of images in a target area based on a plurality of ground acquisition devices; The image includes a vehicle; a labeling information acquisition unit, configured to input the plurality of images into a vehicle labeling model respectively, and acquire vehicle labeling information corresponding to each image obtained by the vehicle labeling model according to each image; The track fusion unit is configured to perform vehicle track fusion according to the vehicle labeling information corresponding to each image.

21. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 17 are implemented.

22. An electronic device, characterized in that: include: one or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of claims 1 to 17.

23. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 17 are implemented.

Images