Multi-view feature fusion road vehicle detection method and system

Abstract

The invention discloses a road vehicle detection method and system based on multi-view feature fusion, and relates to the technical field of automatic driving scene vehicle detection, and the method comprises the steps: obtaining image collection data of all view angles of a driving vehicle, carrying out the synchronous arrangement of all image collection data through timestamp information, and obtaining a plurality of synchronous image data sets; labeling, screening and analyzing each synchronous image data group, and taking the synchronous image data group meeting the screening requirement as a standard data set; a reference image is calibrated from the images, other images are clipped and transformed and then spliced with the reference image, the clipping transformation means that the other images are subjected to scale transformation and then are clipped with an overlapped area of the reference image, and a panoramic data set is obtained; and performing road vehicle target detection by using the panoramic data set and based on a convolutional neural network model. According to the detection method and system, multi-view-angle image data is subjected to synchronous arrangement and then annotation quality screening, it is ensured that the to-be-fused image data has reliable basic quality, the image data of the same group are registered in a clipping transformation mode, the influence caused by parallax of images of different view angles is greatly reduced, and the image fusion accuracy is improved. Therefore, the fused panoramic image can be more beneficial to learning and detection of the convolutional neural network model.

Description

Technical Field

[0001] The present invention relates to the technical field of vehicle detection in autonomous driving scenarios, and in particular to a road vehicle detection method and system with multi-view feature fusion. Background Art

[0002] In the automatic detection of road vehicles, a single camera is generally used to obtain image data from an independent perspective for convolutional neural network model training and detection. In this case, a single perspective is more susceptible to occlusion or limited perspective, resulting in the inability to accurately detect obstacles around the vehicle. However, most vehicles currently install multiple cameras for detection. By integrating multi-perspective image data for vehicle target detection, it is possible to analyze and judge the surrounding obstacles from a relative panoramic perspective, thereby obtaining more accurate and comprehensive analysis and detection results.

[0003] Although there is a method in the prior art that uses image blocks formed by carefully stitching multi-view data and then inputs them into a convolutional neural network for learning or detection, it ignores the time difference problem of different viewpoints when stitching image data from different viewpoints, resulting in a decrease in the accuracy of feature matching, thereby affecting the quality of the fused image and the accuracy of the information. Summary of the invention

[0004] The purpose of the present invention is to provide a road vehicle detection method and system with multi-view feature fusion. The detection method and system ensure that the image data to be fused has reliable basic quality by synchronously sorting the image data from multiple perspectives and then performing annotation quality screening. The image data in the same group are then aligned by means of cropping transformation, which greatly reduces the impact of parallax of images from different perspectives, thereby making the fused panoramic image more conducive to learning and detection of the convolutional neural network model.

[0005] The embodiment of the present invention is achieved as follows: In a first aspect, a road vehicle detection method with multi-view feature fusion includes the following steps: acquiring image acquisition data of all viewing angles of a driving vehicle, synchronizing and arranging all the image acquisition data using timestamp information to obtain multiple groups of synchronized image data groups; performing annotation screening analysis on each group of the synchronized image data groups, and using the synchronized image data groups that meet the screening requirements as standard data sets, wherein the annotation screening analysis refers to performing a difference comparison on the initial annotation information of each sub-object in the synchronized image data set; performing feature fusion on each image in the standard data set to obtain a panoramic data set, wherein the feature fusion on each image in the standard data set includes: calibrating a reference image from each image, cropping and transforming the remaining images and splicing them with the reference image, wherein the cropping transformation refers to performing a scale transformation on the remaining images and then cropping the overlapping area with the reference image; and performing road vehicle target detection using the panoramic data set and based on a convolutional neural network model.

[0006] In some optional embodiments, the calibrating the reference image from each image includes the following steps: detecting the detection target attributes in each image in the standard data set, the detection target attributes including the type and quantity of the detection target; comparing all images pairwise to obtain the overlap comparison results of the detection target attributes in each image; and selecting one of the images as the reference image based on the overlap comparison results.

[0007] In some optional embodiments, the detection target attributes in each image in the standard data set are determined based on the initial annotation information of each image; all images are compared pairwise, including: determining a target set in each image, the target set being composed of detection targets with different coordinates; and performing pairwise effective overlapping comparisons on the target sets of each image, wherein the effective overlapping comparisons are determined based on the similarity of the coordinates of typical feature points of the detection targets.

[0008] In some optional embodiments, the cropping and transforming of the remaining images and then splicing them with the reference image includes the following steps: determining at least one detection target corresponding to the remaining images and the reference image, and determining a scale transformation value based on a ratio of neighbor distances between corresponding typical feature point coordinates of the detection targets of the two; performing scale transformation processing on the remaining images based on the scale transformation value to obtain a transformed image; determining an overlapping area of ​​the transformed image relative to the reference image, cropping the overlapping area of ​​the transformed image to obtain a cropped image; and splicing the cropped image with the reference image.

[0009] In some optional implementations, determining the overlapping area of ​​the transformed image relative to the reference image and cropping the overlapping area of ​​the transformed image specifically includes the following steps: aligning the transformed image with the reference image and determining the alignment edges of the two respectively; delineating the alignment edges of the transformed image based on the alignment edges of the reference image to obtain the stitching edges of the transformed image; cropping the transformed image according to the stitching edges; wherein, when aligning the transformed image with the reference image, a multi-level alignment method is used for alignment.

[0010] In some optional embodiments, the one-time multi-level registration method includes the following steps: obtaining image data of the transformed image and the reference image under a first pixel condition, and aligning them according to the image data under the first pixel condition; obtaining image data of the transformed image and the reference image under a second pixel condition, and aligning them according to the image data under the second pixel condition; the second pixel is larger than the first pixel; wherein, when aligning under the first pixel condition, the boundary range outline of the edge to be aligned is calibrated, and when aligning under the second pixel condition, key pixel points are selected for the boundary range outline.

[0011] In some optional embodiments, stitching all the cropped images with the reference image comprises the following steps: determining a plurality of stitching sides of the reference image; performing preliminary stitching using aligned edges of different cropped images and corresponding stitching sides of the reference image to form a stitching seam; performing stitching detection on each stitching seam to determine a stitching matching coefficient, and marking a stitching seam whose stitching matching coefficient is lower than a preset value as a pre-splicing seam; performing secondary multi-level registration processing on the corresponding aligned edges of the pre-splicing seam until the stitching matching coefficient of the pre-splicing seam formed by the corresponding aligned edges is not lower than the preset value; wherein the pixel condition in the secondary multi-level registration processing is higher than the pixel condition in the primary multi-level registration processing, and the corresponding aligned edge refers to the aligned edge of the transformed image and the reference image.

[0012] In some optional implementations, when performing stitching detection on each stitching seam, it is determined whether there is a pixel point of the detection target in the stitching seam, and if so, a correction coefficient is assigned to the stitching matching coefficient.

[0013] In some optional embodiments, determining the coordinates of typical feature points of the detection target includes the following steps: obtaining a key point coordinate group and a contour coordinate group of the detection target, judging at least one special pixel point from the key point coordinate group and the contour coordinate group, constructing a special feature area using associated special pixel points, and obtaining multiple feature area points, wherein the feature area points refer to coordinate points that can reflect the typical features of the target; and screening the feature area points based on the labeling rules to obtain typical feature point coordinates.

[0014] In a second aspect, a road vehicle detection system with multi-view feature fusion includes: A first acquisition unit is used to acquire image acquisition data from all viewing angles of the driving vehicle, and synchronize and sort all the image acquisition data using timestamp information to obtain multiple groups of synchronized image data groups; The first processing unit is used to perform annotation screening analysis on each group of the synchronous image data groups, and use the synchronous image data groups that meet the screening requirements as standard data sets, wherein the annotation screening analysis refers to performing difference comparison on the initial annotation information of each sub-object in the synchronous image data group.

[0015] A second processing unit is configured to perform feature fusion on each image in the standard data set to obtain a panoramic data set, wherein the feature fusion on each image in the standard data set includes: calibrating a reference image from each image, cropping and transforming the remaining images and then splicing them with the reference image, wherein the cropping transformation refers to performing a scale transformation on the remaining images and then cropping the area overlapping with the reference image; A first computing unit is used to perform road vehicle target detection based on a convolutional neural network model using the panoramic data set.

[0016] The beneficial effects of the embodiments of the present invention are: The road vehicle detection method and system with multi-view feature fusion provided by the embodiment of the present invention first synchronizes and organizes the image data of all viewing angles of the driving vehicle to obtain multiple synchronized image data groups at the same time, and then compares the differences in the annotation information of the sub-objects of each synchronized image data group, and eliminates the synchronized image data groups that do not meet the requirements, so that the synchronized image data groups that have been annotated, screened and analyzed have a high degree of consistency and similarity in the content and detection targets, thereby ensuring that the content and detection targets have high matching and high accuracy in the subsequent image fusion, and avoiding the fusion image from failing to guarantee the basic illustration quality. Then, feature fusion is performed on each image in the standard data set obtained by the annotated, screened and analyzed. When fusion is performed, the reference image is first determined, and then the images of the remaining images are initially transformed according to the image scale of the reference image, and then the overlapping areas are cropped. Finally, the cropped remaining images are spliced ​​and fused with the reference image, thereby reducing the effects of low splicing quality and unclear and inaccurate fusion display of detection targets caused by parallax problems of each viewing angle.

[0017] In general, the multi-view feature fusion road vehicle detection method and system provided in the embodiments of the present invention will perform baseline image calibration on each image in the preliminarily screened standard data set and carry out cropping transformation based on it, which not only ensures the basic matching accuracy of each image, but also takes into account the stitching and fusion incompatibility problem caused by the parallax of each viewing angle, and ensures that the final panoramic data set has high content and detection target accuracy, matching and integrity, thereby ensuring the accuracy of the final convolutional neural network model in vehicle target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 A flow chart of the main steps of the detection method provided by an embodiment of the present invention; Figure 2 for Figure 1 A flow chart of one of the main steps S300 is shown; Figure 3 for Figure 2 A flowchart of subsequent steps of step S300 is shown; Figure 4 for Figure 3 The flowchart of sub-step S360 following step S300 is shown; Figure 5 for Figure 3 The flowchart of sub-step S370 subsequent to step S300 is shown; Figure 6 for Figure 2 The flowchart of one of the sub-steps S320 of step S300 is shown; Figure 7 A modular schematic diagram of a detection system provided in an embodiment of the present invention.

[0020] Icon: 500 - detection system; 510 - first acquisition unit; 520 - first processing unit; 530 - second processing unit; 540 - first calculation unit. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0022] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] It should be understood that the "system", "device" and / or "module" used in the present invention is a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the words can be replaced by other expressions.

[0024] As shown in the present invention and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular, but also include the plural. In general, the terms "include" and "comprise" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list, and the method or device may also include other steps or elements.

[0025] Flowcharts are used in the present invention to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, the various steps may be processed in reverse order or simultaneously. At the same time, other operations may also be added to these processes, or one or more operations may be removed from these processes. Example

[0026] When driving in traffic, autonomous vehicles need to identify various targets in the surrounding environment at any time in order to accurately identify and plan driving routes and driving operations. However, in some driving scenarios, for example, because the distance to the target is too short, it is difficult for a camera with a certain viewing angle to fully capture the target, which can easily lead to misjudgment in subsequent recognition. To address this issue, the method of using on-board cameras at different points for joint shooting and detection can effectively reduce the possibility of misjudgment. That is, using multiple cameras around the vehicle body to capture images of the same target, the target feature cascade and panoramic construction are performed, so as to detect and identify the full picture of the target as far as possible, so as to achieve the purpose of accurate judgment. In the process of implementing the above-mentioned solution, although the shooting completeness of close-range targets can be improved, and the category and approximate position of targets can be further determined, the precise position and accurate size of targets cannot be determined more accurately. The reason is that in the process of restoring targets using multiple perspectives, we directly use image data from different perspectives for feature fusion, select and process the overlapping parts, and then construct a panoramic model of the target. This method ignores the parallax problem of the target captured by images from different perspectives, that is, there is no compensation for the influence of this parallax through modes such as geometric transformation, resulting in the size difference between the final restored target model and the actual target, as well as the actual position difference. In response to the above-mentioned problem, we have proposed a solution that can eliminate the parallax problem when fusion of multi-perspective image data. We use a set of images that have been synchronously sorted and meet the fusion basis, and then use geometric transformation to eliminate the parallax effect and then perform image fusion to achieve the purpose of accurately judging and detecting targets.

[0027] Please refer to Figure 1 The present embodiment provides a road vehicle detection method using multi-view feature fusion, comprising the following steps: S100: Acquire the image acquisition data of all viewing angles of the driving vehicle, and synchronize and sort all the image acquisition data using the timestamp information to obtain multiple sets of synchronized image data sets; this step means acquiring images acquired by cameras installed at multiple positions on the driving vehicle, so as to obtain image acquisition data at different viewing angles. There is timestamp information on each image, and all the image acquisition data are synchronized and sorted using the same timestamp information to obtain multiple sets of synchronized image data sets. The synchronized image data sets obtained at this time refer to multi-view image data sets taken at the same time, so as to facilitate the determination of the target content of the driving vehicle's surrounding environment at the time of shooting from multiple viewing angles.

[0028] S200: Perform annotation screening analysis on each group of the synchronous image data set, and use the synchronous image data set that meets the screening requirements as the standard data set, wherein the annotation screening analysis refers to performing a difference comparison on the initial annotation information of each sub-object in the synchronous image data set; this step means performing an initial annotation on each sub-object in the obtained synchronous image data set, and then performing a difference comparison based on the initial annotation information, so as to screen out the synchronous image data set that meets the requirements as the standard data set. In this embodiment, the initial annotation refers to performing boundary and category annotation processing on different targets in the image, so as to obtain the annotation information of each sub-object in the synchronous image data set and perform annotation comparison to determine whether the image information after annotating each sub-object is highly correlated and similar, so as to judge whether the synchronous image data set is a qualified data set and can be used for subsequent model learning and training.

[0029] The difference comparison is, for example, to compare the boundary annotations of the same category of targets in each sub-object of the same synchronous image data set, obtain the annotation difference, and then use the synchronous image data set with the annotation difference not greater than the preset value as the standard data set. That is, the same category of targets are selected from each sub-object, and the boundary annotations of the same category of targets are compared to obtain the annotation difference after comparison. If the annotation difference obtained from the boundary annotations of the same category of targets selected from multiple perspectives is not greater than the preset value, the corresponding synchronous image data set is used as the standard data set, which can be used as a qualified data set and can be used for subsequent model learning and training. Furthermore, the boundary annotation includes key point annotation and contour annotation, and the feature point coordinate comparison table is constructed using the key point annotation information and the contour annotation information, that is, the boundary annotation uses the key point annotation information of the target and the contour annotation information of the target to compare, and then constructs the feature point coordinate comparison table according to the comparison situation. The feature point coordinate comparison table, for example, the vertical heading indicates the object situation of the corresponding target in each sub-object, the horizontal heading indicates the coordinate point type situation, and the rest is filled with the coordinate point data (pixel point or pixel area position). Finally, the annotation difference is calculated based on the feature point coordinate comparison table, that is, the annotation difference is obtained according to the difference calculation result of each column of coordinate point data, so as to calculate the value of the annotation difference by using the feature point coordinate comparison table.

[0030] S300: Perform feature fusion on each image in the standard data set to obtain a panoramic data set, wherein the feature fusion on each image in the standard data set includes: calibrating a reference image from each image, cropping and transforming the remaining images and splicing them with the reference image, wherein the cropping transformation refers to performing a scale transformation on the remaining images and then cropping the overlapping area with the reference image; this step represents performing feature fusion on each image in the standard data set obtained above to construct a panoramic data set. In order to eliminate the influence of inaccurate feature fusion caused by the parallax problem between the images, a reference image is pre-calibrated from each image, and the reference image can have the basis of the splicing core of all images, so that the panoramic data can be restored more accurately, and the computing load can be reduced, thereby improving the computing efficiency of the panoramic data set. Specifically, after determining the reference image, each remaining image (referred to as the remaining images) is subjected to geometric transformation processing, that is, each remaining image is cropped and transformed, and the overlapping area (the area overlapping with the reference image) is cropped using the remaining images after the scale transformation, and finally the remaining images of the remaining images are spliced ​​and fused with the reference image, and finally the panoramic data set is obtained.

[0031] Through S300, the image fusion effect caused by parallax can be reduced by using the cropping transformation processing method under the premise of ensuring the consistency and integrity of the detection targets in each image, so as to obtain a panoramic data set with more accurate detection target size and more precise position, and then perform the subsequent model detection step S400: using the panoramic data set and based on the convolutional neural network model to detect road vehicle targets. This step means using the above-obtained panoramic data set as the training set and test set of input data to train and optimize the convolutional neural network model, so that the obtained convolutional neural network model can be more accurately applied to road vehicle target detection, and obtain more accurate and matching target detection results. In this embodiment, the convolutional neural network model can adopt a single-stage target detection algorithm. The single-stage target detection algorithm regards target detection as a regression problem and does not use a candidate region generation network. Compared with the two-stage target detection algorithm, the network model of the single-stage target detection algorithm has fewer parameters and calculations, and the detection speed is faster, which is more suitable for the rapid judgment and target detection of vehicle targets in this traffic driving scene.

[0032] Taking into account that the detection accuracy of the single-stage target detection algorithm is slightly inferior to that of the two-stage target detection algorithm, and the number of parameters of the algorithm model will cause the model to occupy a large storage space during the training process, and the large amount of calculation will cause the forward reasoning time of the algorithm model to be longer, making the algorithm overly dependent on computer hardware resources, making it difficult to use directly in hardware devices with limited storage resources and computing power. In order to solve the problem that the target detection algorithm occupies more system resources in vehicle driving scenarios, the convolutional neural network model in this embodiment can adopt the lightweight target detection optimization algorithm of YOLOv5s. This algorithm can reduce the number of parameters of the model to a certain extent and improve the detection speed while maintaining a certain detection accuracy, thereby solving the problem that target detection in vehicle driving scenarios occupies more system resources. Specifically, the last three BottleneckCSP1 modules in the backbone network of the YOLOv5s algorithm model are first replaced with the corresponding GhostBottleneck modules, and the CBH module before the SPP module is replaced with a depthwise separable convolution, and an ECA attention module is added after the SPP module; secondly, based on the PANet network structure of the neck network, the ASFF adaptive spatial feature module is integrated; thirdly, CIoU Loss is used as the boundary regression loss function of the YOLOv5s algorithm model.

[0033] Through the above technical solution, the image data of all viewing angles of the driving vehicle are obtained, and the timestamp is used for synchronization, which can ensure the consistency of data from different viewing angles in time and space. Then, by annotating, screening and analyzing the synchronized image data group, those data that do not meet the requirements can be excluded, thereby ensuring the accuracy of the subsequent detection results. In addition, in the process of combining image information from different viewing angles by using feature fusion technology, the image data fusion effect caused by the parallax problem of each image is greatly reduced by using the clipping transformation processing method, forming a more accurate panoramic data set. Among them, the selection of the reference image is a crucial step, which can provide a good foundation for the subsequent image stitching, so that in the process of feature fusion, i.e. image stitching and fusion, the inconsistency of the target object displayed at different viewing angles due to the parallax problem can be effectively reduced, so that the final panoramic data set contains information from all viewing angles, and the target object can now be presented in a clearer and more accurate way, which not only solves the problems existing in traditional single-view image detection, but also solves the problem of inaccurate feature fusion in multi-view image stitching.

[0034] Considering that the selection of the reference image is very important, it can usually be determined based on factors such as image quality, viewing angle coverage, target coverage type, etc., so it is necessary to select an image with high clarity, wide coverage of targets and appropriate viewing angle as the reference. For details, please refer to Figure 2, the step of calibrating the reference image from each image comprises the following steps: S310: Detect the detection target attributes in each image in the standard data set, and the detection target attributes include the type and quantity of the detection target; this step indicates that the target detection task needs to be performed on each image in the standard data set, that is, the detection target attributes of each image are determined, and the types and quantities of these detection targets are recorded. For example, in a traffic monitoring scenario, the detection target may be a car, two bicycles, three pedestrians, etc. All detection targets of different types or the same type are detected and confirmed to facilitate subsequent one-by-one comparison.

[0035] S320: Compare all images in pairs to obtain the overlap comparison results of the detection target attributes in each image; this step means comparing each pair of images separately and calculating the overlap of the detection target attributes between them. For example, a car, a bicycle, two pedestrians, etc. overlap. It should be noted that in different implementations, the overlap can be defined in a variety of ways, such as the number and type ratio of the same type of detection targets in two images, or the ratio of the intersection area to the union area of ​​the target positions (such as feature matching, interest point detection, etc.), so as to help determine which detection targets are different perspectives of the same target object in different images, so as to obtain the overlap calculation results.

[0036] S330: Select one of the images as the reference image according to the overlap comparison result. This step represents the overlap comparison result that can be obtained according to the above steps. For example, an image with the highest comprehensive overlap or that meets certain specific conditions (such as containing the most target types, the largest number of targets, the widest viewing angle, etc.) is selected as the reference image. This embodiment uses the highest comprehensive overlap as the selection basis, so as to adapt to the effect of the final image stitching and the accuracy requirements of target detection.

[0037] Through the above technical solution, the basis for selecting the reference image is ensured. On this basis, considering that the detection targets in each image in the standard data set are not exactly the same, but highly similar and have a certain size change caused by the difference in viewing angle, it is possible to determine whether there is overlap of the detection targets based on the consistency of the detection target pixel interest points when comparing the images one by one. Specifically, in step S310, the detection target attributes in each image in the standard data set are determined based on the initial annotation information of each image, that is, each image and its corresponding initial annotation information are loaded from the standard data set. These initial annotation information (manual annotation or machine annotation) generally include the location of the target (such as bounding box coordinates), category labels (such as cars, pedestrians, etc.) and possible quantity information. When determining the detection target attributes, the type (such as vehicles, pedestrians, etc.) and quantity of the detection target in each image are extracted (in different implementations, attribute annotation, relationship annotation, quality annotation and other information can also be extracted).

[0038] Based on the above initial annotation information, all images are compared pairwise, including: determining the target set in each image, which is composed of detection targets of different coordinates; that is, extracting the detection targets at each position in each image, and the detection target at a certain position may belong to the same type as the detection targets at other positions, and it is also necessary to extract the detection targets of the same type, in order to perform subsequent comparisons on all targets at different positions. The target set in each image (including detection targets at all positions) can be obtained through the above technical solution. When performing pairwise comparisons of images, the target sets of the two images are compared pairwise for effective overlap. The effective overlap comparison is determined based on the similarity of the typical feature point coordinates of the detection target, that is, the effective overlap comparison is obtained based on the comparison of the typical feature point coordinates contained in the detection target, and the corresponding typical feature point coordinates are judged for similarity, thereby obtaining a judgment result of whether the two detection targets meet the overlap requirements.

[0039] Determining based on the similarity of typical feature point coordinates of the detection target First, it is necessary to determine the typical feature point coordinates of the detection target. In some embodiments, see Figure 6 , determining the coordinates of typical feature points of the detection target includes the following steps: S321: Obtain the key point coordinate group and contour coordinate group of the detection target; this step means obtaining the coordinate information of these key points and the contour points that make up all the key point information and contour information of each detection target in each image, thereby integrating these coordinate information into a key point coordinate group (coordinates of multiple key points) and a contour coordinate group (coordinates of multiple contour points), so that representative typical feature point coordinates can be determined in these coordinates later.

[0040] S322: Determine at least one special pixel point from the key point coordinate group and the contour coordinate group, construct a special feature area using the associated special pixel points, and obtain multiple feature area points, wherein the feature area point refers to a coordinate point that can reflect the typical features of the target; this step indicates determining at least one pixel point that can characterize the features or texture of the detection target from the key point coordinate group and the contour coordinate group that have been determined, and the judgment rule for the pixel point is determined in advance, and different judgment plans can be determined according to different target types or attributes. The special pixel points selected using the judgment rule can at least be combined to show the characteristics and typical texture of the detection target. The above-mentioned special pixel points are associated and combined to form pixel point areas, namely special feature areas, which are recorded as feature area points of the detection target, which can reflect the typical features of the detection target.

[0041] S323: Screen the feature area points based on the labeling rules to obtain the coordinates of typical feature points. This step means screening all the feature area points obtained above under the determined labeling rules, and screening the remaining feature area points as the typical feature point coordinates. It should be noted that the labeling rules can be determined according to different target types or attributes, and may include labeling positions, labeling methods, labeling quantities, and labeling restrictions. The labeling rules in the screening of feature area points based on the labeling rules here are more about clarifying the number of labelings, that is, determining how many adapted feature area points should be obtained.

[0042] On the basis of the above technical solution, since the judgment rules of special pixels also need to be determined in advance, the content of the annotation rules can also be followed. The annotation rules here are more about clarifying the annotation position, annotation method and annotation restriction content. Specifically, judging at least one special pixel from the key point coordinate group and the contour coordinate group specifically includes the following steps: S322a: Obtain the target's annotation type, the annotation type including at least one of occlusion annotation, small target annotation and living object annotation. This step indicates that the target category is determined in advance so that the annotation method and annotation restriction content can be determined. In this embodiment, three target annotation types are shown, namely occlusion annotation, small target annotation and living object annotation. In actual situations, in addition to targets of other categories, a composite annotation method of the above three annotation types is also included, that is, the three categories appear in pairs in the image data, such as occlusion of small targets, occlusion of living objects, etc.

[0043] S322b: Delimit the annotation range according to the annotation type, and select at least part of the pixel points of the key point coordinate group and the contour coordinate group from the defined annotation range as the special pixel points. This step means that the annotation position is determined by the determined annotation type, that is, the annotation ranges of different categories of objects are defined, and then special pixel points are selected from these annotation ranges, and the special pixel points come from at least part of all the pixel points of the key point coordinate group and the contour coordinate group.

[0044] Through the above technical solution, special pixel points are selected using the annotation rules to construct special feature areas (feature area points), and the feature area points are further screened based on the annotation rules to finally obtain the coordinates of typical feature points, which can characterize the typical information of the detection target, and then use the similarity of these typical feature point coordinates to judge the overlap of the corresponding detection targets in the two images, that is, to judge the effective overlap of the target sets of the two images, so as to achieve the purpose of obtaining the overlap comparison result of the two images. The similarity can be obtained by calculating the neighbor distance ratio of the typical feature point coordinates.

[0045] After determining the reference image, each of the remaining images in the standard data set needs to be spliced ​​with the reference image by feature fusion. As shown above, in order to reduce the influence of feature scale changes caused by different viewing angles during splicing, a cropping transformation is used in this embodiment. For details, please refer to Figure 3 , the step of cropping and transforming the remaining images and then splicing them with the reference image comprises the following steps: S340: Determine at least one detection target corresponding to the remaining images and the reference image, and determine the scale transformation value according to the neighbor distance ratio between the corresponding typical feature point coordinates of the detection target in the two images; this step indicates that when each remaining image is spliced ​​with the reference image, the remaining image needs to be scaled first, and the specific ratio and method of the scale transformation are determined according to the corresponding detection targets in the two (the remaining images and the reference image), which can be one corresponding detection target or multiple corresponding detection targets, wherein the corresponding detection target refers to the detection target at the corresponding position in the two images. The scale transformation value is determined by the neighbor distance ratio between the typical feature point coordinates of at least one corresponding detection target, that is, the typical feature point coordinates (which can also be one or more) at the same position in the corresponding detection target are found, and the scale ratio difference between the two images is determined by the neighbor distance ratio between the typical feature point coordinates in the own image and the remaining typical feature point coordinates, and the scale ratio difference is used as the scale transformation value.

[0046] S350: scale transformation is performed on the remaining images according to the scale transformation value to obtain a transformed image; this step means that the scale transformation value is used to perform scale transformation on the remaining images. Transformation generally refers to scale ratio adjustment, and in other implementations, it may also include rotation angle adjustment, pixel point coordinate translation, etc. The scale transformation value is determined by geometric transformation of typical feature point coordinates (such as affine transformation or homography transformation), and the remaining images are scale transformed to obtain a transformed image, so that the changed image can be used as an object for comparison with the reference image after parallax processing.

[0047] S360: Determine the overlapping area of ​​the transformed image relative to the reference image, crop the overlapping area of ​​the transformed image, and obtain a cropped image; this step means comparing the transformed image with the reference image in pixel points to determine the overlapping area between the two (the area where most of the pixels overlap), and then crop the determined overlapping area of ​​the transformed image to remove the portion that basically overlaps with the reference image to obtain a cropped image.

[0048] S370: stitching the cropped image with the reference image; this step means that the cropped image is directly stitched with the reference image. Since the overlapping area is determined on the cropped image, after the overlapping area is cropped, it can be stitched with the reference image to obtain an image with a more panoramic perspective. Therefore, the cropped image of each of the remaining images obtained through the above steps is stitched with the reference image, and finally a panoramic image can be obtained by stitching the image acquisition data obtained from all perspectives, thereby achieving the purpose of constructing a panoramic data set.

[0049] Through the above technical solution, the consistency of scale can be ensured when the remaining images are spliced ​​with the reference image, and the deformation of the detected target after splicing can be reduced. However, on this basis, considering that the accuracy of cropping the overlapping area directly affects the display accuracy of the detected target at the splicing point, it also affects the expression accuracy of specific pixels (which may be the pixels included in the coordinates of typical feature points), it is necessary to consider the processing of the cropped area during cropping. For details, please refer to Figure 4 The determining of the overlapping area of ​​the transformed image relative to the reference image and clipping the overlapping area of ​​the transformed image specifically comprises the following steps: S361: Align the transformed image with the reference image, and respectively determine the alignment edges of the two; this step means first aligning the transformed image with the reference image, for example, performing image registration, to determine the alignment of the largest part of pixel coordinates or the alignment of the most typical feature points, so as to determine the alignment edges of the two after alignment, that is, the boundaries of the overlapping areas.

[0050] S362: based on the alignment edge of the reference image, the alignment edge of the transformation image is delineated to obtain the splicing edge of the transformation image; this step means that after determining the alignment edges of the two, the situation of one of them is delineated based on the situation of the other, that is, the alignment edge of the transformation image is delineated based on the alignment edge of the reference image, so that the alignment edge of the reference image can be fully retained, and the alignment edge of the transformation image is adjusted and delineated according to the retained alignment edge to avoid the subsequent excessive overlap of splicing or excessive splicing gap. For example, the corresponding pixel points on the alignment edge of the transformation image are marked with reference to the relationship and distance of all pixel points of the alignment edge of the reference image, so as to connect the marked pixel points to obtain the delineated alignment edge, which is recorded as the splicing edge of the transformation image.

[0051] S363: trimming the transformed image according to the stitching edge; this step means trimming the transformed image along the stitching edge of the transformed image, thereby ensuring that the trimmed image has a high degree of stitching fit with the reference image.

[0052] On the basis of the above-mentioned cropping of the overlapping areas of the transformed images, since the accuracy of the aligned edges directly affects the accuracy of the stitched edges, it is necessary to ensure the alignment accuracy of the aligned edges during alignment, that is, in this embodiment, a multi-level alignment method is used when aligning the transformed image with the reference image. The multi-level alignment method refers to a method of aligning non-primary pixels, but a method of aligning under multiple pixel conditions, thereby avoiding the problem of pixel omissions due to low alignment accuracy or too high alignment accuracy caused by the pixel conditions being too low at the beginning.

[0053] Specifically in this embodiment, the one-time multi-level registration method includes the following steps: obtaining image data of the transformed image and the reference image under a first pixel condition, and aligning them according to the image data under the first pixel condition; that is, aligning the transformed image with the reference image under the first pixel condition. Then obtaining image data of the transformed image and the reference image under a second pixel condition, and aligning them according to the image data under the second pixel condition; that is, aligning the transformed image with the reference image under the second pixel condition, wherein the second pixel is larger than the first pixel, that is, the transformed image and the reference image are clearer under the second pixel condition.

[0054] Among them, when aligning under the first pixel condition, the boundary range contour of the edge to be aligned is calibrated, that is, the range contour used to determine the aligned edge of the transformed image and the reference image under the first pixel condition. Since the first pixel is lower, the contour determined at this time is rougher and larger, and it is not easy to have the problem of missing pixels; and when aligning under the second pixel condition, key pixel points are selected for the boundary range contour, which means that the second pixel is higher. At this time, more critical pixels are selected within the above-mentioned boundary range contour to characterize the typical pixel point positions of the aligned edges of the transformed image and the reference image, thereby avoiding the situation where most of the overlap or the connection does not correspond.

[0055] Through the above technical solution, a multi-level registration method is used to determine the alignment edges of the transformed image and the reference image, so that the obtained stitching edges can better fit the stitching of the reference image, so that the final stitched panoramic image is clearer and more accurate at the stitching. On this basis, considering that the use of a multi-level registration method ensures the basic comprehensiveness and accuracy of the alignment edges, in order to further improve the clarity and accuracy of the stitching, please refer to the details. Figure 5 The step of stitching all the cropped images with the reference image comprises the following steps: S371: Determine multiple stitching sides of the reference image; this step indicates determining multiple stitching sides on the reference image for stitching with different cropped images, and the stitching sides are formed by the alignment edges obtained each time the image is aligned with the transformed image. Then proceed to step S372: Use the alignment edges of different cropped images and the corresponding stitching sides of the reference image to perform preliminary stitching and form a stitching seam; this step indicates stitching the different cropped images formed above directly with the corresponding stitching sides of the reference image, and forming a stitching seam at the stitching. It can be seen from the above technical solution that since the stitching seam uses a multi-level registration method to align the edges, the stitching edges obtained have a certain stitching fit, but the stitching fit is only the basic stitching form and parameters, and it is necessary to determine whether the stitching meets the accuracy requirements of feature expression, so a detection step is required.

[0056] S373: Perform stitching detection on each stitching seam, determine the stitching matching coefficient, and mark the stitching seam whose stitching matching coefficient is lower than the preset value as a pre-splicing seam; this step means performing stitching detection on each stitching seam formed as described above, that is, detecting whether the pixel accuracy is sufficient to accurately and clearly express the display characteristics of the stitching location, and the judgment is performed based on the stitching matching coefficient, which is determined by the ratio of the pixel points that meet the requirements to the total pixel points. The stitching seam represented by the stitching matching coefficient that does not meet the preset value (set according to the stitching accuracy requirement) is marked as a pre-splicing seam, and the pre-splicing seam needs to be further processed, that is, the next round of multi-level registration processing is performed.

[0057] S374: Perform secondary multi-level registration processing on the corresponding aligned edge of the pre-joined seam until the stitching matching coefficient of the pre-joined seam formed by the corresponding aligned edge is not less than the preset value, and the corresponding aligned edge refers to the aligned edge of the transformed image and the reference image; wherein the pixel condition in the secondary multi-level registration processing is higher than the pixel condition in the primary multi-level registration processing, which means that the secondary multi-level registration processing is the next high-precision processing mode of the primary multi-level registration processing, and its execution action is consistent with the execution action of the primary multi-level registration processing, except that under the pixel condition, the first pixel in the secondary multi-level registration processing is higher than the second pixel in the primary multi-level registration processing, so that each next multi-level registration processing can obtain a higher precision result. It should be noted that the secondary multi-level registration processing is based on the primary multi-level registration processing, and the pixel points delineated by the most initial (lowest pixel) can be retained to avoid feature omission. If the subsequent stitching matching coefficient does not meet the preset value, a higher precision multi-level registration processing is performed.

[0058] Through the above technical solution, the alignment edge of the transformed image and the reference image is obtained by continuously improving the pixel conditions, so that the subsequently obtained splicing edges and corresponding splicing seams have the characteristics of becoming more and more accurate and matching, thereby strengthening the expression of the pixel points at the splicing. On this basis, if the pixel points at the splicing are the pixel points of the detection target, especially the pixel points covered by the coordinates of the typical feature points, it is necessary to retain these pixel points as much as possible without removing them due to overlap, that is, to retain as many pixel points as possible involved in the detection target, so as to improve the accuracy of subsequent model judgment. Therefore, when performing splicing detection on each splicing seam, it is determined whether there are pixel points of the detection target in the splicing seam. If so, a correction coefficient is assigned to the splicing matching coefficient. The correction coefficient can increase the value of the splicing matching coefficient, so that it is easier to meet the preset value, and no secondary multi-level registration processing or even tertiary multi-level registration processing is required, which retains the integrity of the detection target pixel points, thereby being more conducive to the subsequent recognition and judgment of the detection target.

[0059] This embodiment also provides a road vehicle detection system 500 with multi-view feature fusion, see Figure 7The modular schematic diagram of the road vehicle detection system 500 with multi-view feature fusion is mainly used to divide the functional modules of the road vehicle detection system 500 with multi-view feature fusion according to the embodiment of the above method. For example, each functional module can be divided, or two or more functions can be integrated into one processing module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the present invention is schematic and is only a logical functional division. There may be other division methods in actual implementation. For example, in the case of dividing each functional module according to each function, Figure 7 What is shown is only a schematic diagram of a system / device, wherein the road vehicle detection system 500 with multi-view feature fusion may include a first acquisition unit 510, a first processing unit 520, a second processing unit 530 and a first calculation unit 540. The functions of each unit module are described below.

[0060] A first acquisition unit 510 is used to acquire image acquisition data from all viewing angles of the driving vehicle, and synchronize and sort all the image acquisition data using timestamp information to obtain multiple groups of synchronized image data groups; A first processing unit 520 is configured to perform annotation screening analysis on each of the synchronous image data sets, and use the synchronous image data sets that meet the screening requirements as standard data sets, wherein the annotation screening analysis refers to performing a difference comparison on the initial annotation information of each sub-object in the synchronous image data set; The second processing unit 530 is used to perform feature fusion on each image in the standard data set to obtain a panoramic data set, wherein the feature fusion on each image in the standard data set includes: calibrating a reference image from each image, cropping and transforming the remaining images and splicing them with the reference image, wherein the cropping transformation refers to performing a scale transformation on the remaining images and then cropping the overlapping area with the reference image; in some embodiments, the second processing unit 530 is also used to detect the detection target attributes in each image in the standard data set, wherein the detection target attributes include the type and quantity of the detection target; compare all images pairwise to obtain the overlap comparison result of the detection target attributes in each image; select one of the images as the reference image according to the overlap comparison result. It is also used to determine the target set in each image, wherein the target set is composed of detection targets with different coordinates; perform pairwise effective overlap comparison on the target sets of each image, wherein the effective overlap comparison is determined based on the similarity of the coordinates of the typical feature points of the detection target. And used to determine at least one detection target corresponding to the remaining images and the reference image, determine the scale transformation value according to the neighbor distance ratio between the corresponding typical feature point coordinates of the detection target of the two; scale the remaining images according to the scale transformation value to obtain a transformed image; determine the overlapping area of ​​the transformed image relative to the reference image, cut the overlapping area of ​​the transformed image, and obtain a cut image; splice the cut image with the reference image. And also used to align the transformed image with the reference image, respectively determine the alignment edges of the two; delineate the alignment edge of the transformed image based on the alignment edge of the reference image to obtain the splicing edge of the transformed image; cut the transformed image according to the splicing edge; wherein, when aligning the transformed image with the reference image, a multi-level registration method is used for alignment. In some embodiments, it is also used to determine multiple stitching sides of the reference image; use the aligned edges of different cropped images and the corresponding stitching sides of the reference image to perform preliminary stitching and form a stitching seam; perform stitching detection on each stitching seam, determine the stitching matching coefficient, and mark the stitching seam with a stitching matching coefficient lower than a preset value as a pre-splicing seam; perform secondary multi-level registration processing on the corresponding aligned edges of the pre-splicing seam until the stitching matching coefficient of the pre-splicing seam formed by the corresponding aligned edges is not lower than the preset value; wherein the pixel condition in the secondary multi-level registration processing is higher than the pixel condition in the primary multi-level registration processing, and the corresponding aligned edge refers to the aligned edge of the transformed image and the reference image. It is also used to obtain the key point coordinate group and the contour coordinate group of the detection target, determine at least one special pixel point from the key point coordinate group and the contour coordinate group, construct a special feature area using the associated special pixel points, and obtain multiple feature area points, wherein the feature area point refers to a coordinate point that can reflect the typical features of the target; based on the labeling rules, the feature area points are screened to obtain the coordinates of the typical feature points.

[0061] The first computing unit 540 is configured to perform road vehicle target detection based on a convolutional neural network model using the panoramic data set.

[0062] In the above embodiments, the more specific working process of each functional unit can refer to the corresponding content disclosed in the above embodiments. In addition, each functional unit can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk (sol-identified state disk, SSD)), etc.

[0063] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0064] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0065] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0066] Obviously, those skilled in the art can make various changes and modifications to the embodiments of the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the embodiments of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A road vehicle detection method based on multi-view feature fusion, characterized in that: The steps include: Acquire image data collected from all viewing angles of the driving vehicle, and synchronize and sort all the image data collected using timestamp information to obtain multiple groups of synchronized image data; Performing annotation screening analysis on each group of the synchronous image data set, and using the synchronous image data set that meets the screening requirements as the standard data set, wherein the annotation screening analysis refers to performing a difference comparison on the initial annotation information of each sub-object in the synchronous image data set; Performing feature fusion on each image in the standard data set to obtain a panoramic data set, wherein the performing feature fusion on each image in the standard data set includes: calibrating a reference image from each image, cropping and transforming the remaining images and then splicing them with the reference image, wherein the cropping transformation refers to performing a scale transformation on the remaining images and then cropping the area overlapping with the reference image; The panoramic dataset is used to detect road vehicle targets based on a convolutional neural network model.

2. The road vehicle detection method based on multi-view feature fusion according to claim 1, characterized in that: The step of calibrating the reference image from each image comprises the following steps: Detect the detection target attributes in each image in the standard data set, wherein the detection target attributes include the type and quantity of the detection target; compare all images pairwise to obtain the overlap comparison result of the detection target attributes in each image; and select one of the images as the reference image according to the overlap comparison result.

3. The road vehicle detection method based on multi-view feature fusion according to claim 2 is characterized in that: The detecting target attribute in each image in the standard data set is determined based on initial annotation information of each image; Comparing all images pairwise includes: determining a target set in each image, the target set being composed of detection targets with different coordinates; and performing pairwise effective overlapping comparison on the target sets of each image, wherein the effective overlapping comparison is determined based on the similarity of coordinates of typical feature points of the detection targets.

4. The road vehicle detection method based on multi-view feature fusion according to claim 3 is characterized in that: The cutting and transforming of the remaining images and then splicing them with the reference image comprises the following steps: Determine at least one detection target corresponding to the remaining images and the reference image, determine a scale transformation value according to a neighbor distance ratio between the corresponding typical feature point coordinates of the detection target in the two images; perform scale transformation processing on the remaining images according to the scale transformation value to obtain a transformed image; Determine the overlapping area of ​​the transformed image relative to the reference image, clip the overlapping area of ​​the transformed image to obtain a clipped image; and splice the clipped image with the reference image.

5. The road vehicle detection method based on multi-view feature fusion according to claim 4 is characterized in that: Determining the overlapping area of ​​the transformed image relative to the reference image and clipping the overlapping area of ​​the transformed image specifically comprises the following steps: The transformed image is aligned with the reference image to determine the alignment edges of the two images respectively; the alignment edge of the transformed image is delineated based on the alignment edge of the reference image to obtain the splicing edge of the transformed image; the transformed image is trimmed according to the splicing edge; wherein, when aligning the transformed image with the reference image, a multi-level registration method is used for alignment.

6. The road vehicle detection method based on multi-view feature fusion according to claim 5 is characterized in that: The one-step multi-level registration method comprises the following steps: Acquire image data of the transformed image and the reference image under a first pixel condition, and align them according to the image data under the first pixel condition; acquire image data of the transformed image and the reference image under a second pixel condition, and align them according to the image data under the second pixel condition; The second pixel is larger than the first pixel; When aligning under the first pixel condition, the boundary range outline of the edge to be aligned is calibrated, and when aligning under the second pixel condition, key pixel points are selected for the boundary range outline.

7. The road vehicle detection method based on multi-view feature fusion according to claim 6 is characterized in that: The step of stitching all the cropped images with the reference image comprises the following steps: Determine multiple stitching sides of the reference image; use the aligned edges of different cropped images and the corresponding stitching sides of the reference image to perform preliminary stitching and form a stitching seam; perform stitching detection on each stitching seam, determine the stitching matching coefficient, and mark the stitching seam whose stitching matching coefficient is lower than a preset value as a pre-splicing seam; perform secondary multi-level registration processing on the corresponding aligned edges of the pre-splicing seam until the stitching matching coefficient of the pre-splicing seam formed by the corresponding aligned edges is not lower than the preset value; wherein the pixel condition in the secondary multi-level registration processing is higher than the pixel condition in the primary multi-level registration processing, and the corresponding aligned edge refers to the aligned edge of the transformed image and the reference image.

8. The road vehicle detection method based on multi-view feature fusion according to claim 7, characterized in that: When performing stitching detection on each stitching seam, it is determined whether there is a pixel point of the detection target in the stitching seam, and if so, a correction coefficient is assigned to the stitching matching coefficient.

9. The road vehicle detection method based on multi-view feature fusion according to claim 3 or 8, characterized in that: Determining the coordinates of typical feature points of the detection target includes the following steps: Obtain a key point coordinate group and a contour coordinate group of the detection target, determine at least one special pixel point from the key point coordinate group and the contour coordinate group, use the associated special pixel points to construct a special feature area, and obtain multiple feature area points, where the feature area points refer to coordinate points that can reflect typical features of the target; screen the feature area points based on the labeling rules to obtain typical feature point coordinates.

10. A road vehicle detection system with multi-view feature fusion, characterized in that: include: A first acquisition unit is used to acquire image acquisition data from all viewing angles of the driving vehicle, and synchronize and sort all the image acquisition data using timestamp information to obtain multiple groups of synchronized image data groups; a first processing unit, which is used to perform annotation screening analysis on each group of the synchronous image data sets, and use the synchronous image data sets that meet the screening requirements as standard data sets, wherein the annotation screening analysis refers to performing a difference comparison on the initial annotation information of each sub-object in the synchronous image data set; A second processing unit is configured to perform feature fusion on each image in the standard data set to obtain a panoramic data set, wherein the feature fusion on each image in the standard data set includes: calibrating a reference image from each image, cropping and transforming the remaining images and then splicing them with the reference image, wherein the cropping transformation refers to performing a scale transformation on the remaining images and then cropping the area overlapping with the reference image; A first computing unit is used to perform road vehicle target detection based on a convolutional neural network model using the panoramic data set.

Images