Method, device, equipment and medium for detecting target in moving vehicle

Abstract

The present application relates to a method, apparatus, device and medium for detecting a target during vehicle movement, wherein the method comprises: obtaining a to-be-detected image of a target vehicle during movement; inputting the to-be-detected image into a pre-trained first detector, so that the first detector outputs a first recognition result of a reference target contained in the to-be-detected image; inputting the to-be-detected image into a pre-trained second detector, so that the second detector outputs a second recognition result of at least one to-be-detected target contained in the to-be-detected image, wherein a corresponding relationship exists between the to-be-detected target and the reference target; comparing the first recognition result with the second recognition result; and correcting the second recognition result according to the comparison result to obtain a target recognition result. The technical solution of the present application can improve the accuracy of identifying obscured targets.

Description

Technical Field

[0001] The present application relates to the technical field of expected functional safety of autonomous driving, and in particular to a method, device, equipment and medium for detecting a target while a vehicle is moving. Background Art

[0002] In recent years, with the rise of deep learning technology, the field of computer vision has made rapid progress. Object detection, as a typical application, has been widely used in fields such as facial recognition and intelligent driving. Object detection in the intelligent driving field primarily focuses on road participant detection, traffic sign detection, and general obstacle detection. These scenarios require the collection of large amounts of road test data using real vehicles. However, during the collection process, due to occlusion by pedestrians, vehicles, non-motorized vehicles, or other obstacles, a large number of occluded samples are present in the road test data. These samples are missing information over a large area during training and prediction, resulting in incomplete or missing features. This deteriorates the performance of the detector, easily leading to inaccurate recognition results and missed recalls. Therefore, improving the accuracy of occluded target recognition has become a pressing technical issue. Summary of the Invention

[0003] The present application provides a method, device, equipment and medium for detecting targets while a vehicle is moving, which solves the problem in the related art of low accuracy in identifying obscured targets, and easy misidentification or missed detection.

[0004] The first aspect of the present application provides a method for detecting a target during vehicle movement, comprising the following steps: obtaining a picture to be detected of a target vehicle during movement; inputting the picture to be detected into a first detector that has been pre-trained, so that the first detector outputs a first recognition result of a reference target contained in the picture to be detected; inputting the picture to be detected into a second detector that has been pre-trained, so that the second detector outputs a second recognition result of at least one target to be detected contained in the picture to be detected, wherein a corresponding relationship exists between the target to be detected and the reference target; comparing the first recognition result with the second recognition result; and correcting the second recognition result based on the comparison result to obtain a target recognition result.

[0005] According to the above-mentioned technical means, the embodiment of the present application can obtain the picture to be detected of the target vehicle during driving, input the picture to be detected into a first detector that has been pre-trained, so that the first detector outputs a first recognition result of the reference target contained in the picture to be detected, and input the picture to be detected into a second detector that has been pre-trained, so that the second detector outputs a second recognition result of at least one target to be detected contained in the picture to be detected. There is a correspondence between the target to be detected and the reference target. Therefore, there is also a certain correspondence between the first recognition result and the second recognition result. The first recognition result and the second recognition result are compared, and the two can verify each other, so that the second recognition result is corrected according to the comparison result to obtain the accuracy of the target recognition result. Therefore, by comparing and correcting the first recognition result corresponding to the reference target and the second recognition result corresponding to the target to be detected, the accuracy of the target recognition result can be guaranteed, and the occurrence of misidentification or missed detection can be avoided.

[0006] Furthermore, the second recognition result includes a predicted category and a first confidence level corresponding to each target to be detected, and the first recognition result includes a baseline category corresponding to each target to be detected; based on the comparison result, the second recognition result is corrected to obtain a target recognition result, including: if the first recognition result and the predicted category corresponding to the current target to be detected are the same, then the predicted category corresponding to the current target to be detected is determined to be a correct result; if the first recognition result and the predicted category corresponding to the current target to be detected are different, then the predicted category corresponding to the current target to be detected is determined to be a confusing result, and based on the first confidence level of the predicted category determined to be the correct result corresponding to other targets to be detected adjacent to the current target to be detected, it is determined that the predicted category corresponding to the current target to be detected should be the second confidence level of the first recognition result; if the second confidence level reaches a predetermined threshold, the predicted category corresponding to the current target to be detected is corrected to the baseline category corresponding to the first recognition result.

[0007] According to the above technical means, the embodiments of the present application can improve the accuracy of the target recognition results of the target to be detected.

[0008] Further, based on the first confidence of the predicted categories determined to be correct results corresponding to other targets to be detected adjacent to the current target to be detected, the second confidence that the predicted category corresponding to the current target to be detected should be the first recognition result is determined, including: taking the average of the first confidences of the predicted categories determined to be correct results corresponding to other targets to be detected adjacent to the current target to be detected, as the second confidence that the predicted category corresponding to the current target to be detected should be the first recognition result.

[0009] According to the above technical means, the embodiment of the present application can ensure the rationality of the correction processing.

[0010] Further, based on the first confidence of the predicted category determined to be the correct result corresponding to other targets to be detected adjacent to the current target to be detected, it is determined that the predicted category corresponding to the current target to be detected should be the second confidence of the first recognition result, including: performing a weighted average operation based on the first confidence of the predicted category determined to be the correct result corresponding to other targets to be detected adjacent to the current target to be detected, and the first confidence of the predicted category determined to be the confusing result corresponding to other targets to be detected adjacent to the current target to be detected, to determine that the predicted category corresponding to the current target to be detected should be the second confidence of the first recognition result, wherein the weight of the first confidence of the predicted category determined to be the correct result is greater than the weight of the first confidence of the predicted category determined to be the confusing result.

[0011] According to the above technical means, the embodiment of the present application can comprehensively consider the confidence of the prediction categories judged as correct results and those judged as confusing results during the correction process, thereby ensuring its rationality.

[0012] Furthermore, the method also includes: obtaining first sample data of the reference target and its corresponding labeled data in the road test data; training a pre-constructed first detector based on the first sample data and its corresponding labeled data, so that the first detector can correctly output a first recognition result for the reference target; obtaining second sample data of the target to be detected in the road test data and its corresponding labeled data; training a pre-constructed second detector based on the second sample data and its corresponding labeled data, so that the second detector can correctly output a second recognition result for the target to be detected.

[0013] According to the above technical means, the first detector and the second detector are trained respectively according to the first sample data and the second sample data in the road test data and their corresponding labeled data, which can ensure the accuracy of the trained first detector and the second detector.

[0014] Furthermore, the reference target is a lane guide sign, and the target to be detected is a road guide arrow.

[0015] Based on the above technical means, the information of the vehicle's current lane can be accurately identified, providing a reliable basis for intelligent driving.

[0016] Furthermore, the first recognition result is compared with the second recognition result, including: if the picture to be detected does not contain both the reference target and the target to be detected, the first recognition result or the second recognition result obtained first is compared with the second recognition result or the first recognition result obtained later within a predetermined time interval.

[0017] According to the above technical means, the accuracy of the target recognition result can be improved even when the reference target and the target to be detected do not exist simultaneously in the image to be detected.

[0018] A second aspect of the present application provides a target detection device for a moving vehicle, comprising: an acquisition module for acquiring a picture to be detected of a target vehicle during driving; a first detection module for inputting the picture to be detected into a pre-trained first detector, so that the first detector outputs a first recognition result of a reference target contained in the picture to be detected; a second detection module for inputting the picture to be detected into a pre-trained second detector, so that the second detector outputs a second recognition result of at least one target to be detected contained in the picture to be detected, and there is a corresponding relationship between the target to be detected and the reference target; a comparison module for comparing the first recognition result with the second recognition result; and a processing module for correcting the second recognition result based on the comparison result to obtain a target recognition result.

[0019] Furthermore, the second recognition result includes a predicted category and a first confidence level corresponding to each target to be detected, and the first recognition result includes a reference category corresponding to each target to be detected; the processing module is used to: if the first recognition result and the predicted category corresponding to the current target to be detected are the same, then determine that the predicted category corresponding to the current target to be detected is a correct result; if the first recognition result and the predicted category corresponding to the current target to be detected are different, then determine that the predicted category corresponding to the current target to be detected is a confusing result, and based on the first confidence level of the predicted category determined as the correct result corresponding to other targets to be detected adjacent to the current target to be detected, determine that the predicted category corresponding to the current target to be detected should be the second confidence level of the first recognition result; if the second confidence level reaches a predetermined threshold, then correct the predicted category corresponding to the current target to be detected to the reference category corresponding to the first recognition result.

[0020] Furthermore, the processing module is used to: take the average of the first confidence levels of the predicted categories determined to be correct results corresponding to other targets to be detected adjacent to the current target to be detected, as the second confidence level that the predicted category corresponding to the current target to be detected should be the first recognition result.

[0021] Further, the processing module is used to: perform a weighted average operation based on the first confidence levels of the predicted categories determined to be correct results corresponding to other targets to be detected adjacent to the current target to be detected, and the first confidence levels of the predicted categories determined to be confusing results corresponding to other targets to be detected adjacent to the current target to be detected, to determine that the predicted category corresponding to the current target to be detected should be the second confidence level of the first recognition result, wherein the weight of the first confidence level of the predicted category determined to be the correct result is greater than the weight of the first confidence level of the predicted category determined to be the confusing result.

[0022] Furthermore, the processing module is also used to: obtain first sample data of the reference target and its corresponding labeled data in the road test data; train a pre-constructed first detector based on the first sample data and its corresponding labeled data, so that the first detector can correctly output a first recognition result for the reference target; obtain second sample data of the target to be detected in the road test data and its corresponding labeled data; train a pre-constructed second detector based on the second sample data and its corresponding labeled data, so that the second detector can correctly output a second recognition result for the target to be detected.

[0023] Furthermore, the reference target is a lane guide sign, and the target to be detected is a road guide arrow.

[0024] Furthermore, the comparison module is used to compare the first recognition result or the second recognition result obtained first with the second recognition result or the first recognition result obtained later within a predetermined time interval if the image to be detected does not contain both the reference target and the target to be detected.

[0025] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the target detection method during vehicle movement as described in the above embodiment.

[0026] A fourth aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the target detection method during vehicle driving as described in the above embodiment.

[0027] The embodiment of the present application can obtain a picture to be detected of a target vehicle during driving, input the picture to be detected into a first detector that has been pre-trained, so that the first detector outputs a first recognition result of a reference target contained in the picture to be detected, and input the picture to be detected into a second detector that has been pre-trained, so that the second detector outputs a second recognition result of at least one target to be detected contained in the picture to be detected. There is a correspondence between the target to be detected and the reference target. Therefore, there is also a certain correspondence between the first recognition result and the second recognition result. The first recognition result and the second recognition result are compared, and the two can verify each other, so that the second recognition result is corrected according to the comparison result to obtain the accuracy of the target recognition result. Therefore, by comparing and correcting the first recognition result corresponding to the reference target and the second recognition result corresponding to the target to be detected, the accuracy of the target recognition result can be guaranteed, and the occurrence of misidentification or missed detection can be avoided.

[0028] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0030] Figure 1 A schematic flow chart of a method for detecting an object while a vehicle is moving according to an embodiment of the present application;

[0031] Figure 2 A system framework diagram of a method for detecting an object in a moving vehicle to which an embodiment of the present application may be applied;

[0032] Figure 3 A schematic diagram of the process of algorithm annotation, training, and actual prediction of a method for detecting an object in a moving vehicle according to an embodiment of the present application;

[0033] Figure 4 This is a flowchart of a method for detecting an object while a vehicle is moving according to one embodiment of the present application;

[0034] Figure 5 is a schematic diagram of a reference image in a method for detecting an object while a vehicle is moving according to an embodiment of the present application;

[0035] Figure 6 A schematic flow chart of an arbitration method in a method for detecting an object in a moving vehicle according to an embodiment of the present application;

[0036] Figure 7 1 is a block diagram of a target detection device for a moving vehicle according to an embodiment of the present application;

[0037] Figure 8 is an example diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0038] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0039] Figure 1 This is a flow chart illustrating a method for detecting an object in a moving vehicle according to an embodiment of the present application. This method can be applied to a terminal, which may include, but is not limited to, one or more of a smartphone, tablet computer, portable computer, or vehicle-mounted device. In other embodiments, this method can also be applied to a server, without particular limitation.

[0040] Reference Figure 1 As shown, the target detection method for a moving vehicle includes at least steps S110 to S150. The following is an example of the method being applied to a terminal, and is described in detail as follows:

[0041] In step S110 , a picture to be detected of the target vehicle during its driving process is obtained.

[0042] In one embodiment, the target vehicle may be provided with an image acquisition device for acquiring a picture in front of the target vehicle in real time while the target vehicle is traveling as the picture to be detected.

[0043] In step S120 , the image to be detected is input into a pre-trained first detector, so that the first detector outputs a first recognition result of a reference target contained in the image to be detected.

[0044] The reference target may be a target used to assist in identifying the target to be detected. For example, if the target to be detected is a guide arrow on the road surface, the reference target may be a lane guide sign standing on both sides of the lane.

[0045] In this embodiment, the terminal may input the image to be detected into a pre-trained first detector. The first detector may have target detection and recognition capabilities. The first detector may perform target recognition on the image to be detected and output a first recognition result of a reference target contained in the image to be detected. For example, if the reference target is a lane guide sign, the first recognition result may include the driving direction of each lane in the lane guide sign, the relative distance between lanes, the relative position, etc.

[0046] In step S130, the image to be detected is input into a pre-trained second detector, so that the second detector outputs a second recognition result of at least one target to be detected contained in the image to be detected, and there is a corresponding relationship between the target to be detected and the reference target.

[0047] The target to be detected may be an actual object to be detected, such as a guide arrow on the road, a traffic light, a traffic sign, etc.

[0048] In this embodiment, the terminal can simultaneously input the image to be detected into a pre-trained second detector, which also has target detection and recognition functions. It should be understood that the difference between the first detector and the second detector lies in the different targets recognized. The second detector can recognize the image to be detected and output a second recognition result of at least one target to be detected contained in the image to be detected. The second recognition result can include the predicted category corresponding to each target to be detected and the corresponding first confidence level.

[0049] It should be noted that, since there is a corresponding relationship between the reference target and the target to be detected, there should also be a certain corresponding relationship between the first recognition result and the second recognition result.

[0050] In one example, one or more target detection algorithms of the YOLO series, the R-CNN series, or the SSD series can be used to train the first detector and the second detector.

[0051] In one embodiment of the present application, the method further includes:

[0052] Acquire first sample data containing the reference target and corresponding annotation data from the road test data;

[0053] Training a pre-built first detector based on the first sample data and its corresponding labeled data so that the first detector can correctly output a first recognition result for the reference target;

[0054] Acquire second sample data of the target to be detected and its corresponding annotation data from the road test data;

[0055] The pre-constructed second detector is trained based on the second sample data and the corresponding labeled data, so that the second detector can correctly output a second recognition result for the target to be detected.

[0056] In this embodiment, the road test data can be road data captured by an actual vehicle, such as a video or image. First sample data containing a reference target can be screened from the road test data and annotated. For example, if the reference target is a lane guide sign, the lane type and relative position of each lane can be annotated. A pre-built first detector is then trained based on the first sample data and its corresponding annotated data, so that the first detector can correctly output a first recognition result for the reference target.

[0057] Then, second sample data containing the target to be detected can be filtered from the road test data and annotated. For example, if the target to be detected is a road guide arrow, the lane category corresponding to each road guide arrow and the relative position between lanes can be annotated. The pre-built second detector is then trained based on the second sample data and its corresponding annotated data so that the second detector can correctly output a second recognition result for the target to be detected.

[0058] Thus, through pre-training, the accuracy of the recognition results of the first detector and the second detector is guaranteed, thereby improving the accuracy of target detection after comparison.

[0059] In step S140, the first recognition result and the second recognition result are compared;

[0060] In step S150, the second recognition result is corrected according to the comparison result to obtain a target recognition result.

[0061] The correction process may be a process of correcting the second recognition result according to the first recognition result.

[0062] In one embodiment, because the first recognition result is derived from a reference target, it can serve as prior information for the second recognition result. That is, the first recognition result can serve as a basis for determining whether the second recognition result is correct. Comparing the first and second recognition results can not only verify the accuracy of the second recognition result, but also provide the correct prediction category for an erroneous second recognition result. Even when the target to be detected is occluded, the accuracy of the target recognition result obtained after correction processing can be guaranteed.

[0063] based on Figure 1 In the embodiment shown, in one embodiment of the present application, the second recognition result includes the predicted category and the first confidence level corresponding to each target to be detected, and the first recognition result includes the reference category corresponding to each target to be detected; then step S150 specifically includes:

[0064] According to the comparison result, the second recognition result is corrected to obtain a target recognition result, including:

[0065] If the first recognition result is the same as the predicted category corresponding to the current target to be detected, determining that the predicted category corresponding to the current target to be detected is a correct result;

[0066] If the first recognition result and the predicted category corresponding to the current target to be detected are different, determining that the predicted category corresponding to the current target to be detected is a confusing result, and determining, based on the first confidence levels of the predicted categories of other targets to be detected adjacent to the current target to be detected that are determined to be correct results, that the predicted category corresponding to the current target to be detected should be the second confidence level of the first recognition result;

[0067] If the second confidence reaches a predetermined threshold, the predicted category corresponding to the current target to be detected is corrected to the same predicted category as the first recognition result.

[0068] In this embodiment, the first recognition result includes a reference category corresponding to each target to be detected. It should be understood that the first recognition result is a recognition result corresponding to the reference target, so the first recognition result can be used as a basis for judging whether the second recognition result is correct. Therefore, the predicted category in the first recognition result can be used as a reference category for each target to be detected. Taking the reference target as a lane guide sign as an example, the first recognition result includes the lane category of each lane in the lane guide sign, such as a left turn lane, a straight lane, a right turn lane, and the relative positions between lanes, such as a left turn lane is located on the left side of a straight lane, etc. Therefore, the lane category of each lane contained in the first recognition result can be used as a reference basis for the reference category of each lane guide arrow (i.e., the target to be detected) in the second recognition result.

[0069] According to the comparison results, if the first recognition result is the same as the predicted category corresponding to the current target to be detected, it means that the predicted category corresponding to the current target to be detected is a correct recognition result and can be retained. It should be understood that the image to be detected may contain more than one target to be detected. For the case of multiple targets to be detected, the second recognition result may include multiple (i.e., two or more than two arbitrary numbers) predicted categories corresponding to the targets to be detected, confidence levels, and relative positions between the targets to be detected. Therefore, when comparing the first recognition result and the second recognition result, they can be compared in sequence according to the relative positions.

[0070] If the predicted categories corresponding to the first recognition result and the current target to be detected are different, that is, the baseline category corresponding to the target to be detected in the first recognition result and the predicted category of the target to be detected in the second recognition result are different, then the predicted category corresponding to the current target to be detected can be determined to be a confusing result, that is, correction processing is required. For the current target to be detected whose predicted category is determined to be a confusing result, the second confidence level of the predicted category corresponding to the current target to be detected can be determined to be the first recognition result based on the first confidence level of the predicted categories corresponding to other targets to be detected that are adjacent to the current target to be detected and are determined to be correct results.

[0071] Specifically, for the current target to be detected whose predicted category is determined to be a confusing result, it can be determined whether there is a predicted category determined to be a correct result in the comparison results of other adjacent targets to be detected. If so, the predicted category corresponding to the other targets to be detected that are determined to be correct can be used to determine the second confidence level that the predicted category corresponding to the current target to be detected should be the first recognition result. It should be understood that the second confidence level can be used to characterize the degree of confidence that the first recognition result is correct. The larger the second confidence level, the greater the degree of confidence that the predicted category should be the baseline category, and vice versa.

[0072] For example, there are three lane guide arrows (i.e., targets to be detected) 1, 2, and 3 set in sequence, where the predicted categories of arrows 1 and 3 are the same as the baseline categories in the corresponding first recognition results and are determined to be correct results, while the predicted category corresponding to arrow 2 is different from the baseline category in the corresponding first recognition result. Then, based on the first confidence of the predicted categories of arrows 1 and 3, it can be determined that the predicted category corresponding to arrow 2 should be the second confidence of the baseline category in the first recognition result. It should be understood that the higher the degree of similarity between the first recognition result and the second recognition result, the more reliable the first recognition result is, and therefore the higher the confidence that the predicted category of the confusing result should be the corresponding baseline category.

[0073] In one example, for the current target to be detected whose predicted category is determined to be a confusing result, the first confidence scores of the predicted categories of other adjacent targets to be detected that are determined to be correct results can be averaged to serve as the second confidence score of the current target to be detected.

[0074] After determining the second confidence level corresponding to the current target to be detected, the second confidence level can be compared with a pre-set threshold. If the second confidence level reaches the pre-set threshold, i.e., is greater than or equal to the pre-set threshold, the predicted category corresponding to the current target can be revised to the corresponding baseline category; otherwise, the predicted category can be retained. The second recognition result that has undergone the above-mentioned revision process is then output as the target recognition result.

[0075] Therefore, in the above embodiment, the first confidence level of the predicted categories of other targets to be detected adjacent to the current target to be detected that are judged to be correct results is used to determine the second confidence level that the predicted category of the current result to be detected should be the baseline category. The predicted category corresponding to the current target to be detected is corrected according to the second confidence level, which can ensure the rationality of the correction process and thus ensure the accuracy of the target recognition result.

[0076] Based on the foregoing embodiment, in one embodiment of the present application, determining that the predicted category corresponding to the current target to be detected should be the second confidence level of the first recognition result based on the first confidence levels of the predicted categories of other targets to be detected adjacent to the current target to be detected that are determined to be correct results includes:

[0077] The first confidence levels of the predicted categories of other targets to be detected adjacent to the current target to be detected and determined to be correct results are averaged to serve as the second confidence level that the predicted category corresponding to the current target to be detected should be the first recognition result.

[0078] In one embodiment of the present application, determining that the predicted category corresponding to the current target to be detected should be the second confidence level of the first recognition result based on the first confidence levels of the predicted categories of other targets to be detected adjacent to the current target to be detected that are determined to be correct results includes:

[0079] A weighted average operation is performed based on the first confidence levels of the predicted categories determined to be correct results corresponding to other targets to be detected adjacent to the current target to be detected, and the first confidence levels of the predicted categories determined to be confusing results corresponding to other targets to be detected adjacent to the current target to be detected, to determine that the predicted category corresponding to the current target to be detected should be the second confidence level of the first recognition result, wherein the weight of the first confidence level of the predicted category determined to be the correct result is greater than the weight of the first confidence level of the predicted category determined to be the confusing result.

[0080] In this embodiment, not only can the second confidence that the predicted category of the current target to be detected should be the baseline category be determined based on the first confidence of the predicted category determined as the correct result in other targets to be detected adjacent to the current target to be detected, but the determination can also be made in combination with the first confidence of the predicted category determined as the confusing result in other adjacent targets to be detected.

[0081] Specifically, a weighted average operation can be performed on the first confidence level of the predicted category determined to be a correct result and the first confidence level of the predicted category determined to be a confusing result to determine that the predicted category corresponding to the current target to be detected should be the second confidence level of the baseline category in the first recognition result. Those skilled in the art can set the corresponding weights based on prior experience, wherein the weight of the first confidence level of the predicted category determined to be a correct result is greater than the weight of the first confidence level of the predicted category determined to be a confusing result to ensure the rationality of the weight setting.

[0082] Based on the foregoing embodiment, in one embodiment of the present application, comparing the first recognition result with the second recognition result includes:

[0083] If the image to be detected does not contain both the reference target and the target to be detected, the first recognition result or the second recognition result obtained earlier is compared with the second recognition result or the first recognition result obtained later within a predetermined time interval.

[0084] In this embodiment, when the image to be detected does not contain both the reference target and the target to be detected, the first recognition result or the second recognition result obtained first can be compared with the second recognition result or the first recognition result obtained later within a predetermined time interval, which can ensure that the comparison can be performed even when the reference target and the target to be detected do not appear in the image to be detected at the same time. At the same time, by setting the predetermined time interval, the validity of the first recognition result or the second recognition result obtained first can be guaranteed, and the failure of the first recognition result or the second recognition result due to the long time interval and the long vehicle travel distance can be avoided. It should be noted that those skilled in the art can set the predetermined time interval based on prior experience, such as 5S, 10S, etc.

[0085] Based on the technical solutions of the above embodiments, a specific application scenario of the embodiments of the present application is introduced below:

[0086] Figure 2 4 is a system framework diagram of a method for detecting a target in a moving vehicle to which an embodiment of the present application can be applied.

[0087] Reference Figure 2 As shown, the system includes a priori information extraction module, a target detection module and a target arbitration module.

[0088] Among them, the prior information extraction module is used to identify the reference target contained in the detection picture and output the corresponding prior information (i.e., the first recognition result); the target detection module is used to identify the target to be detected contained in the detection picture and output the corresponding recognition result (i.e., the second recognition result); the target arbitration module is used to compare the prior information and the recognition result to arbitrate the recognition result (i.e., correction processing), and finally output the recognition result after arbitration, i.e., the target recognition result.

[0089] Figure 3 The following is a flow chart of the algorithm annotation, training, and actual prediction process of the target detection method for a moving vehicle according to an embodiment of the present application. The following uses the method applied to the road guide arrow scene as an example to illustrate.

[0090] Reference Figure 3 As shown in the figure, prior information refers to information obtained before algorithm training or prediction. For the scene of road guide arrow recognition, the prior information can come from multiple sources, including but not limited to traffic lights, point cloud data, lane signs, and other fused information. Preferably, the prior information is extracted from the lane guide signs on each road, including but not limited to lane type, shape, location and other information. The advantages are: (1) lane guide signs are generally high and not easily blocked; (2) lane guide signs have a large contrast with the background, making them easy to accurately identify.

[0091] During the algorithm labeling and training phase: two detectors need to be trained for lane guide signs (i.e., reference targets) and road guide arrows (i.e., targets to be detected), respectively, in two rounds.

[0092] In the first round, a lane guide sign detector (i.e., detector 1) is obtained. During the labeling phase, samples containing lane guide signs are screened based on road test data collected from actual vehicles. These samples can be videos or images. The lane guide signs and each lane in the lane guide signs are then labeled, including their location and category, to obtain a certain amount of labeled data. During the training phase, training is performed using the samples and the corresponding labeled data. In one example, one or more target detection algorithms from the YOLO series, R-CNN series, or SSD series can be used to train a lane guide sign detector capable of recognizing various lane types.

[0093] The second round of analysis follows the same principles as the first, yielding a road guide arrow detector (i.e., Detector 2). However, the second round of annotations focuses on lane samples from the road test data. These two detectors have two applications: one is automated labeling, and the other is deployment in production vehicles to provide underlying intelligent driving service support.

[0094] In the algorithm prediction stage: Taking application two as an example, the detector has been deployed inside the mass-produced vehicle. When the mass-produced vehicle is on the road, the system inputs the collected image information into two detectors when performing ground lane recognition: the lane guide sign detector detects whether there are lane guide signs. If so, it also detects the lane indication information, including the number of lanes and lane category (i.e., the baseline category); the road guide arrow detector detects whether there are guide arrows in the image. If so, it detects the category of the guide arrow (i.e., the predicted category).

[0095] It should be noted that there are two situations in the image to be detected: (1) the lane guide sign and the lane guide arrow on the ground appear in the image at the same time, which is called the information synchronization scenario; (2) the lane guide sign and the lane guide arrow do not appear in the same image, for example, the lane guide sign image appears first, and the lane guide arrow image appears on the ground a few seconds later, or the lane guide arrow image appears first, and the lane guide sign image appears a few seconds later, which is called the information asynchronous scenario. In situation (1), the information output by the two detectors is theoretically highly consistent and can echo each other. In situation (2), the information output by the two detectors may be biased, that is, it does not correspond to the information of the same road, so a comprehensive analysis should be conducted.

[0096] Taking case (1) as an example to illustrate the process of target recognition and arbitration, the difference between case (2) and case (1) lies in the order in which the prior information is obtained. The output information of the two detectors is aligned by setting a time window (i.e., a predetermined time interval). Within the time window, the information obtained first waits for the acquisition of the other information. If no data is obtained beyond the time window, the time window is recreated with the next data obtained.

[0097] The entire detection process of situation (1) is as follows Figure 4 As shown, the lane guide sign detector is referred to as the first detector and the road guide arrow detector is referred to as the second detector hereinafter.

[0098] First, the first detector identifies the position of the lane guide sign in the picture and the lane guide information in the lane guide sign (the number of lanes, the relative position of each guide lane in the entire road, and the relative distance between lanes), such as Figure 5 The diagram shows information about five lanes with four guidance categories (A, B, C, and D). Lanes are separated by a relative distance, forming a baseline graph. The nodes in the graph represent lane guidance categories, and the weights of the edges (connections between nodes) represent the relative distances, forming the prior information for the road.

[0099] Second, the second detector identifies the road guide arrows in the picture, including the road guide arrow information of one or more lanes, obtains the predicted category of each road guide arrow and the relative position and relative distance between the road guide arrows, and constructs a similar Figure 5 The same subgraph forms the prediction information of the road.

[0100] Third, the sub-graph of the predicted information is matched with the reference graph of the prior information. Two mapping methods (i.e., sub-graph matching algorithms) are provided for this scenario: (1) If the road guide arrow of the target vehicle's current lane is not blocked by other vehicles, its information can be used as the matching priority. The reason is that the ground guide arrow of the current lane is in the front position, with a small degree of perspective change and a high proportion in the picture, which is easier to accurately identify. Then, the left and right lanes of the current lane are matched, and then the outer layer is matched. Each matching result is weighted and summed, and the weight decreases layer by layer from the current lane. Finally, the matching method with the highest score is selected as the lane matching result; (2) If the current lane is blocked, the lane with the unobstructed guide arrow is matched first. The search starts from the left and right lanes, and the first unobstructed lane found is the matching priority. If an obstructed lane is encountered, it is skipped.

[0101] Fourth, the baseline category of each node (lane) is obtained based on the isomorphic subgraph. Finally, a certain arbitration method is used to determine the baseline category and the predicted category to obtain the final category of each predicted target.

[0102] The present application also provides an arbitration method, such as Figure 6 As shown:

[0103] (1) Traverse the detection target set (i.e., the first recognition result corresponding to at least one target to be detected), and for each target (i.e., the target to be detected, here is the lane): determine whether the predicted category is consistent with the baseline category: if so, this predicted category is considered to be correctly identified, and the node is considered to be a correct node; if not, the node is considered to be a confusing node and requires arbitration.

[0104] (2) Traverse each confused node: Obtain the set of nodes directly connected to the confused node (i.e., other targets adjacent to the current target to be detected). Each node in the set that is judged to be correct votes on the node to determine whether its predicted category is correct. Specifically, the confidence of each node (i.e., the first confidence) is summed and averaged, and the confidence of the confused node as the baseline category (i.e., the second confidence) is calculated.

[0105] (3) Set a confidence threshold, such as 0.8. When the confidence is higher than the threshold, the category is determined to be the baseline category; otherwise, it is determined to be the predicted category. At the same time, the node with the updated category is considered the correct node, and the average confidence of the neighboring nodes after voting is used as its confidence.

[0106] (4) When all obfuscation nodes are determined to be complete, the arbitration ends.

[0107] Optionally, the nodes used for voting in (2) may include non-arbitrated obfuscated nodes or obfuscated nodes themselves that are being arbitrated. In this case, a weighted average method is used to assign a higher weight to the correct node and a lower weight to the obfuscated node.

[0108] Simply put, for the scenario of road guide sign recognition, based on the current lane and guidance information of the car, the guidance information of other lanes can be accurately calculated based on relative position and relative distance. On this basis, the intelligent driving system can make decisions and plans based on the accurately identified lane guidance to ensure the efficient implementation of its downstream applications.

[0109] Next, a device for detecting a target in a moving vehicle according to an embodiment of the present application will be described with reference to the accompanying drawings.

[0110] Figure 7 4 is a block diagram of a target detection device for a moving vehicle according to an embodiment of the present application.

[0111] like Figure 7 As shown, the target detection device for a moving vehicle includes:

[0112] The acquisition module 710 is used to acquire the image to be detected of the target vehicle during its driving process;

[0113] A first detection module 720 is configured to input the image to be detected into a pre-trained first detector, so that the first detector outputs a first recognition result of a reference target contained in the image to be detected;

[0114] A second detection module 730 is configured to input the image to be detected into a pre-trained second detector, so that the second detector outputs a second recognition result of at least one target to be detected contained in the image to be detected, and there is a corresponding relationship between the target to be detected and the reference target;

[0115] a comparison module 740, configured to compare the first recognition result with the second recognition result;

[0116] The processing module 750 is used to correct the second recognition result according to the comparison result to obtain a target recognition result.

[0117] Furthermore, the second recognition result includes a predicted category and a first confidence level corresponding to each target to be detected, and the first recognition result includes a reference category corresponding to each target to be detected; the processing module 750 is used to: if the first recognition result and the predicted category corresponding to the current target to be detected are the same, then determine that the predicted category corresponding to the current target to be detected is a correct result; if the first recognition result and the predicted category corresponding to the current target to be detected are different, then determine that the predicted category corresponding to the current target to be detected is a confusing result, and based on the first confidence level of the predicted category determined as the correct result corresponding to other targets to be detected adjacent to the current target to be detected, determine that the predicted category corresponding to the current target to be detected should be the second confidence level of the first recognition result; if the second confidence level reaches a predetermined threshold, then correct the predicted category corresponding to the current target to be detected to the same predicted category as the first recognition result.

[0118] Furthermore, the processing module 750 is used to: take the average of the first confidence levels of the predicted categories determined to be correct results corresponding to other targets to be detected adjacent to the current target to be detected, as the second confidence level that the predicted category corresponding to the current target to be detected should be the first recognition result.

[0119] Further, the processing module 750 is used to: perform weighted sum operation based on the first confidence of the predicted category determined to be the correct result corresponding to other targets to be detected adjacent to the current target to be detected, and the first confidence of the predicted category determined to be the confusing result corresponding to other targets to be detected adjacent to the current target to be detected, to determine that the predicted category corresponding to the current target to be detected should be the second confidence of the first recognition result, wherein the weight of the first confidence of the predicted category determined to be the correct result is greater than the weight of the first confidence of the predicted category determined to be the confusing result.

[0120] Furthermore, the processing module 750 is also used to: obtain first sample data of the reference target and its corresponding annotation data in the road test data; train a pre-constructed first detector based on the first sample data and its corresponding annotation data, so that the first detector can correctly output a first recognition result for the reference target; obtain second sample data of the target to be detected in the road test data and its corresponding annotation data; train a pre-constructed second detector based on the second sample data and its corresponding annotation data, so that the second detector can correctly output a second recognition result for the target to be detected.

[0121] Furthermore, the reference target is a lane guide sign, and the target to be detected is a road guide arrow.

[0122] Furthermore, the comparison module 740 is used to compare the first recognition result or the second recognition result obtained first with the second recognition result or the first recognition result obtained later within a predetermined time interval if the image to be detected does not contain both the reference target and the target to be detected.

[0123] It should be noted that the above explanation of the embodiment of the method for detecting a target while a vehicle is moving is also applicable to the device for detecting a target while a vehicle is moving in this embodiment, and will not be repeated here.

[0124] According to the target detection device for a moving vehicle proposed in an embodiment of the present application, a picture to be detected of the target vehicle during driving is obtained, and the picture to be detected is input into a first detector that has been pre-trained so that the first detector outputs a first recognition result of the reference target contained in the picture to be detected, and the picture to be detected is input into a second detector that has been pre-trained so that the second detector outputs a second recognition result of at least one target to be detected contained in the picture to be detected. There is a correspondence between the target to be detected and the reference target, and thus, there is also a certain correspondence between the first recognition result and the second recognition result. The first recognition result and the second recognition result are compared, and the two can verify each other, so that the second recognition result is corrected according to the comparison result to obtain the accuracy of the target recognition result. Therefore, by comparing and correcting the first recognition result corresponding to the reference target and the second recognition result corresponding to the target to be detected, the accuracy of the target recognition result can be guaranteed, and the occurrence of misidentification or missed detection can be avoided.

[0125] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0126] A memory 601 , a processor 602 , and a computer program stored in the memory 601 and executable on the processor 602 .

[0127] When the processor 602 executes the program, the method for detecting an object in a moving vehicle provided in the above embodiment is implemented.

[0128] Furthermore, the electronic device further includes:

[0129] The communication interface 603 is used for communication between the memory 601 and the processor 602 .

[0130] The memory 601 is used to store computer programs that can be run on the processor 602 .

[0131] The memory 601 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0132] If the memory 601, processor 602, and communication interface 603 are implemented independently, the communication interface 603, memory 601, and processor 602 can be connected to each other via a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0133] Optionally, in a specific implementation, if the memory 601, the processor 602 and the communication interface 603 are integrated on a chip, the memory 601, the processor 602 and the communication interface 603 can communicate with each other through an internal interface.

[0134] The processor 602 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0135] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned method for detecting a target in a moving vehicle.

[0136] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0137] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of this application, "N" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0138] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application belong.

[0139] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or N wires (electronic devices), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and a portable compact disc read-only memory (CDROM). Furthermore, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing it in another suitable manner if necessary, and then storing it in a computer memory.

[0140] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiment, the N steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0141] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0142] In addition, the functional units in the various embodiments of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into a module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0143] The storage medium mentioned above may be a read-only memory, a magnetic disk, or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present application. Persons skilled in the art may make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

Claims

1. A method for detecting a target in a moving vehicle, characterized in that: The following steps are involved: Obtain the image of the target vehicle to be detected during driving; Inputting the image to be detected into a pre-trained first detector so that the first detector outputs a first recognition result of a reference target contained in the image to be detected; Inputting the image to be detected into a pre-trained second detector so that the second detector outputs a second recognition result of at least one target to be detected contained in the image to be detected, and there is a corresponding relationship between the target to be detected and the reference target; comparing the first recognition result with the second recognition result; According to the comparison result, the second recognition result is corrected to obtain a target recognition result; The second recognition result includes a predicted category corresponding to each target to be detected and a first confidence level, and the first recognition result includes a reference category corresponding to each target to be detected; According to the comparison result, the second recognition result is corrected to obtain a target recognition result, including: If the first recognition result is the same as the predicted category corresponding to the current target to be detected, determining that the predicted category corresponding to the current target to be detected is a correct result; If the first recognition result and the predicted category corresponding to the current target to be detected are different, determining that the predicted category corresponding to the current target to be detected is a confusing result, and determining, based on the first confidence levels of the predicted categories of other targets to be detected adjacent to the current target to be detected that are determined to be correct results, that the predicted category corresponding to the current target to be detected should be the second confidence level of the first recognition result; If the second confidence level reaches a predetermined threshold, the predicted category corresponding to the current target to be detected is corrected to the corresponding reference category in the first recognition result.

2. The method according to claim 1, characterized in that Determining, based on first confidence levels of predicted categories determined to be correct results corresponding to other targets to be detected adjacent to the current target to be detected, that the predicted category corresponding to the current target to be detected should be a second confidence level of the first recognition result, includes: The first confidence levels of the predicted categories of other targets to be detected adjacent to the current target to be detected and determined to be correct results are averaged to serve as the second confidence level that the predicted category corresponding to the current target to be detected should be the first recognition result.

3. The method according to claim 1, characterized in that Determining, based on first confidence levels of predicted categories determined to be correct results corresponding to other targets to be detected adjacent to the current target to be detected, that the predicted category corresponding to the current target to be detected should be a second confidence level of the first recognition result, includes: A weighted average operation is performed based on the first confidence levels of the predicted categories determined to be correct results corresponding to other targets to be detected adjacent to the current target to be detected, and the first confidence levels of the predicted categories determined to be confusing results corresponding to other targets to be detected adjacent to the current target to be detected, to determine that the predicted category corresponding to the current target to be detected should be the second confidence level of the first recognition result, wherein the weight of the first confidence level of the predicted category determined to be the correct result is greater than the weight of the first confidence level of the predicted category determined to be the confusing result.

4. The method according to claim 1, wherein The method further comprises: Acquire first sample data containing the reference target and corresponding annotation data from the road test data; Training a pre-built first detector based on the first sample data and its corresponding labeled data so that the first detector can correctly output a first recognition result for the reference target; Acquire second sample data of the target to be detected and its corresponding annotation data from the road test data; The pre-constructed second detector is trained based on the second sample data and the corresponding labeled data, so that the second detector can correctly output a second recognition result for the target to be detected.

5. The method according to any one of claims 1 to 4, characterized in that The reference target is a lane guide sign, and the target to be detected is a road guide arrow.

6. The method according to any one of claims 1 to 4, characterized in that Comparing the first recognition result with the second recognition result includes: If the image to be detected does not contain both the reference target and the target to be detected, the first recognition result or the second recognition result obtained earlier is compared with the second recognition result or the first recognition result obtained later within a predetermined time interval.

7. A target detection device for a moving vehicle, characterized in that: include: An acquisition module is used to obtain the image to be detected of the target vehicle during its driving process; A first detection module is configured to input the image to be detected into a pre-trained first detector, so that the first detector outputs a first recognition result of a reference target contained in the image to be detected; A second detection module is configured to input the image to be detected into a pre-trained second detector, so that the second detector outputs a second recognition result of at least one target to be detected contained in the image to be detected, and there is a corresponding relationship between the target to be detected and the reference target; a comparison module, configured to compare the first recognition result with the second recognition result; a processing module, configured to correct the second recognition result according to the comparison result to obtain a target recognition result; The second recognition result includes a predicted category corresponding to each target to be detected and a first confidence level, and the first recognition result includes a reference category corresponding to each target to be detected; The processing module is configured to: if the first recognition result is the same as the predicted category corresponding to the current target to be detected, determine that the predicted category corresponding to the current target to be detected is a correct result; If the first recognition result and the predicted category corresponding to the current target to be detected are different, determining that the predicted category corresponding to the current target to be detected is a confusing result, and determining, based on the first confidence levels of the predicted categories of other targets to be detected adjacent to the current target to be detected that are determined to be correct results, that the predicted category corresponding to the current target to be detected should be the second confidence level of the first recognition result; If the second confidence level reaches a predetermined threshold, the predicted category corresponding to the current target to be detected is corrected to the corresponding reference category in the first recognition result.

8. The device according to claim 7, characterized in that The processing module is used for: The first confidence levels of the predicted categories of other targets to be detected adjacent to the current target to be detected and determined to be correct results are averaged to serve as the second confidence level that the predicted category corresponding to the current target to be detected should be the first recognition result.

9. The device according to claim 7, characterized in that The processing module is used for: A weighted average operation is performed based on the first confidence levels of the predicted categories determined to be correct results corresponding to other targets to be detected adjacent to the current target to be detected, and the first confidence levels of the predicted categories determined to be confusing results corresponding to other targets to be detected adjacent to the current target to be detected, to determine that the predicted category corresponding to the current target to be detected should be the second confidence level of the first recognition result, wherein the weight of the first confidence level of the predicted category determined to be the correct result is greater than the weight of the first confidence level of the predicted category determined to be the confusing result.

10. The device according to claim 7, characterized in that The processing module is further configured to: Acquire first sample data containing the reference target and corresponding annotation data from the road test data; Training a pre-built first detector based on the first sample data and its corresponding labeled data so that the first detector can correctly output a first recognition result for the reference target; Acquire second sample data of the target to be detected and its corresponding annotation data from the road test data; The pre-constructed second detector is trained based on the second sample data and the corresponding labeled data, so that the second detector can correctly output a second recognition result for the target to be detected.

11. The device according to any one of claims 7 to 10, characterized in that The reference target is a lane guide sign, and the target to be detected is a road guide arrow.

12. The device according to any one of claims 7 to 10, characterized in that The comparison module is used to: If the image to be detected does not contain both the reference target and the target to be detected, the first recognition result or the second recognition result obtained earlier is compared with the second recognition result or the first recognition result obtained later within a predetermined time interval.

13. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for detecting a target in a moving vehicle according to any one of claims 1 to 6.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the target detection method for a moving vehicle as described in any one of claims 1 to 6.

Images