Long-tail target identification method, readable storage medium and vehicle
By adding long-tail target recognition branches to common target models and generating simulation data, the problems of high cost and poor robustness of long-tail target recognition data acquisition in the existing technology are solved, efficient long-tail target recognition and driver attention enhancement are achieved, and traffic safety is significantly improved.
Patent Information
- Application Number
- CN202311616687.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-11-29
- Publication Date
- 2025-05-30
AI Technical Summary
The prior art has problems such as high data acquisition cost, poor robustness and sensitivity to severe weather in long-tail target recognition, making it difficult to effectively identify and deal with extreme situations in vehicle driving.
By adding long-tail target recognition branches to common target models and adding augmented data to the training data to generate simulated long-tail target data, the augmented target model is trained to identify common and long-tail targets.
It realizes long-tail target recognition without the need for large amounts of long-tail target data acquisition and labeling, improves robustness and recognition accuracy, is suitable for a variety of sensor environments, and significantly improves driving and traffic safety.
Smart Images
Figure CN120071276A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of vehicle driving, and specifically provides a long-tail target recognition method, a driver attention enhancement method, a computer-readable storage medium, and a vehicle.
Background Art
[0002] At present, the perception system of a vehicle collects information through sensors and extracts effective information from the environment. It can perform context understanding of the driving environment, such as detecting, tracking, and segmenting obstacles, road signs / lanes, and open drivable areas. The environmental perception task is mainly completed by using sensors such as on-vehicle cameras, lidar, millimeter-wave radars, or multi-sensor fusion. The perception system obtains a large amount of surrounding environment information through multi-sensors to ensure the correct understanding of the surrounding environment of the vehicle, so as to make corresponding decision-making plans and controls subsequently.
[0003] If we want the vehicle to recognize various scene targets, we must find these scene targets in advance and use the data of these scene targets to train the target detection model. Common scene targets are particularly easy to find, but there are many special scene targets (such as a vehicle rolling over ahead, a tire falling on the road ahead, a small animal suddenly appearing on the road, etc.), that is, the so-called long-tail targets (also known as abnormal targets or extreme situations) are very difficult to encounter.
[0004] Theoretically, it is only possible to infinitely approach but impossible to completely find all scene targets, so there are many unresolved long-tail targets. Therefore, the accident rate and failure rate of vehicles (especially autonomous vehicles) remain high, and it seriously affects traffic safety.
[0005] In the prior art, the following two solutions are usually adopted to strengthen the recognition of long-tail targets:
[0006] Solution 1: By collecting or artificially creating long-tail data to enhance the recognition and detection capabilities of the long-tail target data of the perception algorithm.
[0007] Solution 2: Develop a target detection algorithm based on lidar + clustering, compare it with the perception result, and find out the targets missed in the perception result, which are the long-tail targets.
[0008] However, Solution 1 has the following problems: The long-tail data cannot be exhausted, so this solution can only partially alleviate the problem of missed detection of long-tail target data and cannot fundamentally solve this problem. Moreover, the collection cost of long-tail target data is very high, and additional annotation costs are also required.
[0009] Moreover, Solution 2 has the following problems: The lidar-based method not only requires the installation of lidar, which is costly, but also the lidar is vulnerable to interference from rain, snow, and fog weather and is basically ineffective in bad weather. The clustering-based method requires manual parameter setting according to different application scenarios, and the algorithm has poor robustness.
[0010] Therefore, there is an urgent need to propose a novel long-tail target recognition method to overcome the problems existing in the above two solutions in the prior art.
Summary of the Invention
[0011] In view of the above, one of the objectives of the present invention is to propose a long-tail target recognition method that does not require collecting and annotating a large amount of data, has good robustness, can solve most long-tail target recognition problems, and does not require enumerating all scenarios of long-tail target types.
[0012] One of the objectives of the present invention is to propose a driver attention enhancement method.
[0013] One of the objectives of the present invention is to propose a computer-readable storage medium storing a computer program, where when the computer program is executed by a processor, the long-tail target recognition method and / or the driver attention enhancement method of the present invention are implemented.
[0014] One of the objectives of the present invention is to propose a vehicle, which includes a memory and a processor, where a computer program is stored in the memory, and when the computer program is executed by the processor, the long-tail target recognition method and / or the driver attention enhancement method of the present invention are implemented.
[0015] According to the first aspect of the present invention, a long-tail target recognition method is provided, including the following steps:
[0016] Common target model training step: Using training data containing common target data to train a common target model including a backbone network and a common target recognition branch;
[0017] Long-tail target recognition branch adding step: Adding an additional long-tail target recognition branch for recognizing long-tail targets to the backbone network in the trained common target model;
[0018] Augmented data adding step: Generating simulated long-tail target data by adding augmented data to the training data;
[0019] Enhanced target model training step: Training the long-tail target recognition branch based on the generated simulated long-tail target data without updating the common target recognition branch to obtain an enhanced target model; and
[0020] Target recognition step: Use the enhanced target model to detect and recognize common targets and long-tail targets. Among them, the common targets are detected and recognized through the common target recognition branch in the enhanced target model, and the long-tail targets are detected and recognized through the long-tail target recognition branch in the enhanced target model.
[0021] According to the second aspect of the present invention, there is provided a method for recognizing long-tail targets for a vehicle, including the following steps:
[0022] Common target detection model training step: Use training data containing common target data to train a common target detection model including a backbone network and a common target detection branch. Among them, the common target data includes common target images and corresponding target detection annotations. Among them, the common target images are images collected by the vehicle's on-vehicle camera or images from an open-source database;
[0023] Long-tail target detection branch adding step: Add an additional long-tail target detection branch for detecting long-tail targets to the trained common target detection model;
[0024] Augmented data adding step: Generate simulated long-tail target data by adding augmented data to the training data. Among them, the augmented data is any camera image or an image generated by a predetermined program;
[0025] Enhanced target detection model training step: Based on the generated simulated long-tail target data, train the long-tail target detection branch while no longer updating the common target detection branch to obtain an enhanced target detection model; and
[0026] Target recognition step: Use the enhanced target detection model to detect and recognize common targets and long-tail targets. Among them, the common targets are detected and recognized through the common target detection branch in the enhanced target detection model, and the long-tail targets are detected and recognized through the long-tail target detection branch in the enhanced target detection model.
[0027] According to the third aspect of the present invention, there is provided a method for recognizing long-tail targets for a vehicle, including the following steps:
[0028] Common semantic segmentation model training step: Use training data containing common target data to train a common semantic segmentation model including a backbone network and a common target segmentation branch. Among them, the common target data includes common target images and corresponding segmentation annotations. Among them, the common target images are images collected by the vehicle's on-vehicle camera or camera images from an open-source database;
[0029] Steps for adding a long-tail object segmentation branch: Add an additional long-tail object segmentation branch for detecting long-tail objects to the trained common semantic segmentation model;
[0030] Steps for adding augmented data: Generate simulated long-tail object data by adding augmented data to the training data, where the augmented data is any camera image or an image generated by a predetermined program;
[0031] Steps for training an enhanced semantic segmentation model: Train the long-tail object segmentation branch based on the generated simulated long-tail object data while no longer updating the common object segmentation branch to obtain an enhanced semantic segmentation model; and
[0032] Steps for object recognition: Detect and recognize common objects and long-tail objects using the enhanced semantic segmentation model, where the common objects are detected and recognized through the common object segmentation branch in the enhanced semantic segmentation model, and the long-tail objects are detected and recognized through the long-tail object segmentation branch in the enhanced semantic segmentation model.
[0033] According to the fourth aspect of the present invention, a long-tail object recognition method for a vehicle is provided, including the following steps:
[0034] Steps for training a common object detection model: Train a common object detection model including a backbone network and a common object detection branch using training data containing common object data, where the common object data includes common object point cloud images and corresponding object detection annotations, and the common object point cloud images are collected by the vehicle's on-vehicle lidar or millimeter-wave radar or obtained from an open-source database;
[0035] Steps for adding a long-tail object detection branch: Add an additional long-tail object detection branch for detecting long-tail objects to the trained common object detection model;
[0036] Steps for adding augmented data: Generate simulated long-tail object data by adding augmented data to the training data, where the augmented data is point cloud image data, collected by the vehicle's on-vehicle lidar or millimeter-wave radar or obtained from an open-source database;
[0037] Steps for training an enhanced object detection model: Train the long-tail object detection branch based on the generated simulated long-tail object data while no longer updating the common object detection branch to obtain an enhanced object detection model; and
[0038] Target recognition step: Detect and recognize common targets and long-tail targets using the enhanced target detection model. Among them, the common targets are detected and recognized through the common target detection branch in the enhanced target detection model, and the long-tail targets are detected and recognized through the long-tail target detection branch in the enhanced target detection model.
[0039] According to the fifth aspect of the present invention, there is provided a long-tail target recognition method for a vehicle, including the following steps:
[0040] Common semantic segmentation model training step: Use training data containing common target data to train a common semantic segmentation model including a backbone network and a common target segmentation branch. Among them, the common target data includes common target point cloud images and corresponding segmentation annotations. The common target point cloud images are collected by the vehicle's on-vehicle lidar or millimeter-wave radar or obtained from an open-source database.
[0041] Long-tail target segmentation branch adding step: Add an additional long-tail target segmentation branch for detecting long-tail targets to the trained common semantic segmentation model.
[0042] Augmented data adding step: Generate simulated long-tail target data by adding augmented data to the training data. Among them, the point cloud augmented data is point cloud image data, which is collected by the vehicle's on-vehicle lidar or millimeter-wave radar or obtained from an open-source database.
[0043] Enhanced semantic segmentation model training step: Train the long-tail target segmentation branch based on the generated simulated long-tail target data without updating the common target segmentation branch to obtain an enhanced semantic segmentation model; and
[0044] Target recognition step: Detect and recognize common targets and long-tail targets using the enhanced semantic segmentation model. Among them, the common targets are detected and recognized through the common target segmentation branch in the enhanced semantic segmentation model, and the long-tail targets are detected and recognized through the long-tail target segmentation branch in the enhanced semantic segmentation model.
[0045] According to the sixth aspect of the present invention, there is provided a driver attention enhancement method, which includes:
[0046] Long-tail target recognition step: Use the long-tail target recognition method of the present invention described above to recognize long-tail targets;
[0047] Dangerous area determination step: Determine the area where the long-tail target is located as a dangerous area; and
[0048] Steps to enhance driver attention: Use AR technology to highlight the dangerous area on the vehicle's windshield to enhance the driver's attention to it.
[0049] Preferably, the steps to enhance driver attention further include: enhancing the driver's attention to the dangerous area by flashing the lights in the cab and / or emitting a reminder sound through the speaker in the cab and / or vibrating the driver's seat.
[0050] According to the seventh aspect of the present invention, there is provided a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, the long-tail target recognition method and / or the driver attention enhancement method of the present invention are implemented.
[0051] According to the eighth aspect of the present invention, there is provided a vehicle, characterized by including a memory and a processor, in which a computer program is stored, and when the computer program is executed by the processor, the long-tail target recognition method and / or the driver attention enhancement method of the present invention are implemented.
[0052] Compared with the prior art, the long-tail target recognition method and the driver attention enhancement method of the present invention have achieved the following beneficial technical effects:
[0053] 1. The long-tail target recognition method of the present invention does not require collecting and annotating a large amount of long-tail target data.
[0054] 2. When training the target model, since only the long-tail target recognition branch / long-tail target detection branch / long-tail target segmentation branch needs to be trained, a satisfactory enhanced performance of the target model can be obtained with only a small amount of computing resources.
[0055] 3. The enhanced target model / enhanced target detection model / enhanced semantic segmentation model of the present invention can learn the classification boundary between common targets and long-tail targets in the training data, and can detect and identify the common targets through the common target recognition branch / common target detection branch / common target segmentation branch in the enhanced target model / enhanced target detection model / enhanced semantic segmentation model and detect and identify the long-tail targets through the long-tail target recognition branch / long-tail target detection branch / long-tail target segmentation branch in the enhanced target model / enhanced target detection model / enhanced semantic segmentation model. Therefore, the long-tail target recognition method of the present invention has good robustness, can solve most long-tail target recognition problems, and does not require enumerating all long-tail target data.
[0056] 4. The long-tail target recognition method of the present invention has a wide range of applications and can be applied to one or a combination of various sensors such as vehicle-mounted cameras and lidar.
[0057] 5. The driver attention enhancement method of the present invention can not only identify the vast majority of long-tail targets, but also significantly improve driving and traffic safety. [Description of the Drawings]
[0058] Figure 1 It is a flowchart of the long-tail target recognition method of the present invention.
[0059] Figure 2 It is a flowchart of the long-tail target recognition method according to the first embodiment of the present invention.
[0060] Figure 3 It is a flowchart of the long-tail target recognition method according to the second embodiment of the present invention.
[0061] Figures 4A - 4H It is a schematic diagram of exemplary training data, augmented data, long-tail target data, and target recognition results used in the long-tail target recognition method according to the first and second embodiments of the present invention.
[0062] Figure 5 It is a flowchart of the augmented data addition step in the long-tail target recognition method according to the third embodiment of the present invention.
[0063] Figures 6A - 6D It is a schematic diagram of exemplary training data, augmented data, and long-tail target data used in the long-tail target recognition method according to the third embodiment of the present invention.
[0064] Figure 7 It is a flowchart of the augmented data addition step in the long-tail target recognition method according to the fourth embodiment of the present invention.
[0065] Figure 8 It is a flowchart of the long-tail target recognition method according to the fifth embodiment of the present invention.
[0066] Figure 9 It is a flowchart of the long-tail target recognition method according to the sixth embodiment of the present invention.
[0067] Figures 10A - 10G It is a schematic diagram of exemplary training data, augmented data, long-tail target data, and target recognition results used in the long-tail target recognition method according to the fifth and sixth embodiments of the present invention.
[0068] Figure 11 It is a flowchart of the driver attention enhancement method according to the seventh embodiment of the present invention. [Detailed Description of the Invention]
[0069] The present invention will be further described below in conjunction with the specific embodiments with reference to the accompanying drawings.
[0070] In the present invention, each embodiment is only intended to illustrate the solution of the present invention and should not be construed as restrictive.
[0071] In the present invention, unless otherwise specified, the quantifiers "a" and "one" do not exclude the scenario of multiple elements.
[0072] It should also be noted here that in the embodiments of the present invention, for the sake of clarity and simplicity, only a part of the components or assemblies may be shown. However, those of ordinary skill in the art can understand that, under the teaching of the present invention, the required components or assemblies can be added according to the specific scenario requirements. Additionally, unless otherwise stated, the features in different embodiments of the present invention can be combined with each other. For example, a certain feature in the second embodiment can be used to replace the corresponding or functionally identical or similar feature in the first embodiment, and the resulting embodiment also falls within the scope of the disclosure or the scope of the record of this application.
[0073] It should also be noted here that within the scope of the present invention, the terms "same", "equal", "equal to", etc. do not mean that the two values are absolutely equal, but allow a certain reasonable error. That is to say, these terms also cover "substantially the same", "substantially equal", "substantially equal to". By analogy, in the present invention, the directional terms "perpendicular to", "parallel to", etc. also cover the meanings of "substantially perpendicular to" and "substantially parallel to".
[0074] In addition, the numbering of the steps of each method of the present invention does not limit the execution order of the method steps. Unless otherwise specified, the method steps can be executed in different orders.
[0075] The following further elaborates on the long-tail target recognition method and the driver attention enhancement method proposed by the present invention in conjunction with the accompanying drawings and specific embodiments. According to the following description, the advantages and features of the present invention will be clearer. It should be noted that the accompanying drawings are all in a very simplified form and use non-precise scales, only for the purpose of facilitating and clearly assisting in explaining the purpose of the embodiments of the present invention.
[0076] As Figure 1 shown, a long-tail target recognition method 100 of the present invention includes the following steps:
[0077] Common target model training step 102: Use training data containing common target data to train a common target model including a backbone network and a common target recognition branch;
[0078] Long-tail target recognition branch addition step 104: Add an additional long-tail target recognition branch for recognizing long-tail targets to the backbone network in the trained common target model;
[0079] Augmented data addition step 106: Generate simulated long-tail target data by adding augmented data to the training data;
[0080] Enhanced target model training step 108: Train the long-tail target recognition branch based on the generated simulated long-tail target data while no longer updating the common target recognition branch to obtain an enhanced target model; and
[0081] Target recognition step 110: Detect and recognize common targets and long-tail targets using the enhanced target model, wherein the common targets are detected and recognized through the common target recognition branch in the enhanced target model, and the long-tail targets are detected and recognized through the long-tail target recognition branch in the enhanced target model.
[0082] Preferably, in the enhanced target model training step 108, the long-tail target recognition branch is trained based on the generated simulated long-tail target data and the trained common target recognition branch while no longer updating the common target recognition branch to obtain the enhanced target model.
[0083] Combined Figure 2 and Figures 4A - 4F , the long-tail target recognition method 100A for a vehicle according to the first embodiment of the present invention is specifically described. The long-tail target recognition method 100A includes the following steps:
[0084] Common target detection model training step 102A: Use training data containing common target data to train a common target detection model including a backbone network and a common target detection branch, wherein the common target data includes common target images and corresponding target detection annotations, and the common target images are images collected by the vehicle's on-board camera or from an open-source database. Figure 4A Exemplarily, a scene image of a training data is shown, which includes images of various traffic participants (such as vehicles, pedestrians, bicycles, motorcycles, traffic cones, crash barrels, etc.) as common targets. Among them, any one of common target detection models applicable to image data, such as CenterNet, YOLO, RetinaNet, SSD, FCOS, etc., can be used as the common target detection model.
[0085] Long-tail target detection branch addition step 104A: Add an additional long-tail target detection branch for detecting long-tail targets to the trained common target detection model.
[0086] Augmented data addition step 106A: Generate simulated long-tail target data by adding augmented data to the training data, wherein the augmented data is any camera image or an image generated by a predetermined program, Figure 4BAn overall camera image of a puppy playing with a ball on the grass is exemplarily shown, where the image of the puppy playing with the ball is a long-tail target image. Through its corresponding segmentation annotation, the image of the puppy playing with the ball is segmented from the overall camera image and taken out as augmented data. Figure 4C Exemplarily shown is that the augmented data (i.e., the image of the puppy playing with the ball) is pasted as a long-tail target image Figure 4A to a position in the scene image of the training data shown, and the generated simulated long-tail target data is generated. The generated simulated long-tail target data includes the long-tail target image (i.e., the image of the puppy playing with the ball) and the generated corresponding object detection annotation (not shown in the figure).
[0087] Enhanced object detection model training step 108A: Based on the generated simulated long-tail target data, train the long-tail target detection branch while no longer updating the common object detection branch to obtain an enhanced object detection model. And
[0088] Object recognition step 110A: Use the enhanced object detection model to detect and recognize common objects and long-tail objects. Among them, the common object detection branch in the enhanced object detection model is used to detect and recognize common objects, and the long-tail object detection branch in the enhanced object detection model is used to detect and recognize long-tail objects. Figure 4D Exemplarily shown is a scene image to be detected and recognized (including a vehicle, a traffic sign, and a puppy). As Figure 4E Exemplarily shown, the common object detection branch in the enhanced object detection model will Figure 4D the vehicle and the traffic sign in it are recognized as common objects (i.e., marked as common objects with a white frame). As Figure 4F Exemplarily shown, the long-tail object detection branch in the enhanced object detection model will Figure 4D the puppy in it is recognized as a long-tail object (i.e., marked as a long-tail object with a gray frame).
[0089] In a preferred case, in the enhanced object detection model training step 108A, based on the generated multiple simulated long-tail target data and the trained common object detection branch, train the long-tail object detection branch while no longer updating the common object detection branch to obtain an enhanced object detection model.
[0090] Combined with Figure 3 and Figures 4A - 4D and 4G-4H, the long-tail object recognition method 100B for vehicles according to the second embodiment of the present invention is specifically described. The long-tail object recognition method 100B includes the following steps:
[0091] Common semantic segmentation model training step 102B: Use training data containing common target data to train a common semantic segmentation model including a backbone network and a common target segmentation branch. Among them, the common target data includes common target images and corresponding segmentation annotations. Among them, the common target images are images collected by the vehicle's on-vehicle camera or from an open-source database. Figure 4A Exemplarily, a scene image of a training data is shown, which includes images of various traffic participants (such as vehicles, pedestrians, bicycles, motorcycles, traffic cones, crash barrels, etc.) as common targets. Among them, any one of common semantic segmentation models applicable to image data, such as FCN, DeepLab, SegNet, etc., can be used as the common semantic segmentation model.
[0092] Long-tail target segmentation branch adding step 104B: Add an additional long-tail target segmentation branch for detecting long-tail targets to the trained common semantic segmentation model.
[0093] Augmented data adding step 106B: Generate simulated long-tail target data by adding augmented data to the training data. Among them, the augmented data is any camera image or an image generated by a predetermined program. Figure 4B Exemplarily, an overall camera image of a puppy playing with a ball on the grass is shown. Among them, the image of the puppy playing with the ball is a long-tail target image. Through its corresponding segmentation annotation, the image of the puppy playing with the ball is segmented and taken out from the overall image as augmented data. Figure 4C Exemplarily, the augmented data (i.e., the image of the puppy playing with the ball) is pasted as a long-tail target image to Figure 4A a position in the scene image of the shown training data to generate simulated long-tail target data. The simulated long-tail target data includes the long-tail target image (i.e., the image of the puppy playing with the ball) and the generated corresponding semantic segmentation annotation (not shown in the figure);
[0094] Enhanced semantic segmentation model training step 108B: Train the long-tail target segmentation branch based on the generated simulated long-tail target data while no longer updating the common target segmentation branch to obtain an enhanced semantic segmentation model; and
[0095] Target recognition step 110B: Use the enhanced semantic segmentation model to detect and recognize common targets and long-tail targets. Among them, the common target segmentation branch in the enhanced semantic segmentation model is used to detect and recognize common targets, and the long-tail target segmentation branch in the enhanced semantic segmentation model is used to detect and recognize long-tail targets. Figure 4D Exemplarily, a scene image to be detected and recognized (including a vehicle, a traffic sign, and a puppy) is shown. As Figure 4GExemplarily shown, the common object segmentation branch in the enhanced semantic segmentation model will Figure 4D the vehicles and traffic signs in Figure 4H be recognized as common objects. As Figure 4D exemplarily shown, the long-tail object segmentation branch of the enhanced semantic segmentation model will
[0096] In a preferred case, in the enhanced semantic segmentation model training step 108B, based on the generated simulated long-tail object data and the trained common object segmentation branch, the long-tail object segmentation branch is trained while the common object segmentation branch is no longer updated, so as to obtain an enhanced semantic segmentation model.
[0097] Combined with Figure 5 and Figures 6A - 6D , a long-tail object recognition method 100C for vehicles according to the third embodiment of the present invention is described. The long-tail object recognition method 100C is mostly the same as the above long-tail object recognition method 100A or 100B. The difference is that the augmented data addition step 106C in the long-tail object recognition method 100C includes:
[0098] Target image generation step 1062C: Generate a plurality of target images with different shapes through a preset program. Among them, the preset program can be a self-designed image generation program or generate a closed curve by using a Bezier curve or a B-spline curve, etc., and generate a target image by filling the inside of the closed curve. Figure 6B Exemplarily shown is the target image generated after filling the closed curve with black.
[0099] Augmented data generation step 1064C: Through the preset program, fill each of the plurality of target images with arbitrary content as the augmented data, where the arbitrary content is a random color or the background content segmented from the training data. Figure 6C Exemplarily shown is filling with gray Figure 6B the target image shown in Figure 6C to generate the augmented data (i.e.,
[0100] the figure filled with gray in Figure 6A Exemplarily shown is a scene image of training data, which includes multiple vehicles. Figure 6D Exemplarily shown is pasting Figure 6C the augmented data shown in Figure 6CThe figure filled with gray) is pasted as the long-tail target image into Figure 6A the scene image of the training data shown, and the generated simulated long-tail target data includes the long-tail target image (i.e., Figure 6D the figure filled with gray in ) and the generated corresponding object detection annotation or semantic segmentation annotation (not shown in the figure).
[0101] As Figure 7 shown, the long-tail target recognition method 100D for vehicles according to the fourth embodiment of the present invention is described. The long-tail target recognition method 100D is mostly the same as the above long-tail target recognition method 100A or 100B, except that the augmented data adding step 106D in the long-tail target recognition method 100D includes:
[0102] Augmented data generation step 1062D: Search for and copy the long-tail target image in the long-tail scene image from an open-source database as the augmented data. Among them, the open-source database can adopt any one of public scene target databases such as the COCO dataset and ImageNet, and the augmented data (i.e., the long-tail target image) is the remaining target image after removing the target types annotated in the common target data; and
[0103] Long-tail target data generation step 1064D: Paste the augmented data to any position in the training data to generate the long-tail target data, where the number of the any positions is the same as the number of the augmented data.
[0104] Combined with Figure 8 and Figures 10A - 10E , the long-tail target recognition method 100E for vehicles according to the fifth embodiment of the present invention is described. The long-tail target recognition method 100E includes the following steps:
[0105] Common object detection model training step 102E: Use the training data containing common target data to train a common object detection model including a backbone network and a common object detection branch, where the common target data includes common target point cloud images and corresponding object detection annotations, and the common target point cloud images are collected by the vehicle's on-vehicle lidar or millimeter-wave radar (such as a 4D millimeter-wave radar) or obtained from an open-source database. Figure 10A Exemplarily, a point cloud scene image of a training data is shown, including images of various traffic participants (such as vehicles, pedestrians, bicycles, motorcycles, traffic cones, etc.) as common targets. Among them, any one of common object detection models applicable to point cloud image data such as VoxelNet, SECOND, and PointPillar can be used as the common object detection model;
[0106] Long-tail object detection branch adding step 104E: Add an additional long-tail object detection branch for detecting long-tail objects to the trained common object detection model;
[0107] Augmented data adding step 106E: Generate simulated long-tail object data by adding augmented data to the training data, where the augmented data is point cloud image data, collected by the vehicle's on-board lidar or millimeter-wave radar (such as 4D millimeter-wave radar) or obtained from an open-source database. 10B exemplarily shows that Figure 10A augmented data (i.e., Figure 10B the second white figure from left to right in Figure 10B corresponding to an actual object being a football) is added to the point cloud scene image of the shown training data as a long-tail object point cloud image to generate simulated long-tail object data. The generated simulated long-tail object data includes the long-tail object point cloud image (i.e.,
[0108] the second white figure from left to right in
[0109] and the corresponding generated object detection annotation (not shown in the figure); Figure 10C Exemplarily shows a scene image to be detected and recognized, where the second white figure from left to right corresponds to an actual object being a football, and the remaining multiple white figures respectively correspond to actual objects such as vehicles, pedestrians, bicycles, motorcycles, traffic cones, etc. As Figure 10D exemplarily shown, the common object detection branch in the enhanced object detection model will Figure 10C the vehicles, pedestrians, bicycles, motorcycles, traffic cones, etc. in Figure 10D be recognized as common objects (i.e., Figure 10E the multiple figures with white frames in Figure 10C ). As Figure 10E exemplarily shown, the long-tail object detection branch of the enhanced object detection model will
[0110] In a preferred case, in the enhanced target detection model training step 108E, based on the generated simulated long-tail target data and the trained common target detection branch, the long-tail target detection branch is trained while the common target detection branch is no longer updated, so as to obtain an enhanced target detection model.
[0111] Combined with Figure 9 and Figure 10A -C and Figures 10F - 10G , the long-tail target recognition method 100F for a vehicle according to the sixth embodiment of the present invention is described. The long-tail target recognition method 100F includes the following steps:
[0112] Common semantic segmentation model training step 102F: Using training data containing common target data to train a common semantic segmentation model including a backbone network and a common target segmentation branch, wherein the common target data includes common target point cloud images and corresponding segmentation annotations, and the common target point cloud images are collected by the vehicle's on-vehicle lidar or millimeter-wave radar (such as 4D millimeter-wave radar) or obtained from an open-source database. Figure 10A Exemplarily, a point cloud scene image of a training data is shown, including images of various traffic participants (such as vehicles, pedestrians, bicycles, motorcycles, traffic cones, etc.) as common targets. Among them, any one of common semantic segmentation models applicable to point cloud image data, such as FCPN, LiDARSeg, SqueezeSeg, etc., can be used as the common semantic segmentation model;
[0113] Long-tail target segmentation branch adding step 104F: Adding an additional long-tail target segmentation branch for detecting long-tail targets to the trained common semantic segmentation model;
[0114] Augmented data adding step 106F: Generating simulated long-tail target data by adding augmented data to the training data, wherein the augmented data is point cloud image data, collected by the vehicle's on-vehicle lidar or millimeter-wave radar (such as 4D millimeter-wave radar) or obtained from an open-source database. 10B Exemplarily shows that in Figure 10A the point cloud scene image of the shown training data, augmented data (i.e., Figure 10B the second white figure from left to right in Figure 10B , and the actual target corresponding to it is a football) is added as a long-tail target point cloud image to generate simulated long-tail target data. The generated simulated long-tail target data includes the long-tail target point cloud image (i.e.,
[0115] Enhanced semantic segmentation model training step 108F: Train the long-tail target segmentation branch based on the generated simulated long-tail target data while no longer updating the common target segmentation branch to obtain an enhanced semantic segmentation model; and
[0116] Target recognition step 110F: Detect and recognize common targets and long-tail targets using the enhanced semantic segmentation model, wherein the common targets are detected and recognized through the common target segmentation branch in the enhanced semantic segmentation model, and the long-tail targets are detected and recognized through the long-tail target segmentation branch in the enhanced semantic segmentation model. Figure 10C Exemplarily, a scene image to be detected and recognized is shown, where the second white figure from left to right corresponds to an actual target of a football, and the remaining multiple white figures respectively correspond to actual targets such as vehicles, pedestrians, bicycles, motorcycles, traffic cones, etc. As Figure 10F Exemplarily shown, the common target segmentation branch in the enhanced semantic segmentation model will Figure 10C the vehicles, pedestrians, bicycles, motorcycles, traffic cones, etc. in (i.e., Figure 10C the multiple white figures in except the second white figure from left to right) are recognized as common targets. As Figure 10G Exemplarily shown, the long-tail target segmentation branch of the enhanced semantic segmentation model will Figure 10C the football in (i.e., Figure 10G one of the white figures in) is recognized as a long-tail target.
[0117] In a preferred case, in the enhanced target detection model training step 108F, the long-tail target segmentation branch is trained based on the generated multiple simulated long-tail target data and the trained common target segmentation branch while no longer updating the common target segmentation branch to obtain an enhanced semantic segmentation model
[0118] Those skilled in the art can understand that for different detection tasks, the types of training data used for the augmented data are different. For example, when the detection task is moving target detection or lane line detection, etc., the training data corresponding to the augmented data are moving target (vehicles, pedestrians, etc.) data or lane line data respectively.
[0119] As Figure 11 shown, the vehicle driver attention enhancement method 200 according to the seventh embodiment of the present invention includes the following steps:
[0120] Long-tail target recognition step 202: Use any one of the above long-tail target recognition methods 100A - 100F of the present invention to recognize long-tail targets;
[0121] Hazard area determination step 204: Determine the area where the long-tail target is located as a hazard area (in this embodiment, the hazard area range is the drivable road area visible in the field of view in the vehicle traveling direction); and
[0122] Driver attention enhancement step 206: Use AR technology to prominently display the hazard area on the vehicle's windshield to enhance the driver's attention to it.
[0123] In a variant of the seventh embodiment (not shown), the driver attention enhancement step 206 further includes: enhancing the driver's attention to the hazard area by flashing the lights in the cab and / or emitting a reminder sound through the in-cab speaker and / or vibrating the driver's seat.
[0124] The above embodiments have described in detail different configurations of the long-tail target recognition method of the present invention. Of course, the present invention includes but is not limited to the configurations listed in the above embodiments. Any content obtained by transformation based on the configurations provided in the above embodiments belongs to the scope protected by the present invention. Those skilled in the art can draw inferences by analogy based on the content of the above embodiments.
[0125] Each embodiment in this specification is described in a progressive manner. The key point of each embodiment is to illustrate the differences from other embodiments. The same or similar parts among the embodiments can be referred to each other.
[0126] The above description is only a description of the preferred embodiments of the present invention and does not limit the scope of the present invention in any way. Any changes and modifications made by those of ordinary skill in the art of the present invention based on the above disclosure belong to the scope protected by the claims.
Claims
1. A long-tail object recognition method, comprising the following steps: Common object model training step: Using training data containing common object data to train a common object model including a backbone network and a common object recognition branch; Long-tail object recognition branch adding step: Adding an additional long-tail object recognition branch for recognizing long-tail objects to the backbone network in the trained common object model; Augmented data adding step: Generating simulated long-tail object data by adding augmented data to the training data; Enhanced object model training step: Training the long-tail object recognition branch based on the generated simulated long-tail object data while no longer updating the common object recognition branch to obtain an enhanced object model; And Object recognition step: Using the enhanced object model to detect and recognize common objects and long-tail objects, wherein the common objects are detected and recognized through the common object recognition branch in the enhanced object model, and the long-tail objects are detected and recognized through the long-tail object recognition branch in the enhanced object model.
2. The long-tail object recognition method according to claim 1, Wherein, In the enhanced object model training step, the long-tail object recognition branch is trained based on the generated simulated long-tail object data and the trained common object recognition branch while no longer updating the common object recognition branch to obtain the enhanced object model.
3. A long-tail object recognition method for a vehicle, comprising the following steps: Common object detection model training step: Using training data containing common object data to train a common object detection model including a backbone network and a common object detection branch, Wherein, The common object data includes common object images and corresponding object detection annotations, wherein the common object images are images collected by an on-vehicle camera of the vehicle or images from an open-source database; Long-tail object detection branch adding step: Adding an additional long-tail object detection branch for detecting long-tail objects to the trained common object detection model; Augmented data adding step: Generating simulated long-tail object data by adding augmented data to the training data, wherein the augmented data is any camera image or an image generated by a predetermined program; Enhanced object detection model training step: Training the long-tail object detection branch based on the generated simulated long-tail object data while no longer updating the common object detection branch to obtain an enhanced object detection model; and Object recognition step: Using the enhanced object detection model to detect and recognize common objects and long-tail objects, wherein the common objects are detected and recognized through the common object detection branch in the enhanced object detection model, and the long-tail objects are detected and recognized through the long-tail object detection branch in the enhanced object detection model.
4. The long-tail object recognition method according to claim 3, Wherein, In the step of training the enhanced object detection model, the long-tail object detection branch is trained based on the generated simulated long-tail object data and the trained common object detection branch, while the common object detection branch is no longer updated, so as to obtain the enhanced object detection model.
5. A long-tail object recognition method for a vehicle, comprising the following steps: Step of training a common semantic segmentation model: Use training data containing common object data to train a common semantic segmentation model including a backbone network and a common object segmentation branch. Wherein, The common object data includes common object images and corresponding segmentation annotations, wherein the common object images are images collected by an on-vehicle camera of the vehicle or images from an open-source database. Step of adding a long-tail object segmentation branch: Add an additional long-tail object segmentation branch for detecting long-tail objects to the trained common semantic segmentation model. Step of adding augmented data: Generate simulated long-tail object data by adding augmented data to the training data, wherein the augmented data is any camera image or an image generated by a predetermined program. Step of training an enhanced semantic segmentation model: Train the long-tail object segmentation branch based on the generated simulated long-tail object data, while the common object segmentation branch is no longer updated, so as to obtain an enhanced semantic segmentation model. And Step of object recognition: Use the enhanced semantic segmentation model to detect and recognize common objects and long-tail objects, wherein the common objects are detected and recognized through the common object segmentation branch in the enhanced semantic segmentation model, and the long-tail objects are detected and recognized through the long-tail object segmentation branch in the enhanced semantic segmentation model.
6. The long-tail object recognition method according to claim 5, Wherein, In the step of training the enhanced semantic segmentation model, the long-tail object segmentation branch is trained based on the generated simulated long-tail object data and the trained common object segmentation branch, while the common object segmentation branch is no longer updated, so as to obtain the enhanced semantic segmentation model.
7. The long-tail object recognition method according to any one of claims 2-6, Wherein, The step of adding augmented data includes: Step of generating target images: Generate a plurality of target images with different shapes through a preset program. Step of generating augmented data: Through the preset program, fill each of the plurality of target images with any content as the augmented data, wherein the any content is random colors or background content segmented from the training data. And Step of generating long-tail object data: Paste the augmented data to any position selected in the training data to generate the simulated long-tail object data, wherein the number of the any positions is the same as the number of the augmented data.
8. The long-tail object recognition method according to any one of claims 2-6, Wherein, The step of adding augmented data includes: Augmented data generation step: Search for and copy the long-tail target images in the long-tail scenario images from the open-source database as the augmented data, where the augmented data is the target images remaining after removing the target types annotated in the common target data; and Long-tail target data generation step: Paste the augmented data at any position selected in the training data to generate the long-tail target data, where the number of the any positions is the same as the number of the augmented data.
9. A method for long-tail target recognition for a vehicle, comprising the following steps: Common target detection model training step: Use the training data containing common target data to train a common target detection model including a backbone network and a common target detection branch, wherein, the common target data includes common target point cloud images and corresponding target detection annotations, where the common target point cloud images are collected by the vehicle's on-vehicle lidar or millimeter-wave radar or obtained from an open-source database; Long-tail target detection branch addition step: Add an additional long-tail target detection branch for detecting long-tail targets to the trained common target detection model; Augmented data addition step: Generate simulated long-tail target data by adding augmented data to the training data, where the augmented data is point cloud image data, collected by the vehicle's on-vehicle lidar or millimeter-wave radar or obtained from an open-source database; Enhanced target detection model training step: Train the long-tail target detection branch based on the generated simulated long-tail target data while no longer updating the common target detection branch to obtain an enhanced target detection model; and Target recognition step: Use the enhanced target detection model to detect and recognize common targets and long-tail targets, where the common targets are detected and recognized through the common target detection branch in the enhanced target detection model, and the long-tail targets are detected and recognized through the long-tail target detection branch in the enhanced target detection model.
10. The long-tail target recognition method according to claim 9, wherein, in the enhanced target detection model training step, train the long-tail target detection branch based on the generated simulated long-tail target data and the trained common target detection branch while no longer updating the common target detection branch to obtain the enhanced target detection model.
11. A method for long-tail target recognition for a vehicle, comprising the following steps: Common semantic segmentation model training step: Use the training data containing common target data to train a common semantic segmentation model including a backbone network and a common target segmentation branch, wherein, the common target data includes common target point cloud images and corresponding segmentation annotations, where the common target point cloud images are collected by the vehicle's on-vehicle lidar or millimeter-wave radar or obtained from an open-source database; Long-tail target segmentation branch addition step: Add an additional long-tail target segmentation branch for detecting long-tail targets to the trained common semantic segmentation model; Augmented data addition step: Generate simulated long-tail target data by adding augmented data to the training data, where the point cloud augmented data is point cloud image data, which is collected by the vehicle's on-vehicle lidar or millimeter-wave radar or obtained from an open-source database; Enhanced semantic segmentation model training step: Train the long-tail target segmentation branch based on the generated simulated long-tail target data while no longer updating the common target segmentation branch to obtain an enhanced semantic segmentation model; and Target recognition step: Use the enhanced semantic segmentation model to detect and recognize common targets and long-tail targets, where the common targets are detected and recognized through the common target segmentation branch in the enhanced semantic segmentation model, and the long-tail targets are detected and recognized through the long-tail target segmentation branch in the enhanced semantic segmentation model.
12. The long-tail target recognition method according to claim 11, wherein, In the enhanced semantic segmentation model training step, train the long-tail target segmentation branch based on the generated simulated long-tail target data and the trained common target segmentation branch while no longer updating the common target segmentation branch to obtain the enhanced semantic segmentation model.
13. A method for enhancing a vehicle driver's attention, comprising the following steps: Long-tail target recognition step: Use the long-tail target recognition method according to any one of claims 1-12 to recognize long-tail targets; Hazardous area determination step: Determine the area where the long-tail target is located as a hazardous area; and Driver attention enhancement step: Use AR technology to prominently display the hazardous area on the vehicle's windshield to enhance the driver's attention to it.
14. The method according to claim 13, wherein, The driver attention enhancement step further includes: enhancing the driver's attention to the hazardous area by flashing the lights in the cab and / or emitting a reminder sound through the speaker in the cab and / or vibrating the driver's seat.
15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it realizes using the long-tail target recognition method according to any one of claims 1-12 and / or the driver attention enhancement method according to any one of claims 13-14.
16. A vehicle, characterized in that, It includes a memory and a processor, and the memory stores a computer program, and when the computer program is executed by the processor, it realizes using the long-tail target recognition method according to any one of claims 1-12 and / or the driver attention enhancement method according to any one of claims 13-14.