Target detection and segmentation method and device for multi-task model, electronic equipment and storage medium
By adopting a multi-task model in the automatic lane keeping technology, using semantic segmentation results to correct the detection results, and recalling missed detection targets, the problem of insufficient generalization capabilities of deep learning models is solved, and the accuracy and speed of object detection and segmentation are improved.
Patent Information
- Application Number
- CN202411466271.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-21
- Publication Date
- 2025-02-11
AI Technical Summary
In the existing automatic lane keeping technology, the generalization ability of the deep learning model is insufficient, resulting in the inability to detect all targets in the visual area, and post-processing is required to filter and recall the targets.
The multi-task model is used to correct the detection results using the results of semantic segmentation, and the semantic segmentation results are recalled by the detection results to optimize the effect of object detection and segmentation. The specific steps include outputting detection and segmentation results based on the pre-trained model, correcting the location and type of the detection box through the refine model, and correcting the category of point cloud classification, and finally recalling possible missed detection targets.
By improving the feature extraction method by deep learning models, the detection effect of special small targets is improved, and the detection speed is improved through GPU acceleration, meeting the real-time detection requirements.
Smart Images
Figure CN119418324A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of target detection, and in particular to a target detection and segmentation method, device, electronic device, and storage medium of a multi-task model. Background Art
[0002] Intelligent connected vehicles are adding more and more driver assistance features, one of which is automatic lane keeping. Automatic lane keeping allows the car to be correctly positioned within the road lane, which is also crucial for completing subsequent lane departure or trajectory planning decisions for autonomous vehicles.
[0003] In related technologies, the target detection method used in automatic lane keeping relies on a deep learning algorithm. Due to the generalization problem of the deep learning model, it cannot detect all targets in the visible area. Therefore, effective post-processing is required to filter and recall the targets. Summary of the invention
[0004] The embodiments of the present application provide a method, device, electronic device, and storage medium for target detection and segmentation based on a multi-task model. The detection results are corrected using the results of semantic segmentation based on the multi-task model, and the semantic segmentation results are recalled using the detection results to optimize the effect of target detection and segmentation.
[0005] The present application embodiment adopts the following technical solutions:
[0006] In a first aspect, an embodiment of the present application provides a method for object detection and segmentation of a multi-task model, wherein the method comprises:
[0007] Outputting target detection results and target semantic segmentation results according to the pre-trained target detection and segmentation model, wherein the target detection results include target detection boxes, and the target semantic segmentation results include point cloud classification of the target;
[0008] Correcting the position and size of the target detection frame in the target detection result according to the refine model, and re-predicting the type of the target detection frame;
[0009] According to the re-predicted target detection frame, the category of the point cloud classification is corrected to the category of the target detection frame;
[0010] According to the category correction result of the point cloud classification, the target detection frames that may have been missed are recalled to obtain the final target detection and segmentation results.
[0011] In some embodiments, the method of revising the category of the point cloud classification and recalling the target detection frames that may have been missed to obtain the final target detection and segmentation results includes:
[0012] Determine the target detection boxes that may have missed detections;
[0013] According to the category correction result of the point cloud classification, a corrected target semantic segmentation result is obtained;
[0014] The target detection frames that may have missed detection are recalled according to the corrected target semantic segmentation result to obtain the final target detection and segmentation result.
[0015] In some embodiments, the process of revising the category of the point cloud classification to retrieve the target detection frames that may have been missed to obtain the final target detection and segmentation results further includes:
[0016] According to the corrected target semantic segmentation result, for point cloud data of a preset category, the corrected target detection frame is used to filter out the point cloud of the preset category to obtain a point cloud that does not belong to the target detection frame;
[0017] The point cloud that does not belong to the target detection frame is clustered using a clustering algorithm to obtain usable point cloud clusters.
[0018] In some embodiments, the method further comprises:
[0019] The available point cloud clusters are re-input into the refine model to obtain the position and size of the target detection frame after regression of each point cloud cluster and the category of the corresponding target detection frame.
[0020] In some embodiments, the correcting the position and size of the target detection frame in the target detection result according to the refine model and re-predicting the type of the target detection frame includes:
[0021] According to the target detection box, the refine model is used to regress the position and size of the target detection box in the target detection result, and the type of the target detection box BBOX is re-predicted.
[0022] In some embodiments, the correcting the category of the point cloud classification according to the re-predicted target detection frame to correct the category of the point cloud classification to the category of the target detection frame includes:
[0023] The type of the target detection box is re-predicted according to the refine model, and the point cloud classification is corrected to the category of the detection box according to the confidence score of the target detection box.
[0024] In a second aspect, an embodiment of the present application further provides a target detection method, wherein the target detection and segmentation method of the multi-task model described in the first aspect is adopted, and the target detection method includes:
[0025] The final target detection and segmentation result is used as the target detection result.
[0026] In a third aspect, an embodiment of the present application further provides a target detection and segmentation device of a multi-task model, wherein the device comprises:
[0027] A result output module, used to output a target detection result and a target semantic segmentation result according to a pre-trained target detection and segmentation model, wherein the target detection result includes a target detection box, and the target semantic segmentation result includes a point cloud classification of the target;
[0028] A first correction module, used to correct the position and size of the target detection frame in the target detection result according to a refine model, and re-predict the type of the target detection frame;
[0029] A second correction module is used to correct the category of the point cloud classification according to the re-predicted target detection frame, so as to correct the category of the point cloud classification to the category of the target detection frame;
[0030] The recall module is used to recall the target detection frames that may have been missed according to the category correction result of the point cloud classification, so as to obtain the final target detection and segmentation results.
[0031] In a fourth aspect, an embodiment of the present application further provides an electronic device, comprising: a processor; and a memory arranged to store computer executable instructions, wherein the executable instructions, when executed, cause the processor to perform the above method.
[0032] In a fifth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores one or more programs. When the one or more programs are executed by an electronic device including multiple application programs, the electronic device executes the above method.
[0033] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects: first, according to the pre-trained target detection and segmentation model, the target detection result and the target semantic segmentation result are output, wherein the target detection result includes a target detection frame and the target semantic segmentation result includes a point cloud classification of the target. Then, the position and size of the target detection frame in the target detection result are corrected according to the refine model, and the type of the target detection frame is re-predicted, and further according to the re-predicted target detection frame, the category of the point cloud classification is corrected to correct the category of the point cloud classification to the category of the target detection frame, and finally, according to the category correction result of the point cloud classification, the target detection frame that may be missed is recalled to obtain the final target detection and segmentation results. The target detection and segmentation using the above method improves the detection effect of some special small targets by modifying the feature extraction method of the deep learning model. In addition, the detection speed can be improved by GPU acceleration to meet the real-time detection requirements. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0035] Figure 1 Schematic diagram of the process of target detection and segmentation method of the multi-task model in the embodiment of the present application;
[0036] Figure 2 This is a schematic diagram of the structure of a target detection and segmentation device of a multi-task model in an embodiment of the present application;
[0037] Figure 3 This is a schematic diagram of the structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION
[0038] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.
[0039] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.
[0040] The present application embodiment provides a target detection and segmentation method of a multi-task model, such as Figure 1As shown, a flow chart of a target detection and segmentation method of a multi-task model in an embodiment of the present application is provided, and the method at least includes the following steps S110 to S140:
[0041] Step S110, outputting a target detection result and a target semantic segmentation result according to a pre-trained target detection and segmentation model, wherein the target detection result includes a target detection box, and the target semantic segmentation result includes a point cloud classification of the target.
[0042] The above method is mainly executed by the vehicle side. In the intelligent connected car, the vehicle-mounted laser radar has been installed, and the model parameters have been adjusted to obtain normal lane information in real time. The vehicle has a similar Xavier vehicle-mounted machine that can run the deep learning model. Through the designed neural network, a deep learning model can be trained that can correctly detect and classify targets and output semantic segmentation prediction results.
[0043] The deep learning model is a pre-trained target detection and segmentation model, which outputs target detection results and target semantic segmentation results. It can be understood that the target detection result at least includes a target detection box, which can detect and determine the location of the target. The target semantic segmentation result at least includes the point cloud classification of the target. Based on the laser radar, a 3D laser point cloud can be obtained and the 3D laser point cloud can be classified to obtain the point cloud classification of the target.
[0044] Step S120, correcting the position and size of the target detection frame in the target detection result according to the refine model, and re-predicting the type of the target detection frame.
[0045] A refine model usually refers to the part of a machine learning model that is used to refine and optimize the original output. This model usually corresponds to a base model, which is responsible for generating a preliminary result, while the base model is responsible for improving the quality and accuracy of these results. The refine model has a wide range of application scenarios. Those skilled in the art can use the refine model for tasks including but not limited to image processing, natural language processing, and more complex tasks such as 3D detection and tracking.
[0046] Specifically, the location and size of the target detection box in the target detection result are corrected according to the refine model. The location of the target detection box is the location of the target that needs to be located. The size of the target detection box includes the length and width of the 2D target detection box and the length, width and height of the 3Dbox target box. After correction, the type of the target detection box is re-predicted.
[0047] Step S130, correcting the category of the point cloud classification according to the re-predicted target detection frame, so as to correct the category of the point cloud classification to the category of the target detection frame.
[0048] According to the above steps, the re-predicted target detection frame can be obtained, and the re-predicted target detection frame is used to correct the category of the point cloud classification. The purpose of correcting the category of the point cloud classification is to correct the category of the point cloud classification to the category of the target detection frame, that is, the category of the target detection frame is more accurate, so the category of the point cloud classification is corrected to the category of the target detection frame. And generally speaking, the target semantic segmentation results based on the lidar point cloud are more than the target detection frame.
[0049] Step S140 , based on the category correction result of the point cloud classification, recall the target detection frames that may have been missed, and obtain the final target detection and segmentation results.
[0050] When recalling the detection frame using semantic segmentation, the target detection frame that may have been missed is recalled to obtain the final target detection and segmentation results. The category correction result of point cloud classification can obtain a more accurate category classification result. For small targets, based on the category correction result of the point cloud classification, the target detection frame that may have been missed is recalled to obtain the final target detection and segmentation results, which can improve the accuracy of the target detection process.
[0051] It can be understood that the above-mentioned step S140 and step S130 can be executed sequentially or in parallel according to actual conditions, and are not specifically limited in the embodiments of the present application.
[0052] Through the above method, the target detection method based on detection and segmentation of the perceptual multi-task model adopts a pre-trained target detection model and a target semantic segmentation model to output the target detection result and the target semantic segmentation result, and then corrects the position and size of the target detection frame in the target detection result according to the refine model, and re-predicts the type of the target detection frame, and then corrects the category of the point cloud classification according to the re-predicted target detection frame, so as to correct the category of the point cloud classification to the category of the target detection frame, which can improve the accuracy of semantic segmentation. According to the category correction result of the point cloud classification, the target detection frame that may be missed is recalled, and the category of the point is corrected using the re-predicted target frame, which solves the problem of missed detection of the detection frame.
[0053] Different from the related technologies, the target detection method relies on deep learning algorithms. Due to the generalization ability of deep learning models, it is not possible to detect all targets in the visible area. Through the above method, the detection results can be used to correct the semantic segmentation results and recall the possible missed detection problems of the detection box.
[0054] In one embodiment of the present application, the target detection box that may have been missed is recalled according to the category correction result of the point cloud classification to obtain the final target detection and segmentation result, including: determining the target detection box that may have been missed; obtaining a corrected target semantic segmentation result according to the category correction result of the point cloud classification; and recalling the target detection box that may have been missed according to the corrected target semantic segmentation result to obtain the final target detection and segmentation result.
[0055] When recalling the target detection frame that may have missed detection, the target detection frame that may have missed detection is recalled according to the corrected target semantic segmentation result to obtain the final target detection and segmentation result.
[0056] In one embodiment of the present application, the category correction result based on the point cloud classification is used to recall the target detection frame that may have missed detection to obtain the final target detection and segmentation result, and also includes: according to the corrected target semantic segmentation result, for the point cloud data of a preset category, the target detection frame obtained after correction is used to filter out the point cloud of the preset category to obtain the point cloud that does not belong to the target detection frame; and the point cloud that does not belong to the target detection frame is clustered using a clustering algorithm to obtain a usable point cloud cluster.
[0057] The obtained point cloud semantic segmentation results are used to filter out point clouds of a certain category. The target detection box obtained based on the re-prediction is first used to filter out the point clouds that have been detected, and then the DBSCAN method is used for clustering. The available point cloud clusters are filtered out according to the clustering results.
[0058] It can be understood that the DBSCAN clustering method is only an example and is not used to limit the scope of protection in the embodiments of the present application.
[0059] In one embodiment of the present application, the method further includes: re-inputting the available point cloud cluster into a refine model to obtain the position and size of the target detection box after regression of each point cloud cluster and the category of the corresponding target detection box.
[0060] Specifically, according to the screening of point cloud clusters, each point cloud cluster is input into the refine model to obtain the location and size of each point cloud cluster regression, as well as the category of the corresponding detection box.
[0061] The above method can prevent the omission of small targets.
[0062] In one embodiment of the present application, the correcting the position and size of the target detection box in the target detection result according to the refine model and re-predicting the type of the target detection box includes: according to the target detection box, using the refine model to regress the position and size of the target detection box in the target detection result, and re-predicting the type of the target detection box BBOX.
[0063] When re-predicting the type of the target detection box, the refine model is used to regress the position and size of the target detection box in the target detection result according to the target detection box, and the type of the target detection box BBOX also needs to be re-predicted.
[0064] Through the above method, the results of small target detection can be guaranteed to be more accurate.
[0065] In one embodiment of the present application, the category of the point cloud classification is corrected based on the re-predicted target detection frame to correct the category of the point cloud classification to the category of the target detection frame, including: re-predicting the type of the target detection frame according to the refine model, and correcting the point cloud classification to the category of the detection frame according to the confidence score of the target detection frame.
[0066] The type of the target detection box BBOX is re-predicted according to the refine model, and the point cloud classification is corrected to the category of the detection box according to the confidence score of the target detection box. It can be understood that the point cloud classification can be corrected to the category of the detection box based on the result of a high confidence score of the target detection box.
[0067] Through the above method, the results of point cloud classification can be optimized and some possible false detections can be corrected.
[0068] An embodiment of the present application provides a target detection method, wherein the target detection and segmentation method of the multi-task model is adopted, and the target detection method includes:
[0069] The final target detection and segmentation result is used as the target detection result.
[0070] The target detection and segmentation method of the multi-task model includes:
[0071] Outputting target detection results and target semantic segmentation results according to the pre-trained target detection and segmentation model, wherein the target detection results include target detection boxes, and the target semantic segmentation results include point cloud classification of the target;
[0072] Correcting the position and size of the target detection frame in the target detection result according to the refine model, and re-predicting the type of the target detection frame;
[0073] According to the re-predicted target detection frame, the category of the point cloud classification is corrected to the category of the target detection frame;
[0074] According to the category correction result of the point cloud classification, the target detection frames that may have been missed are recalled to obtain the final target detection and segmentation results.
[0075] The present application also provides a multi-task model target detection and segmentation device 200, such as Figure 2 As shown, a schematic diagram of the structure of the target detection and segmentation device of the multi-task model in the embodiment of the present application is provided, and the target detection and segmentation device 200 of the multi-task model at least includes: a result output module 210, a first correction module 220, a second correction module 230 and a recall module 240, wherein:
[0076] In one embodiment of the present application, the result output module 210 is specifically used to: output target detection results and target semantic segmentation results based on a pre-trained target detection and segmentation model, the target detection results include a target detection box, and the target semantic segmentation results include a point cloud classification of the target.
[0077] The above method is mainly executed by the vehicle side. In the intelligent connected car, the vehicle-mounted laser radar has been installed, and the model parameters have been adjusted to obtain normal lane information in real time. The vehicle has a similar Xavier vehicle-mounted machine that can run the deep learning model. Through the designed neural network, a deep learning model can be trained that can correctly detect and classify targets and output semantic segmentation prediction results.
[0078] The deep learning model is a pre-trained target detection and segmentation model, which outputs target detection results and target semantic segmentation results. It can be understood that the target detection result at least includes a target detection box, which can detect and determine the location of the target. The target semantic segmentation result at least includes the point cloud classification of the target. Based on the laser radar, a 3D laser point cloud can be obtained and the 3D laser point cloud can be classified to obtain the point cloud classification of the target.
[0079] In one embodiment of the present application, the first correction module 220 is specifically used to: correct the position and size of the target detection frame in the target detection result according to a refine model, and re-predict the type of the target detection frame.
[0080] A refine model usually refers to the part of a machine learning model that is used to refine and optimize the original output. This model usually corresponds to a base model, which is responsible for generating a preliminary result, while the base model is responsible for improving the quality and accuracy of these results. The refine model has a wide range of application scenarios. Those skilled in the art can use the refine model for tasks including but not limited to image processing, natural language processing, and more complex tasks such as 3D detection and tracking.
[0081] Specifically, the location and size of the target detection box in the target detection result are corrected according to the refine model. The location of the target detection box is the location of the target that needs to be located. The size of the target detection box includes the length and width of the 2D target detection box and the length, width and height of the 3Dbox target box. After correction, the type of the target detection box is re-predicted.
[0082] In one embodiment of the present application, the second correction module 230 is specifically used to correct the category of the point cloud classification according to the re-predicted target detection frame, so as to correct the category of the point cloud classification to the category of the target detection frame.
[0083] According to the above module, the re-predicted target detection frame can be obtained, and the re-predicted target detection frame is used to correct the category of the point cloud classification. The purpose of correcting the category of the point cloud classification is to correct the category of the point cloud classification to the category of the target detection frame, that is, the category of the target detection frame is more accurate, so the category of the point cloud classification is corrected to the category of the target detection frame. And generally speaking, the target semantic segmentation results based on the lidar point cloud are more than the target detection frame.
[0084] In one embodiment of the present application, the recall module 240 is specifically used to: recall the target detection frames that may have missed detection according to the category correction result of the point cloud classification, and obtain the final target detection and segmentation results.
[0085] When recalling the detection box using semantic segmentation, the target detection box that may have been missed is recalled to obtain the final target detection and segmentation results. The category correction result of point cloud classification can obtain more accurate category classification results.
[0086] In one embodiment of the present application, the recall module 240 is also used to
[0087] Determine the target detection boxes that may have missed detections;
[0088] According to the category correction result of the point cloud classification, a corrected target semantic segmentation result is obtained;
[0089] The target detection frames that may have missed detection are recalled according to the corrected target semantic segmentation result to obtain the final target detection and segmentation result.
[0090] In one embodiment of the present application, the recall module 240 is also used to
[0091] According to the corrected target semantic segmentation result, for point cloud data of a preset category, the corrected target detection frame is used to filter out the point cloud of the preset category to obtain a point cloud that does not belong to the target detection frame;
[0092] The point cloud that does not belong to the target detection frame is clustered using a clustering algorithm to obtain usable point cloud clusters.
[0093] In one embodiment of the present application, the recall module 240 is also used to
[0094] The available point cloud clusters are re-input into the refine model to obtain the position and size of the target detection frame after regression of each point cloud cluster and the category of the corresponding target detection frame.
[0095] In one embodiment of the present application, the first correction module 220 is also used to
[0096] According to the target detection box, the refine model is used to regress the position and size of the target detection box in the target detection result, and the type of the target detection box BBOX is re-predicted.
[0097] In one embodiment of the present application, the second correction module 230 is also used to
[0098] The type of the target detection box BBOX is re-predicted according to the refine model, and the point cloud classification is corrected to the category of the detection box according to the confidence score of the target detection box.
[0099] It can be understood that the target detection and segmentation device of the above-mentioned multi-task model can implement the various steps of the target detection and segmentation method of the multi-task model provided in the aforementioned embodiments. The relevant explanations about the target detection and segmentation method of the multi-task model are applicable to the target detection and segmentation device of the multi-task model and will not be repeated here.
[0100] Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Figure 3 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. The memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. Of course, the electronic device may also include hardware required for other services.
[0101] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0102] The memory is used to store the program. Specifically, the program may include a program code, and the program code includes a computer operation instruction. The memory may include a memory and a non-volatile memory, and provides instructions and data to the processor.
[0103] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a target detection and segmentation device of a multi-task model at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:
[0104] Outputting target detection results and target semantic segmentation results according to the pre-trained target detection and segmentation model, wherein the target detection results include target detection boxes, and the target semantic segmentation results include point cloud classification of the target;
[0105] Correcting the position and size of the target detection frame in the target detection result according to the refine model, and re-predicting the type of the target detection frame;
[0106] According to the re-predicted target detection frame, the category of the point cloud classification is corrected to the category of the target detection frame;
[0107] According to the category correction result of the point cloud classification, the target detection frames that may have been missed are recalled to obtain the final target detection and segmentation results.
[0108] The above application Figure 1 The method performed by the target detection and segmentation device of the multi-task model disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.
[0109] The electronic device may also perform Figure 1 A method for executing a target detection and segmentation device of a multi-task model in a Figure 1 The functions of the illustrated embodiment will not be described in detail in the embodiments of the present application.
[0110] The present application also provides a computer-readable storage medium, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by an electronic device including multiple application programs, enable the electronic device to execute Figure 1 The method performed by the target detection and segmentation device of the multi-task model in the illustrated embodiment is specifically used to perform:
[0111] Outputting target detection results and target semantic segmentation results according to the pre-trained target detection and segmentation model, wherein the target detection results include target detection boxes, and the target semantic segmentation results include point cloud classification of the target;
[0112] Correcting the position and size of the target detection frame in the target detection result according to the refine model, and re-predicting the type of the target detection frame;
[0113] According to the re-predicted target detection frame, the category of the point cloud classification is corrected to the category of the target detection frame;
[0114] According to the category correction result of the point cloud classification, the target detection frames that may have been missed are recalled to obtain the final target detection and segmentation results.
[0115] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0116] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0117] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0118] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0119] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0120] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0121] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0122] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0123] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.
[0124] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims
1. A multi-task model for object detection and segmentation, wherein: The method comprises: Outputting a target detection result and a target semantic segmentation result according to a pre-trained target detection model and a target semantic segmentation model, wherein the target detection result includes a target detection frame, and the target semantic segmentation result includes a point cloud classification of the target; Correcting the position and size of the target detection frame in the target detection result according to the refine model, and re-predicting the type of the target detection frame; According to the re-predicted target detection frame, the category of the point cloud classification is corrected to the category of the target detection frame; According to the category correction result of the point cloud classification, the target detection frames that may have been missed are recalled to obtain the final target detection and segmentation results.
2. The method according to claim 1, wherein: The method of revising the category of the point cloud classification and recalling the target detection frames that may have been missed to obtain the final target detection and segmentation results includes: Determine the target detection boxes that may have missed detections; According to the category correction result of the point cloud classification, a corrected target semantic segmentation result is obtained; The target detection frames that may have missed detection are recalled according to the corrected target semantic segmentation result to obtain the final target detection and segmentation result.
3. The method according to claim 2, wherein: The method further includes: recalling the target detection frames that may have been missed according to the category correction result of the point cloud classification to obtain the final target detection and segmentation result; According to the corrected target semantic segmentation result, for point cloud data of a preset category, the corrected target detection frame is used to filter out the point cloud of the preset category to obtain a point cloud that does not belong to the target detection frame; The point cloud that does not belong to the target detection frame is clustered using a clustering algorithm to obtain usable point cloud clusters.
4. The method according to claim 3, wherein: The method further comprises: The available point cloud clusters are re-input into the refine model to obtain the position and size of the target detection frame after regression of each point cloud cluster and the category of the corresponding target detection frame.
5. The method according to claim 1, wherein: The correcting the position and size of the target detection frame in the target detection result according to the refine model and re-predicting the type of the target detection frame includes: According to the target detection box, the refine model is used to regress the position and size of the target detection box in the target detection result, and the type of the target detection box BBOX is re-predicted.
6. The method according to claim 1, wherein: The step of correcting the category of the point cloud classification according to the re-predicted target detection frame to correct the category of the point cloud classification to the category of the target detection frame includes: The type of the target detection box is re-predicted according to the refine model, and the point cloud classification is corrected to the category of the detection box according to the confidence score of the target detection box.
7. A target detection method, wherein: The target detection and segmentation method of the multi-task model according to any one of claims 1 to 6 is adopted, wherein the target detection method comprises: The final target detection and segmentation result is used as the target detection result.
8. A multi-task model for object detection and segmentation, wherein: The device comprises: A result output module, used to output a target detection result and a target semantic segmentation result according to a pre-trained target detection and segmentation model, wherein the target detection result includes a target detection box, and the target semantic segmentation result includes a point cloud classification of the target; A first correction module, used to correct the position and size of the target detection frame in the target detection result according to a refine model, and re-predict the type of the target detection frame; A second correction module is used to correct the category of the point cloud classification according to the re-predicted target detection frame, so as to correct the category of the point cloud classification to the category of the target detection frame; The recall module is used to recall the target detection frames that may have been missed according to the category correction result of the point cloud classification, so as to obtain the final target detection and segmentation results.
9. An electronic device, comprising: processor; as well as A memory arranged to store computer executable instructions, which when executed cause the processor to perform the method of any one of claims 1 to 7.
10. A computer-readable storage medium storing one or more programs, which, when executed by an electronic device including a plurality of application programs, causes the electronic device to execute any one of the methods of claims 1 to 7.