Object detection method, device and equipment and storage medium
Patent Information
- Application Number
- CN202410999623.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-24
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-07-24
Smart Images

Figure CN118918565A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of autonomous driving technology, and in particular to an object detection method, device, equipment and storage medium. Background Art
[0002] With the continuous development of information technology, vision-based 3D object detection plays a vital role in autonomous driving systems. Autonomous driving solutions rely on accurate 3D detection results to predict the future driving behavior of other vehicles and plan the driving trajectory of the vehicle.
[0003] However, existing 3D object detectors usually follow an offline training and deployment process. Once the detector is trained and deployed on an autonomous vehicle, these offline solutions cannot update their parameters in real time and correct errors immediately when the detector cannot perceive new objects or new scenes due to changes in the application domain. Such defects pose a significant safety risk to reliable driving systems, such as incorrect lane changes, turns, and even crashes due to abnormal 3D detector results. Summary of the invention
[0004] The present invention provides an object detection method, device, equipment and storage medium to solve the problem of poor error correction effect of object detection.
[0005] In a first aspect, the present invention provides an object detection method, comprising:
[0006] Acquire an image frame of the current scene, and determine target information of the image frame using a preset three-dimensional detection model, wherein the target information includes image feature information and depth information;
[0007] Using the preset 3D detection model, generating an object frame and a corresponding confidence level of the object in the image frame according to a visual cue image and the target information, wherein the visual cue image includes a cue image associated with a missed object determined by a user, and the missed object is determined according to a historical output of the preset 3D detection model;
[0008] The object frame is screened according to the confidence level to obtain a first target object frame.
[0009] In a second aspect, the present invention provides an object detection device, comprising:
[0010] A target information determination module, used to obtain an image frame of the current scene, and determine the target information of the image frame using a preset three-dimensional detection model, wherein the target information includes image feature information and depth information;
[0011] an object frame determination module, configured to generate an object frame and a corresponding confidence level of an object in the image frame according to a visual cue image and the target information using the preset 3D detection model, wherein the visual cue image includes a cue image associated with a missed object determined by a user, and the missed object is determined according to a historical output of the preset 3D detection model;
[0012] The object frame screening module is used to screen the object frame according to the confidence level to obtain a first target object frame.
[0013] In a third aspect, the present invention provides an electronic device, the electronic device comprising:
[0014] at least one processor;
[0015] and a memory communicatively coupled to the at least one processor;
[0016] The memory stores a computer program that can be executed by at least one processor, and the computer program is executed by at least one processor so that the at least one processor can execute the object detection method of the first aspect.
[0017] In a fourth aspect, the present invention provides a computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a processor to implement the object detection method of the first aspect when executed.
[0018] The object detection scheme provided by the present invention obtains an image frame of the current scene, and determines the target information of the image frame using a preset three-dimensional detection model, wherein the target information includes image feature information and depth information, and uses the preset three-dimensional detection model to generate an object frame and a corresponding confidence level of the object in the image frame according to a visual cue image and the target information, wherein the visual cue image includes a cue image associated with a missed object determined by a received user, and the missed object is determined according to the historical output of the preset three-dimensional detection model, and the object frame is screened according to the confidence level to obtain a first target object frame. By adopting the above technical scheme, the visual cue image determined by the user according to the historical output of the preset three-dimensional detection model, as well as the image feature information and depth information of the image frame are input into the preset three-dimensional detection model, so as to achieve a rapid response to the failure of model detection, immediately correct the detection error, identify the target object that was not recognized before, avoid the risk caused by the detection error, and achieve reliable and safe autonomous driving.
[0019] It should be understood that the content described in this section is not intended to identify the key or important features of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0021] Figure 1 is a flow chart of an object detection method provided according to Embodiment 1 of the present invention;
[0022] Figure 2 is a schematic diagram of an object detection system provided according to Embodiment 1 of the present invention;
[0023] Figure 3 is a flow chart of an object detection method provided according to Embodiment 2 of the present invention;
[0024] Figure 4 is a flow chart of an object detection method provided according to Embodiment 3 of the present invention;
[0025] Figure 5 is a flow chart of an object detection method provided according to a fourth embodiment of the present invention;
[0026] Figure 6 is a structural schematic diagram of an object detection device provided according to Embodiment 5 of the present invention;
[0027] Figure 7 It is a structural schematic diagram of an electronic device provided according to Embodiment 6 of the present invention. DETAILED DESCRIPTION
[0028] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In the description of the present invention, unless otherwise specified, "multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are an "or" relationship. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0030] Embodiment 1
[0031] Figure 1 A flowchart of an object detection method is provided for the first embodiment of the present invention. This embodiment is applicable to the case where an intelligent driving vehicle detects an object. The method can be performed by an object detection device. The object detection device can be implemented in the form of hardware and / or software. The object detection device can be configured in an electronic device. The electronic device can be composed of two or more physical entities or one physical entity.
[0032] like Figure 1 As shown, the object detection method provided in the first embodiment of the present invention specifically includes the following steps:
[0033] S101. Acquire an image frame of a current scene, and determine target information of the image frame using a preset three-dimensional detection model, wherein the target information includes image feature information and depth information.
[0034] In this embodiment, the camera configured in the vehicle can be used to obtain the image frame of the current scene. The submodule in the preset 3D detection model can be used to extract the image feature information of the image frame, and the depth information of the current scene can be predicted based on the feature information. Among them, the preset 3D detection model can be understood as a monocular online 3D detector (Online 3D Detector, preset 3D detection model).
[0035] S102. Utilizing the preset three-dimensional detection model, generating an object frame and a corresponding confidence level of the object in the image frame according to a visual cue image and the target information, wherein the visual cue image includes a cue image received and determined by a user to be associated with a missed object, and the missed object is determined according to a historical output of the preset three-dimensional detection model.
[0036] In this embodiment, a preset three-dimensional detection model can be used to evenly set points in the image frame (the points are set points), and the position information of the set points can be determined, and the position information is the position information of the default mode. Then the position information, visual cue image and target information are input into the preset three-dimensional detection model to obtain the object frame and corresponding confidence of the object in the image frame. Image feature information and depth information can be used as the key and value of the feature decoder in the preset three-dimensional detection model. The preset three-dimensional detection model can predict the position information of the missed object based on the visual cue image, and the position information can be used as a query of the feature decoder in the preset three-dimensional detection model after feature encoding. Among them, the preset three-dimensional detection model can output a three-dimensional object frame, object frame position, object frame size, object frame confidence, and object frame depth, etc.
[0037] S103: Screen the object frame according to the confidence level to obtain a first target object frame.
[0038] In this embodiment, the object frame with a larger confidence level may be retained to obtain a first target object frame, thereby realizing detection of the object in the image frame.
[0039] The object detection method provided by the embodiment of the present invention obtains an image frame of the current scene, and determines the target information of the image frame using a preset three-dimensional detection model, wherein the target information includes image feature information and depth information, and uses the preset three-dimensional detection model to generate an object frame and a corresponding confidence of the object in the image frame according to a visual cue image and the target information, wherein the visual cue image includes a cue image associated with a missed object determined by the received user, and the missed object is determined according to the historical output of the preset three-dimensional detection model, and the object frame is screened according to the confidence to obtain a first target object frame. The technical solution of the embodiment of the present invention inputs the visual cue image determined by the user according to the historical output of the preset three-dimensional detection model, as well as the image feature information and depth information of the image frame into the preset three-dimensional detection model, thereby realizing a rapid response to the failure of model detection, immediately correcting the detection error, identifying the target object that was not previously identified, avoiding the risk caused by the detection error, and realizing reliable and safe autonomous driving.
[0040] Optionally, the above method also includes: if the visual cue image is empty, using the preset three-dimensional detection model to determine the position information of the set point in the image frame; inputting the position information into the cue encoder in the preset three-dimensional detection model to obtain a first position code; inputting the target information and the first position code into the feature decoder in the preset three-dimensional detection model to obtain the object frame and the corresponding confidence of the object in the image frame, and screening the object frame according to the confidence to obtain a second target object frame.
[0041] Specifically, Figure 2 is a schematic diagram of an object detection system. If there is no visual cue image, that is, the visual cue image is empty, such as Figure 2 As shown, the position information of the default mode can be input into the prompt encoder, that is, the preset three-dimensional detection model is first used to evenly set points in the image frame (the points are the set points), and the position information of the set points is determined, and the position information is the position information of the default mode. After the position information is input into the prompt encoder in the preset three-dimensional detection model, the first position code can be obtained. After the target information and the first position code are input into the feature decoder in the preset three-dimensional detection model, the preset three-dimensional detection model can determine the object frame and the corresponding confidence of the object in the image frame according to the output result of the feature decoder. The object frame with a larger confidence can be screened out to obtain the second target object frame.
[0042] Embodiment 2
[0043] Figure 3 This is a flow chart of an object detection method provided in Embodiment 2 of the present invention. The technical solution of the embodiment of the present invention is further optimized on the basis of the above-mentioned optional technical solutions, and provides a specific method for an intelligent driving vehicle to detect an object.
[0044] Optionally, before using the preset three-dimensional detection model to generate the object frame and the corresponding confidence of the object in the image frame according to the visual cue image and the target information, it also includes: receiving the associated points and / or associated frames of the missed objects in the historical image frames input by the user, and determining the position information of the associated points and / or the position information of the associated frames; using the preset three-dimensional detection model, according to the position information of the associated points and / or the position information of the associated frames, and the target information of the historical image frames, determining the visual cue frame, and determining the image corresponding to the visual cue frame in the historical image frame as the first visual cue image. The advantage of such a setting is that whenever the model misses an object, the user can provide user feedback information to the model by clicking or framing the unrecognized object in the image, and the detection error is immediately corrected by using the feedback information.
[0045] Optionally, the use of the preset 3D detection model to generate the object frame and corresponding confidence of the object in the image frame according to the visual cue image and the target information includes: inputting the first visual cue image into the cue encoder in the preset 3D detection model to obtain the second position code; inputting the second position code and the target information into the feature decoder in the preset 3D detection model to obtain the object frame and corresponding confidence of the object in the image frame. The advantage of such a setting is that accurate error correction of missed objects is achieved by using the cue encoder and feature decoder in the preset 3D detection model.
[0046] like Figure 3 As shown, the object detection method provided by the second embodiment of the present invention specifically includes the following steps:
[0047] S201, acquiring an image frame of a current scene, and determining target information of the image frame using a preset three-dimensional detection model.
[0048] S202 , determining whether the visual prompt image is empty, if so, executing step 203 , if not, executing step 205 .
[0049] Specifically, Figure 2 As shown, a visual cue buffer can be pre-set, and the buffer is used to store visual cue images corresponding to undetected objects and objects of interest determined by the user during the online reasoning process of the preset three-dimensional detection model.
[0050] S203: Determine position information of a set point in the image frame using the preset 3D detection model, and input the position information into a prompt encoder in the preset 3D detection model to obtain a first position code.
[0051] S204. Input the target information and the first position code into a feature decoder in the preset three-dimensional detection model to obtain an object frame and a corresponding confidence level of the object in the image frame, and filter the object frame according to the confidence level to obtain a second target object frame.
[0052] S205: Receive the associated points and / or associated frames of the missed objects in the historical image frames input by the user, and determine the position information of the associated points and / or the position information of the associated frames.
[0053] Specifically, Figure 2 As shown in the figure, whenever the preset 3D detection model makes an error (i.e. misses an object), the user can provide human feedback information to the model by clicking or selecting the unrecognized object in the image. Clicking the corresponding point is the associated point of the missed object, and selecting the image is the associated frame. The historical image frame can be the previous frame or the previous N frames, where N is greater than 1.
[0054] S206. Using the preset three-dimensional detection model, determine a visual prompt frame according to the position information of the associated point and / or the position information of the associated frame, and the target information of the historical image frame, and determine the image corresponding to the visual prompt frame in the historical image frame as the first visual prompt image.
[0055] Specifically, the preset three-dimensional detection model can be used to extract the location information of the associated points and / or the location information of the associated boxes, and the visual prompt box can be determined based on the location information and the target information of the historical image frame. The image corresponding to the prompt box is the first visual prompt image.
[0056] Optional, such as Figure 2 As shown, the visual cue buffer includes an "entry" and "de-queue" mechanism. The "entry" mechanism refers to receiving visual cue images. The "de-queue" mechanism refers to discarding unnecessary visual cues in the buffer using preset rules in order to prevent the buffer from growing indefinitely. For example, deleting: (1) visual cue images with low confidence output by the preset 3D detection model, the objects referred to by the images may not exist in the current scene; (2) visual cue images corresponding to detection boxes with high intersection over union (IoU) output by the preset 3D detection model, which means that these visual cue images may appear multiple times in the buffer. The advantage of this setting is that by maintaining the visual cue buffer, the input records of the visual cue images can be tracked and the visual cue image queue can be dynamically adjusted, so that the model can use the visual cue images in the visual cue buffer to detect and track previously missed objects online, thereby improving the detection robustness of the model and achieving rapid error correction.
[0057] S207: Input the first visual cue image into a cue encoder in the preset three-dimensional detection model to obtain a second position code.
[0058] S208: Input the second position code and the target information into a feature decoder in the preset three-dimensional detection model to obtain an object frame and a corresponding confidence level of the object in the image frame.
[0059] Specifically, the second position code can be used as the initialization of the query of the feature decoder. Figure 2 As shown, when there is a visual cue image, that is, the visual cue image is not empty, the position information of the set point in the above image frame and the visual cue image can be input into the cue encoder to obtain a second position code. Then, the second position code and the target information can be input into the feature decoder in the preset three-dimensional detection model, and the 3D frame prediction module in the preset three-dimensional detection model can be used to output the three-dimensional object frame, object frame position, object frame size, object frame confidence, object frame depth, etc. according to the result output by the feature decoder.
[0060] S209: Filter the object frame according to the confidence level to obtain a first target object frame.
[0061] According to the object detection method provided by the embodiment of the present invention, whenever the model misses an object, the user can provide user feedback information to the model by clicking or framing the unrecognized object in the image. The detection error is immediately corrected by utilizing the feedback information, and accurate error correction of missed objects is achieved by presetting the prompt encoder and feature decoder in the three-dimensional detection model.
[0062] Embodiment 3
[0063] Figure 4 This is a flow chart of an object detection method provided in Example 3 of the present invention. The technical solution of the embodiment of the present invention is further optimized on the basis of the above-mentioned optional technical solutions, and provides another specific method for an intelligent driving vehicle to detect an object.
[0064] Optionally, before using the preset three-dimensional detection model to generate the object frame and the corresponding confidence of the object in the image frame according to the visual cue image and the target information, it also includes: receiving an image block associated with the missed object in the historical image frame input by the user; determining the image block as the second visual cue image. The advantage of this setting is that by receiving the image block associated with the missed object in the historical image frame directly uploaded by the user, the preset three-dimensional detection model can correct the object detection across scenes and timestamps, thereby realizing online error correction of the detection results, so that the vehicle can perform autonomous driving more safely and robustly.
[0065] Optionally, the method of using the preset three-dimensional detection model to generate an object frame and a corresponding confidence of the object in the image frame according to the visual cue image and the target information includes: inputting the second visual cue image into the visual alignment module in the preset three-dimensional detection model to obtain a candidate position of the object corresponding to the second visual cue image in the image frame; inputting the candidate position into the cue encoder in the preset three-dimensional detection model to obtain a third position code; inputting the third position code and the target information into the feature decoder in the preset three-dimensional detection model to obtain the object frame and the corresponding confidence of the object in the image frame. The advantage of such a setting is that the visual alignment of the visual cue image and the image frame is achieved by using the visual alignment module, thereby accurately predicting the candidate position of the object corresponding to the visual cue image in the image frame, further ensuring the accuracy of error correction.
[0066] like Figure 4 As shown, the object detection method provided by the third embodiment of the present invention specifically includes the following steps:
[0067] S301: Acquire an image frame of a current scene, and determine target information of the image frame using a preset three-dimensional detection model.
[0068] S302 , determining whether the visual prompt image is empty, if so, executing step 303 , if not, executing step 305 .
[0069] S303: Determine position information of a set point in the image frame using the preset 3D detection model, and input the position information into a prompt encoder in the preset 3D detection model to obtain a first position code.
[0070] S304: input the target information and the first position code into a feature decoder in the preset three-dimensional detection model to obtain an object frame and a corresponding confidence level of the object in the image frame, and filter the object frame according to the confidence level to obtain a second target object frame.
[0071] S305: Receive an image block associated with a missed object in a historical image frame input by a user.
[0072] S306: Determine the image block as a second visual prompt image.
[0073] Specifically, the visual prompt images may also be image blocks associated with objects missed in historical image frames, such as images from the Internet, etc. Users may directly upload customized image blocks corresponding to frequently missed objects based on experience and use them as visual prompt images.
[0074] S307: Input the second visual cue image into a visual alignment module in the preset three-dimensional detection model to obtain a candidate position of the object corresponding to the second visual cue image in the image frame.
[0075] Specifically, Figure 2 As shown, the visual cue (i.e., the second visual cue image) can be input into a visual alignment module of a preset three-dimensional detection model. The module can visually align the second visual cue image with the current input image to obtain a predicted position after alignment, i.e., a candidate position of the object corresponding to the second visual cue image in the image frame.
[0076] Optionally, the second visual cue image is input into a visual alignment module in the preset three-dimensional detection model to obtain a candidate position of an object corresponding to the second visual cue image in the image frame, including: using the visual alignment module to determine the likelihood of image feature information in the target information and image features of the second visual cue image, and determining the candidate position of the object corresponding to the second visual cue image in the image frame based on the likelihood.
[0077] Specifically, the visual alignment module can calculate the likelihood of the image feature information in the target information and the image feature of the second visual cue image, and output a candidate position according to the likelihood. There can be multiple candidate positions, and in the supervised training stage of the visual alignment module, the position with the smallest Dice loss value can be determined as the candidate position, and there can be multiple second visual cue images, thereby obtaining multiple candidate positions.
[0078] S308: Input the candidate position into a prompt encoder in the preset three-dimensional detection model to obtain a third position code.
[0079] Specifically, the position information of the set point in the above image frame and the candidate position may be input into the prompt encoder to obtain the third position code.
[0080] S309: input the third position code and the target information into the feature decoder in the preset three-dimensional detection model to obtain the object frame and the corresponding confidence of the object in the image frame, and filter the object frame according to the confidence to obtain the first target object frame.
[0081] The object detection method provided by the embodiment of the present invention receives image blocks directly uploaded by users and associated with missed objects in historical image frames, so that a preset three-dimensional detection model can correct object detection across scenes and timestamps, thereby realizing online error correction of detection results, allowing vehicles to perform autonomous driving more safely and robustly, and utilizing a visual alignment module to realize visual alignment of visual cue images with image frames, thereby accurately predicting the candidate positions of objects corresponding to the visual cue images in the image frames, further ensuring the accuracy of error correction.
[0082] Embodiment 4
[0083] Figure 5 This is a flow chart of an object detection method provided in Embodiment 4 of the present invention. The technical solution of the embodiment of the present invention is further optimized on the basis of the above-mentioned optional technical solutions, and provides another specific method for an intelligent driving vehicle to detect an object.
[0084] like Figure 5 As shown, a method for detecting an object provided by a fourth embodiment of the present invention specifically includes the following steps:
[0085] S401, obtaining an image frame of the current scene, and determining target information of the image frame using a preset three-dimensional detection model.
[0086] S402 , determining whether the visual prompt image is empty, if so, executing step 403 , if not, executing step 405 .
[0087] S403: Determine position information of a set point in the image frame using a preset three-dimensional detection model, and input the position information into a prompt encoder in the preset three-dimensional detection model to obtain a first position code.
[0088] S404: Input the target information and the first position code into a feature decoder in a preset three-dimensional detection model to obtain an object frame and a corresponding confidence level of the object in the image frame, and filter the object frame according to the confidence level to obtain a second target object frame.
[0089] S405, receiving the associated points and / or associated boxes of the missed objects in the historical image frames input by the user, and determining the position information of the associated points and / or the position information of the associated boxes; receiving the image blocks associated with the missed objects in the historical image frames input by the user.
[0090] S406. Using a preset three-dimensional detection model, determine a visual prompt frame according to the position information of the associated point and / or the position information of the associated frame, and the target information of the historical image frame, and determine the image corresponding to the visual prompt frame in the historical image frame as the first visual prompt image; determine the image block as the second visual prompt image.
[0091] S407: Input the first visual cue image into a cue encoder in a preset three-dimensional detection model to obtain a second position code.
[0092] S408. Using the visual alignment module, determine the likelihood of the image feature information in the target information and the image features of the second visual cue image, and determine the candidate position of the object corresponding to the second visual cue image in the image frame based on the likelihood; input the candidate position into the cue encoder in the preset three-dimensional detection model to obtain a third position code.
[0093] S409: input the second position code, the third position code and the target information into a feature decoder in a preset three-dimensional detection model to obtain an object frame and a corresponding confidence level of the object in the image frame, and filter the object frame according to the confidence level to obtain a first target object frame.
[0094] Specifically, object detection methods can be evaluated from the following three aspects:
[0095] 1. When using points, boxes, or image patches for error correction detection, how does the performance of the preset 3D detection model compare to the traditional offline 3D detection mode?
[0096] 2. How does the preset 3D detection model perform in terms of instant online error correction?
[0097] 3. When faced with scenes beyond the training distribution, how does the preset 3D detection model enhance the performance of offline trained detectors?
[0098] The experimental setup includes the following four aspects:
[0099] 1) Dataset:
[0100] Experiments can be performed on the nuScenes dataset, which contains 1,000 autonomous driving sequences and is one of the most commonly used large-scale datasets in autonomous driving research.
[0101] 2) Tasks:
[0102] The first experiment aims to test the performance of the preset 3D detection model in online error correction. This experiment is conducted by simulating user intervention. Specifically, user feedback is simulated by comparing the distance between the ground truth and the detected 3D box. Objects with no detection box match within a range of 2 meters are considered missed and added to the hint buffer.
[0103] Then, verify the ability of the preset 3D detection model to correct detection errors in scenarios beyond the training distribution. Set the following four tasks:
[0104] 80% of distant objects beyond 30 meters are discarded during training, and error correction for detecting distant objects is tested.
[0105] During training, 80% of objects labeled as vehicles (including “car”, “truck”, “commercial vehicle”, “bus”, and “trailer”) are discarded, and the error correction of detecting vehicle objects is tested.
[0106] All objects of the “truck” and “bus” categories are discarded during training, and the zero-shot detection capability of the pre-set 3D detection model is tested on these discarded categories.
[0107] All data from the "night" and "rainy day" scenes are discarded during training, and the improvements of the preset 3D detection model in scenes with domain gaps are tested.
[0108] Finally, experiments are designed to verify the effectiveness of each specific cue (including point cue, box cue, image patch cue, and default mode). The performance of these four cues is compared, and the robustness of visual cues from different viewpoints is tested by comparing the recall rate of detection in the next 15 frames using the visual cues in the current frame.
[0109] 3) Entity Detection Scoring:
[0110] Prioritize the ability of the pre-set 3D detection model to zero-shot localize and detect new objects during the test phase rather than its classification ability. Therefore, the category annotations of all 3D objects are removed during training and all objects are simply considered as entities. Similarly, the entity detection score (EDS), a category-free version of the nuScenes detection score (NDS), is used during evaluation to evaluate the performance of different models.
[0111] 4) Implementation details:
[0112] The above methods are implemented based on the mmDet3D code base, and all experiments are performed on a server with 8 A100 GPUs. In the preset 3D detection model, ResNet101 and FPN neck are used as image encoders to extract the target image feature Z. The visual cue encoder is ResNet18. All visual cues are resized to 224×224 before being sent to the visual cue encoder. For the depth branch, the head structure is directly followed to predict the depth, the sparse point cloud is used for supervision, and the loss function is the same as Zoedepth. The depth map is encoded into high-dimensional features according to MonoDETR. When training, the AdamW optimizer is used with a batch size of 16, which is evenly distributed to 8 GPUs. The learning rate is initialized to 2e-4 and adjusted by a cosine annealing strategy. When training point and box cues, user input is simulated by adding perturbations to the true values. To ensure the robustness of the visual cues, the selected visual cues are not only from the image patches of the current frame, but also randomly selected from the corresponding image patches within the range of 5 frames before and after it. Flip and resize operations are used as image-level data augmentation.
[0113] Test results:
[0114] 1) Effectiveness of preset 3D detection models in real-time error correction:
[0115] The online error correction capability is the core capability of the preset 3D detection model. The preset 3D detection model brings a significant improvement of 4.7% mAP (Mean Average Precision) and 5.0% EDS (Entity Detection Score) without any training. This proves the effectiveness of the preset 3D detection model in instantly correcting test errors during online reasoning.
[0116] 2) Effectiveness of the preset 3D detection model in scenarios beyond training:
[0117] The pre-set 3D detection model improves performance on vehicle objects by 20.8% mAP and 21.6% EDS. The pre-set 3D detection model is effective in successfully detecting new objects that are not annotated in the training set, achieving 8.5% mAP and 13.6% EDS on these new objects, significantly exceeding the offline baseline’s 0.0% mAP and EDS. The pre-set 3D detection model is effective in performing online error correction when driving into scenes with domain shifts, providing 4.5% mAP and 4.7% EDS improvements. These experiments demonstrate that the pre-set 3D detection model is an effective and versatile on-the-fly error correction system that excels in handling lost distant objects, unseen object categories, and scene domain shifts. The pre-set 3D detection model’s superior performance in these challenging real-world scenarios highlights its potential to make 3D object detection systems robust and adaptable.
[0118] 3) Validity of the preset 3D detection model:
[0119] Finally, the effectiveness of the preset 3D detection model in exploiting different cues is verified. First, the ability of the preset 3D detection model to handle flexible visual cues, including arbitrary views of the target object across scenes and time, is studied. For each video clip of the nuScenes validation set, the image patch of the target object in the first frame is used as a visual cue, and then the target object is detected and tracked in subsequent frames, and the recall rate is calculated to evaluate the performance of the model in handling arbitrary visual cues. The results show that despite the large differences in viewpoints and object poses between the first and subsequent frames, the recall rate does not drop significantly as the vehicle moves. This highlights the robustness of the preset 3D detection model in handling diverse visual cues.
[0120] Then, the performance of the preset 3D detection model in handling other interactive cues of objects in the target frame is evaluated, including object queries, boxes, points, and visual cues (corresponding image patches of objects in the target frame). The results show that the preset 3D detection model can effectively handle various forms of cues. In addition, compared with object query cues, the preset 3D detection model can effectively utilize human interactive cues (boxes, points, vision) for better 3D detection, with continuous improvements of 4.3%, 7.2%, and 5.4% in mAP, respectively. This highlights the potential of using human feedback to improve the online performance of offline detectors by performing error correction through interactive feedback.
[0121] Embodiment 5
[0122] Figure 6 This is a schematic diagram of the structure of an object detection device provided in Embodiment 5 of the present invention. Figure 6 As shown, the device includes: a target information determination module 501, an object frame determination module 502 and an object frame screening module 503, wherein:
[0123] A target information determination module, used to obtain an image frame of the current scene, and determine the target information of the image frame using a preset three-dimensional detection model, wherein the target information includes image feature information and depth information;
[0124] an object frame determination module, configured to generate an object frame and a corresponding confidence level of an object in the image frame according to a visual cue image and the target information using the preset 3D detection model, wherein the visual cue image includes a cue image associated with a missed object determined by a user, and the missed object is determined according to a historical output of the preset 3D detection model;
[0125] The object frame screening module is used to screen the object frame according to the confidence level to obtain a first target object frame.
[0126] The object detection device provided by the embodiment of the present invention inputs the visual prompt image determined by the user according to the historical output of the preset three-dimensional detection model, as well as the image feature information and depth information of the image frame into the preset three-dimensional detection model, thereby quickly responding to the failure of model detection, immediately correcting the detection error, identifying the target object that was not identified before, avoiding the risks caused by detection errors, and realizing reliable and safe autonomous driving.
[0127] Optionally, the device further comprises:
[0128] a first position information determining module, configured to determine position information of a set point in the image frame by using the preset three-dimensional detection model if the visual prompt image is empty;
[0129] A first position code determination module, configured to input the position information into a prompt encoder in the preset three-dimensional detection model to obtain a first position code;
[0130] The second target object frame determination module is used to input the target information and the first position code into the feature decoder in the preset three-dimensional detection model to obtain the object frame and the corresponding confidence of the object in the image frame, and screen the object frame according to the confidence to obtain the second target object frame.
[0131] Optionally, the device further comprises:
[0132] A second position information determination module is used to receive the associated points and / or associated frames of the missed objects in the historical image frames input by the user, and determine the position information of the associated points and / or the position information of the associated frames, before generating the object frames and corresponding confidence levels of the objects in the image frames according to the visual cue images and the target information using the preset three-dimensional detection model;
[0133] The first prompt image determination module is used to use the preset three-dimensional detection model to determine the visual prompt frame according to the position information of the associated point and / or the position information of the associated frame, and the target information of the historical image frame, and determine the image corresponding to the visual prompt frame in the historical image frame as the first visual prompt image.
[0134] Optionally, the object frame determination module includes:
[0135] A second position code determining unit, configured to input the first visual cue image into a cue encoder in the preset 3D detection model to obtain a second position code;
[0136] The first object frame determination unit is used to input the second position code and the target information into a feature decoder in the preset three-dimensional detection model to obtain the object frame of the object in the image frame and the corresponding confidence.
[0137] Optionally, the device further comprises:
[0138] A receiving module, configured to receive an image block associated with a missed object in a historical image frame input by a user before generating an object frame and a corresponding confidence level of the object in the image frame according to the visual cue image and the target information using the preset three-dimensional detection model;
[0139] The second prompt image determining module is used to determine the image block as a second visual prompt image.
[0140] Optionally, the object frame determination module includes:
[0141] a candidate position determination unit, configured to input the second visual cue image into a visual alignment module in the preset three-dimensional detection model to obtain a candidate position of the object corresponding to the second visual cue image in the image frame;
[0142] A third position code determining unit, configured to input the candidate position into a prompt encoder in the preset three-dimensional detection model to obtain a third position code;
[0143] The second object frame determination unit is used to input the third position code and the target information into a feature decoder in the preset three-dimensional detection model to obtain the object frame of the object in the image frame and the corresponding confidence.
[0144] Optionally, the second visual cue image is input into a visual alignment module in the preset three-dimensional detection model to obtain a candidate position of an object corresponding to the second visual cue image in the image frame, including: using the visual alignment module to determine the likelihood of image feature information in the target information and image features of the second visual cue image, and determining the candidate position of the object corresponding to the second visual cue image in the image frame based on the likelihood.
[0145] The object detection device provided in the embodiment of the present invention can execute the object detection method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.
[0146] Embodiment 6
[0147] Figure 7 A schematic diagram of an electronic device 70 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.
[0148] like Figure 7 As shown, the electronic device 60 includes at least one processor 61, and a memory connected to the at least one processor 61, such as a read-only memory (ROM) 62, a random access memory (RAM) 63, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 61 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 62 or the computer program loaded from the storage unit 68 to the random access memory (RAM) 63. In the RAM 63, various programs and data required for the operation of the electronic device 60 can also be stored. The processor 61, the ROM 62, and the RAM 63 are connected to each other via a bus 64. An input / output (I / O) interface 65 is also connected to the bus 64.
[0149] A number of components in the electronic device 60 are connected to the I / O interface 65, including: an input unit 66, such as a keyboard, a mouse, etc.; an output unit 67, such as various types of displays, speakers, etc.; a storage unit 78, such as a disk, an optical disk, etc.; and a communication unit 69, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 69 allows the electronic device 60 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0150] The processor 61 may be a variety of general and / or dedicated processing components with processing and computing capabilities. Some examples of the processor 61 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 61 performs the various methods and processes described above, such as an object detection method.
[0151] In some embodiments, the object detection method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 68. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 60 via the ROM 62 and / or the communication unit 69. When the computer program is loaded into the RAM 63 and executed by the processor 61, one or more steps of the object detection method described above may be performed. Alternatively, in other embodiments, the processor 61 may be configured to perform the object detection method in any other suitable manner (e.g., by means of firmware).
[0152] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0153] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0154] The computer device provided above can be used to execute the object detection method provided in any of the above embodiments, and has corresponding functions and beneficial effects.
[0155] Embodiment 7
[0156] In the context of the present invention, a computer readable storage medium may be a tangible medium, wherein the computer executable instructions, when executed by a computer processor, are used to perform an object detection method, the method comprising:
[0157] Acquire an image frame of the current scene, and determine target information of the image frame using a preset three-dimensional detection model, wherein the target information includes image feature information and depth information;
[0158] Using the preset 3D detection model, generating an object frame and a corresponding confidence level of the object in the image frame according to a visual cue image and the target information, wherein the visual cue image includes a cue image associated with a missed object determined by a user, and the missed object is determined according to a historical output of the preset 3D detection model;
[0159] The object frame is screened according to the confidence level to obtain a first target object frame.
[0160] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by an instruction execution system, device or equipment or used with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0161] The computer device provided above can be used to execute the object detection method provided in any of the above embodiments, and has corresponding functions and beneficial effects.
[0162] It is worth noting that in the embodiment of the above-mentioned object detection device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.
[0163] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims
1. An object detection method, characterized in that: include: Acquire an image frame of the current scene, and determine target information of the image frame using a preset three-dimensional detection model, wherein the target information includes image feature information and depth information; Using the preset 3D detection model, generating an object frame and a corresponding confidence level of the object in the image frame according to a visual cue image and the target information, wherein the visual cue image includes a cue image associated with a missed object determined by a user, and the missed object is determined according to a historical output of the preset 3D detection model; The object frame is screened according to the confidence level to obtain a first target object frame.
2. The method according to claim 1, characterized in that Also includes: If the visual prompt image is empty, determining the position information of the set point in the image frame using the preset three-dimensional detection model; Inputting the position information into a prompt encoder in the preset three-dimensional detection model to obtain a first position code; The target information and the first position code are input into a feature decoder in the preset three-dimensional detection model to obtain an object frame and a corresponding confidence level of the object in the image frame, and the object frame is screened according to the confidence level to obtain a second target object frame.
3. The method according to claim 1, characterized in that: Before using the preset three-dimensional detection model to generate an object frame and a corresponding confidence level of the object in the image frame according to the visual cue image and the target information, the method further includes: Receiving the associated points and / or associated frames of the missed objects in the historical image frames input by the user, and determining the position information of the associated points and / or the position information of the associated frames; Using the preset three-dimensional detection model, a visual prompt frame is determined according to the position information of the associated point and / or the position information of the associated frame, and the target information of the historical image frame, and the image corresponding to the visual prompt frame in the historical image frame is determined as the first visual prompt image.
4. The method according to claim 3, characterized in that The using the preset three-dimensional detection model to generate an object frame and a corresponding confidence level of the object in the image frame according to the visual cue image and the target information includes: Inputting the first visual cue image into a cue encoder in the preset three-dimensional detection model to obtain a second position code; The second position code and the target information are input into a feature decoder in the preset three-dimensional detection model to obtain an object frame and a corresponding confidence level of the object in the image frame.
5. The method according to claim 1, characterized in that Before using the preset three-dimensional detection model to generate an object frame and a corresponding confidence level of the object in the image frame according to the visual cue image and the target information, the method further includes: receiving an image block associated with a missed object in a historical image frame input by a user; The image block is determined as a second visual cue image.
6. The method according to claim 5, characterized in that The using the preset three-dimensional detection model to generate an object frame and a corresponding confidence level of the object in the image frame according to the visual cue image and the target information includes: Inputting the second visual cue image into a visual alignment module in the preset three-dimensional detection model to obtain a candidate position of an object corresponding to the second visual cue image in the image frame; Inputting the candidate position into a prompt encoder in the preset three-dimensional detection model to obtain a third position code; The third position code and the target information are input into a feature decoder in the preset three-dimensional detection model to obtain an object frame and a corresponding confidence level of the object in the image frame.
7. The method according to claim 6, characterized in that The step of inputting the second visual cue image into a visual alignment module in the preset three-dimensional detection model to obtain a candidate position of an object corresponding to the second visual cue image in the image frame includes: The visual alignment module is used to determine the likelihood of the image feature information in the target information and the image feature of the second visual cue image, and the candidate position of the object corresponding to the second visual cue image in the image frame is determined according to the likelihood.
8. An object detection device, characterized in that: include: A target information determination module, used to obtain an image frame of the current scene, and determine the target information of the image frame using a preset three-dimensional detection model, wherein the target information includes image feature information and depth information; an object frame determination module, configured to generate an object frame and a corresponding confidence level of an object in the image frame according to a visual cue image and the target information using the preset 3D detection model, wherein the visual cue image includes a cue image associated with a missed object determined by a user, and the missed object is determined according to a historical output of the preset 3D detection model; The object frame screening module is used to screen the object frame according to the confidence level to obtain a first target object frame.
9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the object detection method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the object detection method according to any one of claims 1 to 7 when executed.
