Target detection method and device, computer readable storage medium and electronic device

Abstract

The application belongs to the technical field of target detection, and particularly relates to a target detection method and device, a computer readable storage medium and an electronic device. The method comprises: acquiring an RGB image to be detected and a corresponding depth image; performing phased progressive feature extraction and fusion on the RGB image and the depth image to obtain fusion features of each phase; and performing target detection based on the fusion features of each phase to obtain a target detection result corresponding to the RGB image and the depth image. Through the application, phased progressive feature extraction and fusion can be performed on the RGB image and the depth image, so that the depth information can be deeply utilized, thereby effectively reducing misrecognition caused by mutual occlusion between objects.

Description

Technical Field

[0001] This application belongs to the field of target detection technology, and in particular relates to a target detection method, apparatus, computer-readable storage medium and electronic device. Background Technology

[0002] Object detection is a key task in computer vision. Its core objective is to accurately locate and identify instances of objects of interest in a given image. Its output is usually a combination of bounding boxes and categories.

[0003] In existing technologies, there are many mature object detection methods that can achieve good detection results in general scenarios. However, in practical object detection applications, chaotic scenarios with multiple objects occluding each other are often encountered, which can easily lead to confusion and misidentification during the object detection process. Summary of the Invention

[0004] In view of this, embodiments of this application provide a target detection method, apparatus, computer-readable storage medium, and electronic device to solve the problem that existing target detection methods are prone to misidentification in chaotic scenes where objects occlude each other.

[0005] A first aspect of this application provides a target detection method, which may include: Obtain the RGB image to be detected and the corresponding depth image; The RGB image and the depth image are subjected to progressive feature extraction and fusion in stages to obtain the fused features at each stage; Target detection is performed based on the fusion features of each stage to obtain target detection results corresponding to the RGB image and the depth image.

[0006] In one specific implementation of the first aspect, the step of performing phased progressive feature extraction and fusion on the RGB image and the depth image to obtain fused features at each stage may include: In the current stage, feature extraction is performed on the fusion features from the previous stage to obtain the RGB features of the current stage; Based on the feature size of the current stage, feature extraction is performed on the depth image to obtain the depth features of the current stage; The RGB features and depth features of the current stage are fused to obtain the fused features of the current stage.

[0007] In one specific implementation of the first aspect, the feature fusion of the RGB features and the depth features of the current stage to obtain the fused features of the current stage may include: By using the RGB features of the current stage to perform correlation perception on the depth features of the current stage, the first correlation perception result is obtained. By using the deep features of the current stage to perform correlation perception on the RGB features of the current stage, a second correlation perception result is obtained. The first correlation perception result and the second correlation perception result are fused to obtain a fused correlation perception result. The fusion correlation perception results are processed by convolutional block attention to obtain the fusion features at the current stage.

[0008] In one specific implementation of the first aspect, the step of performing correlation perception on the depth features of the current stage using the RGB features of the current stage to obtain a first correlation perception result may include: The first query matrix, the first key matrix, and the first value matrix are determined respectively; wherein, the first query matrix is ​​the query matrix corresponding to the RGB features of the current stage, the first key matrix is ​​the key matrix corresponding to the depth features of the current stage, and the first value matrix is ​​the value matrix corresponding to the depth features of the current stage. The first relevance perception result is determined based on the first query matrix, the first key matrix, and the first value matrix.

[0009] In one specific implementation of the first aspect, the step of performing correlation perception on the RGB features of the current stage using the depth features of the current stage to obtain a second correlation perception result may include: The second query matrix, the second key matrix, and the second value matrix are determined respectively; wherein, the second query matrix is ​​the query matrix corresponding to the depth features of the current stage, the second key matrix is ​​the key matrix corresponding to the RGB features of the current stage, and the second value matrix is ​​the value matrix corresponding to the RGB features of the current stage. The second relevance perception result is determined based on the second query matrix, the second key matrix, and the second value matrix.

[0010] In one specific implementation of the first aspect, the step of performing convolutional block attention processing on the fusion relevance perception result to obtain the fusion features at the current stage may include: The fusion correlation perception result is subjected to channel attention processing to obtain the channel attention processing result; Spatial attention processing is performed on the channel attention processing result to obtain the spatial attention processing result; The fusion relevance perception result and the spatial attention processing result are superimposed to obtain the fusion feature at the current stage.

[0011] In one specific implementation of the first aspect, after obtaining the target detection result corresponding to the RGB image and the depth image, the method may further include: Based on the target detection results, the preset object grasping mechanism is controlled to perform the task of grasping the target object.

[0012] A second aspect of the embodiments of this application provides a target detection device, which may include: The image acquisition module is used to acquire the RGB image to be detected and the corresponding depth image; The feature extraction and fusion module is used to perform progressive feature extraction and fusion on the RGB image and the depth image in stages to obtain fused features at each stage. The target detection module is used to perform target detection based on the fusion features of each stage, and obtain the target detection results corresponding to the RGB image and the depth image.

[0013] In one specific implementation of the second aspect, the feature extraction and fusion module may include: The RGB feature extraction submodule is used to extract features from the fused features of the previous stage in the current stage to obtain the RGB features of the current stage. The depth feature extraction submodule is used to extract features from the depth image based on the feature size of the current stage to obtain the depth features of the current stage. The feature fusion submodule is used to fuse the RGB features and the depth features of the current stage to obtain the fused features of the current stage.

[0014] In one specific implementation of the second aspect, the feature fusion submodule may include: The first correlation perception unit is used to perceive the correlation between the current stage's depth features and the current stage's RGB features to obtain the first correlation perception result. The second correlation perception unit is used to perceive the correlation of the RGB features of the current stage through the depth features of the current stage, and obtain the second correlation perception result. The perception result fusion unit is used to fuse the first correlation perception result and the second correlation perception result to obtain a fused correlation perception result; The convolutional block attention processing unit is used to perform convolutional block attention processing on the fusion correlation perception result to obtain the fusion features at the current stage.

[0015] In one specific implementation of the second aspect, the first relevance sensing unit may be specifically used to: determine a first query matrix, a first key matrix, and a first value matrix respectively; wherein, the first query matrix is ​​a query matrix corresponding to the RGB features of the current stage, the first key matrix is ​​a key matrix corresponding to the depth features of the current stage, and the first value matrix is ​​a value matrix corresponding to the depth features of the current stage; and determine the first relevance sensing result based on the first query matrix, the first key matrix, and the first value matrix.

[0016] In one specific implementation of the second aspect, the second relevance sensing unit may be specifically used to: determine a second query matrix, a second key matrix, and a second value matrix respectively; wherein, the second query matrix is ​​a query matrix corresponding to the depth features of the current stage, the second key matrix is ​​a key matrix corresponding to the RGB features of the current stage, and the second value matrix is ​​a value matrix corresponding to the RGB features of the current stage; and determine the second relevance sensing result based on the second query matrix, the second key matrix, and the second value matrix.

[0017] In one specific implementation of the second aspect, the convolutional block attention processing unit can be specifically used to: perform channel attention processing on the fusion correlation perception result to obtain a channel attention processing result; perform spatial attention processing on the channel attention processing result to obtain a spatial attention processing result; and superimpose the fusion correlation perception result and the spatial attention processing result to obtain the fusion feature of the current stage.

[0018] In one specific implementation of the second aspect, the target detection device may further include: The grasping task execution module is used to control a preset object grasping mechanism to perform a grasping task on the target object based on the target detection result.

[0019] A third aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the above-described target detection methods.

[0020] A fourth aspect of this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any of the above-described target detection methods.

[0021] A fifth aspect of this application provides a computer program product that, when run on an electronic device, causes the electronic device to perform the steps of any of the above-described target detection methods.

[0022] The beneficial effects of this application embodiment compared with the prior art are as follows: This application embodiment acquires an RGB image to be detected and a corresponding depth image; performs phased progressive feature extraction and fusion on the RGB image and the depth image to obtain fused features at each stage; and performs target detection based on the fused features at each stage to obtain target detection results corresponding to the RGB image and the depth image. Through this application embodiment, phased progressive feature extraction and fusion of RGB images and depth images can be performed, allowing depth information to be utilized effectively, thereby effectively reducing misidentification caused by mutual occlusion between objects. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A diagram illustrating a chaotic scene where multiple objects occlude each other. Figure 2 This is a flowchart of one embodiment of a target detection method in this application. Figure 3 This is a flowchart illustrating the processing steps at the current stage. Figure 4 This is a schematic diagram of the fusion module; Figure 5 This is a schematic flowchart illustrating feature fusion of RGB and depth features; Figure 6 This is a schematic diagram illustrating the correlation perception of depth features using RGB features; Figure 7 This is a schematic diagram of the complete target detection process; Figure 8 This is a structural diagram of one embodiment of a target detection device according to the present application. Figure 9 This is a schematic block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0025] To make the inventive objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0027] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0028] It should also be further understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0029] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [the described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [the described condition or event] is detected," or "in response to detection of [the described condition or event]."

[0030] Furthermore, in the description of this application, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0031] Object detection is a key task in computer vision. Its core objective is to accurately locate and identify instances of objects of interest in a given image. Its output is usually a combination of bounding boxes and categories.

[0032] In existing technologies, there are many mature object detection methods that can achieve good detection results in general scenarios. However, in practical object detection applications, chaotic scenarios with multiple objects occluding each other are frequently encountered, which can easily lead to confusion and misidentification during the object detection process. For example... Figure 1 As shown, the two objects in the box are overlapping each other and can easily be mistaken for one object.

[0033] In view of this, embodiments of this application provide a target detection method, apparatus, computer-readable storage medium, and electronic device to solve the problem that existing target detection methods are prone to misidentification in chaotic scenes where objects occlude each other.

[0034] In this embodiment, RGB images and depth images can be progressively extracted and fused in stages, so that depth information can be utilized in depth, thereby effectively reducing misidentification caused by mutual occlusion between objects.

[0035] Please see Figure 2 One embodiment of a target detection method in this application may include: Step S201: Obtain the RGB image to be detected and the corresponding depth image.

[0036] Compared to RGB images, which are prone to occlusion and confusion, depth images contain depth information, which can help distinguish the contours of objects. In the embodiments of this application, RGB images and depth images can be combined for object detection.

[0037] Step S202: Perform phased progressive feature extraction and fusion on the RGB image and depth image to obtain the fused features at each stage.

[0038] The target detection framework used in the embodiments of this application may include, but is not limited to, a backbone network, a neck network, and a head.

[0039] The process of progressive feature extraction and fusion in stages for RGB and depth images can be completed within the backbone network. The specific network structure of the backbone network can be flexibly configured according to actual needs, and this embodiment does not impose specific limitations on it. As an example, the YOLOv8S network is used here for illustration. The YOLOv8S network processing includes three stages, denoted as stage 1, stage 2, and stage 3, where progressive feature extraction and fusion can be performed. It is easy to understand that other network structures may have more or fewer stages.

[0040] Taking any given stage as an example, let's call it the current stage. Its specific processing procedure is as follows: Figure 3 As shown: Step S2021: In the current stage, feature extraction is performed on the fusion features of the previous stage to obtain the RGB features of the current stage.

[0041] If the current stage is the first stage, feature extraction can be performed directly on the RGB image to obtain the RGB features of the first stage; if the current stage is the second stage, feature extraction can be performed on the fused features of the first stage to obtain the RGB features of the second stage; if the current stage is the third stage, feature extraction can be performed on the fused features of the second stage to obtain the RGB features of the third stage; and so on.

[0042] Step S2022: Based on the feature size of the current stage, perform feature extraction on the depth image to obtain the depth features of the current stage.

[0043] Taking the YOLOv8S network as an example, the feature size of its different stages is different: the feature size of the first stage is 80×80×128, the feature size of the second stage is 40×40×256, and the feature size of the third stage is 20×20×512.

[0044] In this embodiment, feature extraction of a depth image can be performed using a preset depth block. Specifically, the depth image can first be downsampled to align with the feature size of the current stage. Then, a convolutional network (SimpleNet) can be used to extract features from the depth image. The specific network structure of SimpleNet can be flexibly configured according to actual conditions, and this embodiment does not impose specific limitations on it. As an example, SimpleNet may include five 3×3 convolutions, with the first four convolutions having 128 channels, and the last convolution having channels aligned with the feature size of the current stage. In one specific implementation of this embodiment, the convolution weights in the depth block can be initialized with all zeros (Zero-init) to ensure that it can first learn some RGB features and then gradually learn depth features.

[0045] Step S2023: Perform feature fusion on the RGB features and the depth features of the current stage to obtain the fused features of the current stage.

[0046] In the embodiments of this application, it can be achieved through, as follows Figure 4The fusion module (Eff-Fusion) shown performs feature fusion on the RGB features and depth features of the current stage. Its specific processing procedure is as follows: Figure 5 As shown: Step S20231: Perform correlation perception on the depth features of the current stage using the RGB features of the current stage to obtain the first correlation perception result.

[0047] Specifically, such as Figure 6 As shown, the first query matrix (Q), first key matrix (K), and first value matrix (V) can be determined first. The first query matrix is ​​the query matrix corresponding to the RGB features (N×H×W×C) of the current stage; the first key matrix is ​​the key matrix corresponding to the depth features (N×H×W×C) of the current stage; and the first value matrix is ​​the value matrix corresponding to the depth features of the current stage. As an example, a 1×1 convolution can be applied to the RGB features to obtain the first query matrix, and different 1×1 convolutions can be applied to the depth features to obtain the first key matrix and the first value matrix. To reduce computational cost, the channel dimension can be reduced to 1 / 8.

[0048] Then, the first relevance perception result can be determined based on the first query matrix, the first key matrix, and the first value matrix. Specifically, to calculate the relevance of QK, Q can be reshaped to obtain an N×HW×C / 8 matrix, and K can be reshaped to obtain an N×HW×C / 8 matrix. Then, its dimensions are swapped using transpose to obtain an N×C / 8×HW matrix. Matrix multiplication can then be used to obtain the relevance of QK. Softmax can be used for normalization. To prevent gradient vanishing due to excessive focus on a certain relevance, the relevance result of QK can be divided by... ,in, Let C be the channel dimension. After obtaining the correlation results of QK, V can be used to emphasize which correlation is more desirable. Finally, a 1×1 convolution with C channels is used to restore the original dimension. Through the above operations, the perception of depth features by RGB features can be achieved. The calculation process is shown in the following formula:

[0049] To avoid excessive loss of RGB features, the original RGB features will be added to the final output to obtain the first correlation perception result.

[0050] Step S20232: Perform correlation perception on the RGB features of the current stage using the depth features of the current stage to obtain the second correlation perception result.

[0051] Similar to step S20231, the second query matrix (Q), the second key matrix (K), and the second value matrix (V) can be determined first. The second query matrix is ​​the query matrix corresponding to the depth features of the current stage, the second key matrix is ​​the key matrix corresponding to the RGB features of the current stage, and the second value matrix is ​​the value matrix corresponding to the RGB features of the current stage. As an example, a 1×1 convolution can be applied to the depth features to obtain the second query matrix, and different 1×1 convolutions can be applied to the RGB features to obtain the second key matrix and the second value matrix. To reduce computation, the channel dimension can be reduced to 1 / 8. Then, the second relevance perception result can be determined based on the second query matrix, the second key matrix, and the second value matrix. The specific calculation process can be found in the detailed description in step S20231, and will not be repeated here.

[0052] Step S20233: Fuse the first correlation perception result and the second correlation perception result to obtain the fused correlation perception result.

[0053] In this embodiment, the first and second correlation perception results can be concatenated (concat) with a dimension of 2C. Then, a single convolution is used to restore it to the previous dimension of C. Finally, three 3×3 convolutions can be used to combine the previous fusion results to complete further fusion and obtain the fused correlation perception result.

[0054] Step S20234: Perform convolutional block attention processing on the fusion correlation perception results to obtain the fusion features at the current stage.

[0055] In this embodiment of the application, spatial features can be further mined through a preset Convolutional Block Attention Module (CBAM), making it easier to perceive two separated objects.

[0056] The convolutional block attention module can include channel attention units (C-Attention) and spatial attention units (S-Attention). Specifically, in the channel attention unit, the fusion correlation perception result can be processed with channel attention, i.e., mean pooling and max pooling are calculated along the H×W dimension to complete channel attention modeling, and the input feature map is weighted according to the channel dimension to highlight key channel information, thus obtaining the channel attention processing result. In the spatial attention unit, the channel attention processing result can be processed with spatial attention, i.e., mean pooling and max pooling are calculated along the C dimension to complete spatial attention modeling, and spatial dimension weighting calibration is performed on the weighted feature map to strengthen effective region features, thus obtaining the spatial attention processing result. Finally, the fusion correlation perception result and the spatial attention processing result can be superimposed to obtain the fused feature at the current stage.

[0057] It should be noted that the structure of the fusion module is the same in each stage, but the feature sizes processed are different, and they should be aligned with the feature sizes of the current stage.

[0058] Step S203: Perform target detection based on the fusion features of each stage to obtain target detection results corresponding to the RGB image and depth image.

[0059] In this embodiment, a neck network can be used to further optimize the fused features at each stage. The specific network structure of the neck network can be flexibly configured according to actual conditions, and may include, but is not limited to, Path Aggregation Network (PANet), etc. This embodiment does not impose specific limitations on this. The output of the neck network is processed by the detection head to obtain the final target detection result, including the target bounding box and category.

[0060] Figure 7 The diagram shows a complete target detection process, with the three-stage progressive feature extraction and fusion as an example.

[0061] In the first stage, features are extracted from the RGB image to obtain the RGB features of the first stage; features are extracted from the depth image through a depth module that is aligned with the feature size of the first stage to obtain the depth features of the first stage; and the RGB features and depth features of the first stage are fused through a fusion module (Eff-Fusion1) that is aligned with the feature size of the first stage to obtain the fused features of the first stage.

[0062] In the second stage, feature extraction is performed on the fusion features from the first stage to obtain the RGB features of the second stage; feature extraction is performed on the depth image using a depth module that is aligned with the feature size of the second stage to obtain the depth features of the second stage; and feature fusion is performed on the RGB features of the second stage and the depth features of the second stage using a fusion module (Eff-Fusion2) that is aligned with the feature size of the second stage to obtain the fusion features of the second stage.

[0063] In the third stage, feature extraction is performed on the fusion features from the second stage to obtain the RGB features of the third stage; feature extraction is performed on the depth image using a depth module that is aligned with the feature size of the third stage to obtain the depth features of the third stage; and feature fusion is performed on the RGB features and depth features of the third stage using a fusion module (Eff-Fusion3) that is aligned with the feature size of the third stage to obtain the fusion features of the third stage.

[0064] The fusion features from the first stage, the second stage, and the third stage are then processed by the neck network and the head to obtain the final target detection result.

[0065] In one specific implementation of this application, after obtaining the target detection result, a preset object grasping mechanism can be controlled to perform a grasping task on the target object based on the target detection result. The object grasping mechanism may include, but is not limited to, a robot's end effector.

[0066] In summary, this application embodiment acquires an RGB image to be detected and its corresponding depth image; performs phased progressive feature extraction and fusion on the RGB image and depth image to obtain fused features at each stage; and performs target detection based on the fused features at each stage to obtain target detection results corresponding to the RGB image and depth image. Through this application embodiment, phased progressive feature extraction and fusion of RGB image and depth image can be performed, allowing depth information to be utilized effectively, thereby reducing misidentification caused by mutual occlusion between objects.

[0067] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0068] Corresponding to the target detection method described in the above embodiments, Figure 8 This diagram illustrates a structural diagram of an embodiment of a target detection device provided in this application.

[0069] In this embodiment, a target detection device may include: Image acquisition module 801 is used to acquire the RGB image to be detected and the corresponding depth image; The feature extraction and fusion module 802 is used to perform progressive feature extraction and fusion on the RGB image and the depth image in stages to obtain fused features at each stage. The target detection module 803 is used to perform target detection based on the fusion features of each stage to obtain the target detection result corresponding to the RGB image and the depth image.

[0070] In one specific implementation of this application embodiment, the feature extraction and fusion module may include: The RGB feature extraction submodule is used to extract features from the fused features of the previous stage in the current stage to obtain the RGB features of the current stage. The depth feature extraction submodule is used to extract features from the depth image based on the feature size of the current stage to obtain the depth features of the current stage. The feature fusion submodule is used to fuse the RGB features and the depth features of the current stage to obtain the fused features of the current stage.

[0071] In one specific implementation of this application embodiment, the feature fusion submodule may include: The first correlation perception unit is used to perceive the correlation between the current stage's depth features and the current stage's RGB features to obtain the first correlation perception result. The second correlation perception unit is used to perceive the correlation of the RGB features of the current stage through the depth features of the current stage, and obtain the second correlation perception result. The perception result fusion unit is used to fuse the first correlation perception result and the second correlation perception result to obtain a fused correlation perception result; The convolutional block attention processing unit is used to perform convolutional block attention processing on the fusion correlation perception result to obtain the fusion features at the current stage.

[0072] In one specific implementation of this application embodiment, the first relevance perception unit may be specifically used to: determine a first query matrix, a first key matrix, and a first value matrix respectively; wherein, the first query matrix is ​​a query matrix corresponding to the RGB features of the current stage, the first key matrix is ​​a key matrix corresponding to the depth features of the current stage, and the first value matrix is ​​a value matrix corresponding to the depth features of the current stage; and determine the first relevance perception result based on the first query matrix, the first key matrix, and the first value matrix.

[0073] In one specific implementation of this application embodiment, the second relevance perception unit may be specifically used to: determine a second query matrix, a second key matrix, and a second value matrix respectively; wherein, the second query matrix is ​​a query matrix corresponding to the depth features of the current stage, the second key matrix is ​​a key matrix corresponding to the RGB features of the current stage, and the second value matrix is ​​a value matrix corresponding to the RGB features of the current stage; and determine the second relevance perception result based on the second query matrix, the second key matrix, and the second value matrix.

[0074] In one specific implementation of this application, the convolutional block attention processing unit can be specifically used to: perform channel attention processing on the fusion correlation perception result to obtain a channel attention processing result; perform spatial attention processing on the channel attention processing result to obtain a spatial attention processing result; and superimpose the fusion correlation perception result and the spatial attention processing result to obtain the fusion feature of the current stage.

[0075] In one specific implementation of this application embodiment, the target detection device may further include: The grasping task execution module is used to control a preset object grasping mechanism to perform a grasping task on the target object based on the target detection result.

[0076] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0077] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0078] Figure 9 A schematic block diagram of an electronic device provided in an embodiment of this application is shown. For ease of explanation, only the parts related to the embodiment of this application are shown.

[0079] like Figure 9 As shown, the electronic device 9 of this embodiment includes: a processor 90, a memory 91, and a computer program 92 stored in the memory 91 and executable on the processor 90. When the processor 90 executes the computer program 92, it implements the steps in the various target detection method embodiments described above, for example... Figure 2 Steps S201 to S203 are shown. Alternatively, when the processor 90 executes the computer program 92, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 8 The functions of modules 801 to 803 are shown.

[0080] For example, the computer program 92 may be divided into one or more modules / units, which are stored in the memory 91 and executed by the processor 90 to complete this application. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program 92 in the electronic device 9.

[0081] The electronic device 9 may include, but is not limited to, computing devices such as mobile phones, tablets, desktop computers, laptops, handheld computers, robots, and servers. Those skilled in the art will understand that... Figure 9 This is merely an example of electronic device 9 and does not constitute a limitation on electronic device 9. It may include more or fewer components than shown, or combine certain components, or different components. For example, electronic device 9 may also include input / output devices, network access devices, buses, etc.

[0082] The processor 90 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0083] The memory 91 can be an internal storage unit of the electronic device 9, such as a hard disk or memory. The memory 91 can also be an external storage device of the electronic device 9, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory 91 can include both internal and external storage units of the electronic device 9. The memory 91 is used to store the computer program and other programs and data required by the electronic device 9. The memory 91 can also be used to temporarily store data that has been output or will be output.

[0084] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0085] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0086] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0087] In the embodiments provided in this application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings or direct couplings or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0088] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0089] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0090] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable storage medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc. It should be noted that the content included in the computer-readable storage medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable storage medium does not include electrical carrier signals and telecommunication signals.

[0091] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A target detection method, characterized in that, include: Obtain the RGB image to be detected and the corresponding depth image; The RGB image and the depth image are subjected to progressive feature extraction and fusion in stages to obtain fused features at each stage, including: in the current stage, feature extraction is performed on the fused features of the previous stage to obtain the RGB features of the current stage; based on the feature size of the current stage, feature extraction is performed on the depth image to obtain the depth features of the current stage; and feature fusion is performed on the RGB features and the depth features of the current stage to obtain the fused features of the current stage. Target detection is performed based on the fusion features of each stage to obtain target detection results corresponding to the RGB image and the depth image.

2. The target detection method according to claim 1, characterized in that, The process of fusing the RGB features and depth features of the current stage to obtain the fused features of the current stage includes: By using the RGB features of the current stage to perform correlation perception on the depth features of the current stage, the first correlation perception result is obtained. By using the deep features of the current stage to perform correlation perception on the RGB features of the current stage, a second correlation perception result is obtained. The first correlation perception result and the second correlation perception result are fused to obtain a fused correlation perception result. The fusion correlation perception results are processed by convolutional block attention to obtain the fusion features at the current stage.

3. The target detection method according to claim 2, characterized in that, The step of performing correlation perception on the depth features of the current stage using the RGB features of the current stage to obtain the first correlation perception result includes: The first query matrix, the first key matrix, and the first value matrix are determined respectively; wherein, the first query matrix is ​​the query matrix corresponding to the RGB features of the current stage, the first key matrix is ​​the key matrix corresponding to the depth features of the current stage, and the first value matrix is ​​the value matrix corresponding to the depth features of the current stage. The first relevance perception result is determined based on the first query matrix, the first key matrix, and the first value matrix.

4. The target detection method according to claim 2, characterized in that, The step of performing correlation perception on the RGB features of the current stage using the depth features of the current stage to obtain a second correlation perception result includes: The second query matrix, the second key matrix, and the second value matrix are determined respectively; wherein, the second query matrix is ​​the query matrix corresponding to the depth features of the current stage, the second key matrix is ​​the key matrix corresponding to the RGB features of the current stage, and the second value matrix is ​​the value matrix corresponding to the RGB features of the current stage. The second relevance perception result is determined based on the second query matrix, the second key matrix, and the second value matrix.

5. The target detection method according to claim 2, characterized in that, The convolutional block attention processing of the fusion relevance perception result to obtain the fusion features at the current stage includes: The fusion correlation perception result is subjected to channel attention processing to obtain the channel attention processing result; Spatial attention processing is performed on the channel attention processing result to obtain the spatial attention processing result; The fusion relevance perception result and the spatial attention processing result are superimposed to obtain the fusion feature at the current stage.

6. The target detection method according to any one of claims 1 to 5, characterized in that, After obtaining the target detection results corresponding to the RGB image and the depth image, the method further includes: Based on the target detection results, the preset object grasping mechanism is controlled to perform the task of grasping the target object.

7. A target detection device, characterized in that, include: The image acquisition module is used to acquire the RGB image to be detected and the corresponding depth image; The feature extraction and fusion module is used to perform progressive feature extraction and fusion of the RGB image and the depth image in stages to obtain fused features at each stage, including: in the current stage, extracting features from the fused features of the previous stage to obtain the RGB features of the current stage; extracting features from the depth image based on the feature size of the current stage to obtain the depth features of the current stage; and fusing the RGB features and the depth features of the current stage to obtain the fused features of the current stage. The target detection module is used to perform target detection based on the fusion features of each stage, and obtain the target detection results corresponding to the RGB image and the depth image.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the target detection method as described in any one of claims 1 to 6.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the target detection method as described in any one of claims 1 to 6.

Images