Target detection method, device, storage medium and electronic device

Abstract

The present application relates to the field of artificial intelligence, and specifically to a target detection method, device, storage medium and electronic device, which can be applied to various scenarios such as industrial detection, cloud technology, artificial intelligence, smart transportation, and assisted driving. In the method, based on a plurality of preset feature map sizes, the corresponding color feature map and depth feature map are extracted from the image to be detected, and based on a plurality of feature map sizes, feature fusion processing is performed on the corresponding color feature map and depth feature map, respectively, to obtain at least one fused feature map corresponding to each of the plurality of feature map sizes and the initial target features of the image to be detected, and the obtained fused feature maps are used to perform feature interaction processing on the initial target features to obtain fused target features, and then based on the fused target features, at least one target object belonging to the corresponding target detection category is determined among N candidate objects to obtain a target detection result, thereby improving detection accuracy and stability.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a target detection method, device, storage medium and electronic device. Background Art

[0002] With the widespread application of image processing technology in various industrial fields, target detection of industrial objects has been derived based on image processing technology. For example, in the field of lithium battery quality inspection, image processing technology can be used to identify the defect categories of lithium batteries and achieve target detection of lithium batteries.

[0003] In the related art, when performing target detection on lithium batteries, the following method is usually adopted: for the image to be detected containing the lithium battery, the target feature of the lithium battery is obtained based on at least the color feature of the image to be detected, and then the morphological analysis of the lithium battery is completed based on the target feature to identify the defect category of the lithium battery, thereby obtaining the target detection result of the lithium battery.

[0004] However, affected by acquisition environment factors such as acquisition equipment, acquisition light, and acquisition distance, the same lithium battery in the physical space may display different forms (such as size, color, texture, etc.) in two different images to be detected under different acquisition environments; therefore, when the above scheme is used for target detection, it will be difficult to obtain stable target features due to the influence of acquisition environment factors, thereby seriously reducing the accuracy of the target detection results. Summary of the invention

[0005] The present application provides a target detection method, device, storage medium and electronic device for improving the stability and accuracy of target detection.

[0006] In a first aspect, the present application provides a target detection method, the method comprising:

[0007] Based on the preset multiple feature map sizes, the corresponding color feature map and depth feature map are extracted for the image to be detected;

[0008] Based on the multiple feature map sizes, respectively perform feature fusion processing on the corresponding color feature map and depth feature map to obtain at least one fused feature map corresponding to each of the multiple feature map sizes;

[0009] Acquire initial target features of the image to be detected, wherein the initial target features represent: initial association relationships between N candidate objects and target detection categories preset for the image to be detected;

[0010] Using each obtained fusion feature map, perform feature interaction processing on the initial target feature to obtain a fusion target feature;

[0011] Based on the fused target features, at least one target object belonging to a corresponding target detection category is determined among the N candidate objects to obtain a target detection result of the image to be detected.

[0012] In a second aspect, the present application provides a target detection device, the device comprising:

[0013] An extraction unit, used for extracting corresponding color feature maps and depth feature maps for the image to be detected based on a plurality of preset feature map sizes;

[0014] A fusion unit, configured to perform feature fusion processing on the corresponding color feature map and the depth feature map based on the multiple feature map sizes, to obtain at least one fused feature map corresponding to each of the multiple feature map sizes;

[0015] An interaction unit is used to obtain an initial target feature of the image to be detected, wherein the initial target feature represents: an initial association relationship between N candidate objects and each target detection category preset for the image to be detected; and using each fusion feature map obtained, performing feature interaction processing on the initial target feature to obtain a fusion target feature;

[0016] An obtaining unit is used to determine at least one target object belonging to a corresponding target detection category among the N candidate objects based on the fused target feature, so as to obtain a target detection result of the image to be detected.

[0017] Optionally, the image to be detected includes: a color image to be detected and a depth image to be detected collected for the same target scene;

[0018] Then the extraction unit is specifically used for:

[0019] Based on the preset multiple feature map sizes, the following operations are performed on the color map to be detected and the depth map to be detected:

[0020] Based on a preset feature map size, respectively performing image size transformation processing on the color map to be detected and the initial depth map to be detected to obtain a corresponding initial color feature map and an initial depth feature map;

[0021] The initial color feature map and the initial depth feature map are respectively subjected to interactive processing of pixels within the map to obtain corresponding color feature maps and depth feature maps.

[0022] Optionally, the extraction unit is used to perform image size transformation processing on the color map to be detected and the depth map to be detected based on a preset feature map size, respectively, to obtain a corresponding initial color feature map and an initial depth feature map, specifically for:

[0023] Based on a preset feature map size, the following operations are performed on the two attribute maps to be detected, namely, the color map to be detected and the depth map to be detected:

[0024] Based on a size difference value between a size of an attribute graph to be detected and a size of the feature graph, dividing the attribute graph to be detected into a plurality of image blocks to be detected;

[0025] At least one initial pixel point contained in each of the multiple image blocks to be detected is mapped to a corresponding feature space to obtain a corresponding initial attribute feature map, where the initial attribute feature map is an initial color feature map or an initial depth feature map.

[0026] Optionally, the extraction unit is used to perform interactive processing of pixels in the initial color feature map and the initial depth feature map, respectively, to obtain corresponding color feature maps and depth feature maps, specifically for:

[0027] The following operations are performed on the two initial attribute feature maps, the initial color feature map and the initial depth feature map, respectively:

[0028] Based on a preset window size, an initial attribute feature map is divided into a plurality of window blocks; wherein each window block includes at least one candidate pixel point in the initial attribute feature map;

[0029] Self-attention feature interaction processing is performed on at least one candidate pixel point contained in each of the multiple window tiles, and cross-attention feature interaction processing is performed at least between the multiple window tiles to obtain a corresponding attribute feature map, which is a color feature map or a depth feature map.

[0030] Optionally, the fusion unit is specifically used for:

[0031] For each feature map size, perform the following operations:

[0032] Obtain a color feature map and a depth feature map that match the size of the feature map;

[0033] In the color feature map, based on the K depth pixels in the depth feature map, feature fusion processing is performed on each color pixel to obtain a corresponding first fusion feature map;

[0034] In the depth feature map, based on the K color pixels in the color feature map, feature fusion processing is performed on each depth pixel to obtain a corresponding second fusion feature map;

[0035] The first fused feature map and the second fused feature map are both used as fused feature maps corresponding to the feature map size.

[0036] Optionally, the fusion unit is used to perform feature fusion processing on each color pixel in the color feature map based on K depth pixels in the depth feature map to obtain a corresponding first fusion feature map, specifically for:

[0037] For each color pixel in the color feature map, perform the following operations respectively:

[0038] Based on the association relationship between the color pixel and each depth pixel in the depth feature map, K depth pixels satisfying the first association condition and their corresponding first fusion weights are obtained; wherein each first fusion weight represents: the association relationship between the corresponding depth pixel and the color pixel;

[0039] Based on the obtained K first fusion weights, the K depth pixels and the color pixels are fused to obtain first fusion points corresponding to the color pixels;

[0040] Based on the first fusion point corresponding to each color pixel, a corresponding first fusion feature map is obtained.

[0041] Optionally, the fusion unit is used to obtain K depth pixels satisfying a first association condition and their corresponding first fusion weights based on the association relationship between the color pixel and each depth pixel in the depth feature map, specifically for:

[0042] Based on the first relationship prediction parameter, first association information between each of the depth pixels and the color pixel is obtained; wherein the first relationship prediction parameter represents: the association relationship between the color pixel and each of the depth pixels in the depth feature map;

[0043] Select K first association information satisfying the first association condition from among the obtained first association information, and obtain corresponding K depth pixel points;

[0044] Based on the first weight prediction parameter, the influence degree of each depth pixel on the color pixel is calculated respectively, and the first fusion weight corresponding to each of the K depth pixels is obtained; wherein the first weight prediction parameter represents: the correlation relationship between the color pixel and the K depth pixel.

[0045] Optionally, the fusion unit is used to perform feature fusion processing on each depth pixel in the depth feature map based on K color pixels in the color feature map to obtain a corresponding second fused feature map, specifically for:

[0046] For each depth pixel in the depth feature map, perform the following operations respectively:

[0047] Based on the association relationship between the depth pixel and each color pixel in the color feature map, K color pixels satisfying the second association condition and their corresponding second fusion weights are obtained; wherein each second fusion weight represents: the association relationship between the corresponding color pixel and the depth pixel;

[0048] Based on the obtained K second fusion weights, the K color pixels and the depth pixel are fused to obtain a second fusion point corresponding to the depth pixel;

[0049] Based on the second fusion point corresponding to each depth pixel point, a corresponding second fusion feature map is obtained.

[0050] Optionally, the fusion unit is used to obtain K color pixels satisfying a second association condition and their corresponding second fusion weights based on the association relationship between the depth pixel and each color pixel in the color feature map, specifically for:

[0051] Based on the second relationship prediction parameter, second association information between each of the color pixels and the depth pixel is obtained; wherein the second relationship prediction parameter represents: the association relationship between the depth pixel and each of the color pixels in the color feature map;

[0052] Select K pieces of second association information satisfying the second association condition from among the obtained pieces of second association information, and obtain corresponding K color pixel points;

[0053] Based on the second weight prediction parameter, the influence of each color pixel on the depth pixel is calculated respectively to obtain the second fusion weight corresponding to each of the K color pixels; wherein the second weight prediction parameter represents: the correlation relationship between the depth pixel and the K color pixels.

[0054] Optionally, the interaction unit is specifically used to:

[0055] Using each acquired fusion feature map, the initial target feature is iteratively updated in turn to obtain a corresponding updated target feature; wherein, in one updating process, feature association information between a fusion feature map and the initial target feature is acquired, and based on the positive influence of the feature association information on the initial target feature, the initial target feature is updated;

[0056] Based on the object association information between the N candidate objects represented by the updated target feature, the updated target feature is adjusted to obtain a fused target feature.

[0057] Optionally, the interaction unit is used to obtain feature association information between a fused feature map and the initial target feature, and update the initial target feature based on the positive impact of the feature association information on the initial target feature, specifically for:

[0058] For each initial feature point in the initial target feature, the following operations are performed respectively: based on the association relationship between the initial feature point and each fused pixel point in a fused feature map, J fused pixel points that meet the fusion association condition and their corresponding fusion association weights are obtained; wherein each fusion association weight represents: the association relationship between the corresponding fused pixel point and the initial feature point;

[0059] Based on the obtained J fusion association weights, the J fusion pixel points and the initial feature points are fused to obtain updated feature points corresponding to the initial feature points.

[0060] Optionally, the interaction unit is used to adjust the updated target feature based on the object association information between the N candidate objects represented by the updated target feature to obtain the fused target feature, specifically for:

[0061] For the N candidate objects represented by the updated target feature, based on the object association relationship between each candidate object and the N candidate objects, obtaining the object weight corresponding to each of the N candidate objects;

[0062] The obtained N object weights are used to perform weighted sum processing on the updated target features to obtain fused target features.

[0063] Optionally, the obtaining unit is specifically used to:

[0064] Acquire detection parameters associated with each target detection category; wherein each detection parameter represents: the degree of association between each historical object and the corresponding target detection category;

[0065] Based on the acquired detection parameters, feature analysis processing is performed on the fused target features to obtain detection probabilities that each of the N candidate objects belongs to each target detection category;

[0066] For the N candidate objects, the following operations are performed respectively: when the maximum detection probability associated with each of the candidate objects meets the preset detection probability, the candidate object is determined to be a target object belonging to the target detection category corresponding to the maximum detection probability.

[0067] In a third aspect, the present application provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, any one of the target detection methods described in the first aspect is implemented.

[0068] In a fourth aspect, the present application provides a computer storage medium, wherein the computer-readable storage medium stores computer program instructions, and the computer program instructions are executed by a processor to perform any one of the target detection methods described in the first aspect.

[0069] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising computer program instructions, which, when executed by a processor, implement any one of the target detection methods in the first aspect.

[0070] The beneficial effects of this application are as follows:

[0071] In an embodiment of the present application, a target detection method is provided to improve the accuracy and stability of target detection results. Specifically, the computing device first extracts corresponding color feature maps and depth feature maps for the image to be detected based on a plurality of preset feature map sizes, and performs feature fusion processing on the corresponding color feature maps and depth feature maps based on the plurality of feature map sizes, and obtains at least one fused feature map corresponding to each of the plurality of feature map sizes. In this way, for each fused feature map, a depth feature map is fused, and the depth feature map can better characterize the three-dimensional form of the object. In this way, in some poor or unstable acquisition environments, the problem that the object form is unstable based on the color feature map alone, resulting in poor subsequent target detection results, can be solved, so as to improve the accuracy of the target detection results; for each feature map size, its corresponding fused feature map is generated based on the fusion of the corresponding color feature map and the depth feature map, which can better explore the correlation between the two to retain more effective object detail information; for different feature map sizes, the morphological information of the object can be fully reflected, thereby improving the detection performance of subsequent target detection.

[0072] Secondly, the computing device obtains the initial target features of the image to be detected, and uses the obtained fusion feature maps to perform feature interaction processing on the initial target features to obtain fusion target features. The initial target feature characterizes: the initial association relationship between N candidate objects and each target detection category preset for the image to be detected; in this way, an updateable initial target feature is introduced to transform the combination problem between each fusion feature map into an update problem for the initial target feature, so that based on the initial target feature, feature interaction processing is performed on each fusion feature map obtained to achieve the update of the initial target feature. In this way, the obtained fusion target feature can retain the effective information of feature fusion maps of different feature map sizes, that is, the effective information of color feature maps of different feature map sizes and the effective information of depth feature maps of different feature map sizes are fully retained, ensuring the stability and accuracy of subsequent target detection based on the fusion feature map.

[0073] Again, the computing device determines at least one target object belonging to the corresponding target detection category among the N candidate objects based on the fused target feature to obtain the target detection result of the image to be detected. It is easy to understand that the fused target feature is the updated initial target feature, and the initial target feature contains the initial association relationship between the N candidate objects and each target detection category. In other words, the fused target feature contains the predicted association relationship between the N candidate objects and each target detection category, and each predicted association relationship is obtained based on each fused feature map. In this way, based on each predicted association relationship in the fused target feature, at least one target object belonging to the corresponding target detection category can be determined among the N candidate objects to obtain the target detection result of the image to be detected, thereby achieving the accuracy of the target detection result and the stability of the detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0075] Figure 1 A schematic diagram of an optional application scenario in an embodiment of the present application;

[0076] Figure 2 A schematic diagram of an optional application scenario of target detection of a physical object in an embodiment of the present application;

[0077] Figure 3 A schematic diagram of a target detection method provided in an embodiment of the present application;

[0078] Figure 4A A schematic diagram of a possible process of obtaining an initial color feature map in an embodiment of the present application;

[0079] Figure 4B A schematic diagram of a possible process of obtaining an attribute feature map in an embodiment of the present application;

[0080] Figure 5 A schematic diagram of generating a first fused feature map and a second fused feature map corresponding to each of a plurality of feature map sizes in the implementation of the present application;

[0081] Fig. 6A This is a schematic diagram of the process of obtaining the first fusion feature map in an embodiment of the present application;

[0082] Figure 6B This is a schematic diagram of the process of obtaining the second fusion feature map in an embodiment of the present application;

[0083] Figure 7 This is a schematic diagram of a process of performing feature interaction processing on initial target features in an embodiment of the present application;

[0084] Figure 8 A schematic diagram of defect detection provided in an embodiment of the present application;

[0085] Fig. 9 A schematic diagram of the structure of a target detection device provided in an embodiment of the present application;

[0086] Fig.10 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0087] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0088] In the embodiments of the present application, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0089] The embodiments of the present application relate to artificial intelligence technology, mainly to computer vision technology and natural language processing technology in artificial intelligence technology.

[0090] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0091] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, pre-trained model technology, operation / interaction systems, mechatronics, etc. Among them, pre-trained models are also called large models and basic models. After fine-tuning, they can be widely used in downstream tasks in various major directions of artificial intelligence. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0092] Computer Vision (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to using cameras and computers to replace human eyes to identify and measure targets, and further process images so that the computer processes them into images that are more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multidimensional data. Large model technology has brought important changes to the development of computer vision technology. Pre-trained models in the visual field such as SwinT (Swin-Transformer, also known as a deep learning model based on Transformer), ViT (Vision Transformer, visual self-attention model), V-MOE (Vision MoE, visual model), MAE (Masked Autoencoders, masked autoencoders) can be quickly and widely applied to downstream specific tasks after fine tuning. Computer vision technology generally includes image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0093] Natural language processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can enable effective communication between people and computers using natural language. Natural language processing involves natural language, which is the language people use in daily life, and is closely related to linguistics research; it also involves computer science and mathematics. The pre-training model, an important technology for model training in the field of artificial intelligence, is developed from the large language model (LLM) in the field of NLP. After fine-tuning, the large language model can be widely used in downstream tasks. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question answering, knowledge graph and other technologies.

[0094] Machine Learning (ML) is a multi-disciplinary interdisciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning by teaching. The pre-trained model is the latest development of deep learning, which integrates the above technologies.

[0095] Autonomous driving technology refers to the ability of a vehicle to drive itself without the need for a driver. It usually includes technologies such as high-precision maps, environmental perception, computer vision, behavioral decision-making, path planning, and motion control. Autonomous driving includes multiple development paths such as single-vehicle intelligence, vehicle-road collaboration, and networked cloud control. Autonomous driving technology has a wide range of application prospects. In addition to the current fields of logistics, public transportation, taxis, and smart transportation, it will be further developed in the future.

[0096] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common artificial intelligence generated content (AIGC), conversational interaction, smart medical care, smart customer service, game AI, smart home, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, digital twins, virtual humans, robots, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0097] In an embodiment of the present application, artificial intelligence technology is applied to the field of image processing technology, specifically to achieve target detection, so as to improve the accuracy and stability of target detection results.

[0098] To facilitate understanding of the technical solutions provided in the embodiments of the present application, some key terms used in the embodiments of the present application are first explained.

[0099] Target detection: used to identify the target object in the image to be detected to solve the problem of the existence of the target object; and used to determine the specific location of the identified target object in the image to be detected to solve the problem of where the target object is.

[0100] The following is a brief introduction to the design concept of the embodiments of the present application.

[0101] In the industrial field, the mainstream methods for target detection based on the image to be detected can be summarized as follows:

[0102] Method 1: Target detection is achieved based on the color features of the image to be detected (ie, the detection method in the background art).

[0103] In this method, the color features of the image to be detected are used as target features, so that target detection is achieved based on the target features. However, the inventors have discovered through creative work that color features are easily affected by acquisition environment factors such as acquisition equipment, acquisition light, and acquisition distance. In the actual target detection process, it is difficult to obtain stable target features due to the influence of acquisition environment factors, thereby seriously reducing the accuracy of the target detection results.

[0104] Method 2: For the image to be detected, target detection is achieved based on the splicing features of color features and depth features.

[0105] In this method, for the image to be detected, the spliced ​​features of its color features and its depth features are used as target features, so as to achieve target detection based on the target features; however, the inventors have discovered through creative work that the color features and depth features in the spliced ​​features usually represent a single form of the object, thereby ignoring the correlation between different forms. The target features obtained in this way are difficult to represent the overall form of the object, thereby reducing the accuracy of the target detection results.

[0106] Method three: For the image to be detected, target detection is achieved based on the fusion features of color features and depth features.

[0107] In this method, for the image to be detected, the feature dimensions of its color features and its depth features are transformed to the specified feature dimensions, and then the color features and depth features of the specified feature dimensions are feature fused, and the obtained fused features are used as target features, thereby achieving target detection based on the target features; however, the inventors have discovered through creative work that when performing target detection on small objects in the image to be detected, it is easy to lose the effective information of the small object in the process of feature dimension transformation, resulting in a lack of effective information of the small object in the fused features obtained based on this, thereby seriously reducing the stability and accuracy of the target detection results.

[0108] In view of this, an embodiment of the present application provides a target detection method to improve the accuracy and stability of target detection results.

[0109] In this method, a method for obtaining a fused feature map of multiple feature map sizes is provided. Based on a plurality of preset feature map sizes, corresponding color feature maps and depth feature maps are extracted for the image to be detected, and based on the plurality of feature map sizes, feature fusion processing is performed on the corresponding color feature maps and depth feature maps, respectively, to obtain at least one fused feature map corresponding to each of the plurality of feature map sizes. In this way, for each fused feature map, a depth feature map is fused, and the depth feature map can better characterize the three-dimensional form of the object. In this way, in some poor or unstable acquisition environments, the problem of unstable object form characterization based on the color feature map alone, resulting in poor subsequent target detection results, can be solved, so as to improve the accuracy of the target detection results; for each feature map size, its corresponding fused feature map is generated based on the fusion of the corresponding color feature map and the depth feature map, which can better explore the correlation between the two, so as to retain more effective object detail information; for different feature map sizes, the morphological information of the object can be fully reflected, so as to improve the detection performance of subsequent target detection.

[0110] In this method, a combination method of fused feature maps of multiple feature map sizes is provided, by obtaining the initial target features of the image to be detected, and using the obtained fused feature maps to perform feature interaction processing on the initial target features to obtain fused target features. The initial target feature characterization is: the initial association relationship between N candidate objects and each target detection category preset for the image to be detected; in this way, an updateable initial target feature is introduced to transform the combination problem between each fused feature map into an update problem for the initial target feature, so that based on the initial target feature, feature interaction processing is performed on each obtained fused feature map to achieve the update of the initial target feature. In this way, the obtained fused target feature can retain the effective information of feature fusion maps of different feature map sizes, that is, the effective information of color feature maps of different feature map sizes and the effective information of depth feature maps of different feature map sizes are fully retained, so as to ensure the stability and accuracy of subsequent target detection based on the fused feature map.

[0111] In this method, a target detection method with multiple feature sizes is provided, and based on the fused target features, at least one target object belonging to the corresponding target detection category is determined among N candidate objects to obtain the target detection result of the image to be detected. It is easy to understand that the fused target features are updated initial target features, and the initial target features contain the initial association relationship between N candidate objects and each target detection category. In other words, the fused target features contain the predicted association relationship between N candidate objects and each target detection category, and each predicted association relationship is obtained based on each fused feature map. In this way, based on each predicted association relationship in the fused target features, at least one target object belonging to the corresponding target detection category can be determined among the N candidate objects to obtain the target detection result of the image to be detected, so as to achieve the accuracy of the target detection result and the stability of the detection.

[0112] The following briefly introduces the application scenarios to which the technical solutions of the embodiments of the present application can be applied. It should be noted that the application scenarios introduced below are only used to illustrate the embodiments of the present application and are not limited. In the specific implementation process, the technical solutions provided by the embodiments of the present application can be flexibly applied according to actual needs.

[0113] The solution provided in the embodiments of the present application can be applied to most target detection scenarios of physical objects; for example, defect detection for lithium batteries in industrial quality inspection scenarios; for example, obstacle detection on roads in intelligent driving scenarios; and for example, category detection on roads in the field of robot navigation.

[0114] refer to Figure 1 As shown, it is a schematic diagram of an application scenario provided by an embodiment of the present application. In this scenario, a terminal device 110 and a server 120 may be included.

[0115] In the embodiment of the present application, the terminal device 110 includes but is not limited to acquisition equipment, cameras, color acquisition equipment, depth acquisition equipment, mobile phones, computers, tablet computers, laptops, desktop computers, vehicle-mounted terminal computers, e-book readers, intelligent voice interaction equipment, smart home appliances, vehicle-mounted terminals, aircraft and other equipment; the terminal device may be installed with a client for target detection (for example: target detection platform), which may be software (such as browsers, image processing software, etc.), or a web page, a small program, etc. The server 120 is a background server corresponding to the software or web page, small program, etc., or a server specifically used for image processing, which is not specifically limited in the present application. The server 120 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms.

[0116] It should be noted that the target detection method, the method for extracting the color feature map and the depth feature map, the method for extracting the fused feature map, the method for obtaining the initial target feature, the method for obtaining the fused target feature, and the method for obtaining the target detection result in each embodiment of the present application can be executed by an electronic device, and the electronic device can be a terminal device 110 or a server 120, that is, the method can be executed by the terminal device 110 or the server 120 alone, or can be executed by the terminal device 110 and the server 120 together. For example, when the terminal device 110 and the server 120 are executed together, the image to be detected is obtained by the terminal device 110, and the image to be detected is sent to the server 120, or the uniform resource locator (URL) of the image to be detected is sent to the server 120. Locator, URL) is sent to the server 120, which is downloaded by the server 120, etc.; then, through the target detection platform deployed on the server 120, based on the preset multiple feature map sizes, the corresponding color feature map and depth feature map are extracted for the image to be detected, and based on the multiple feature map sizes, the corresponding color feature map and depth feature map are respectively subjected to feature fusion processing to obtain at least one fusion feature map corresponding to each of the multiple feature map sizes; and, the initial target feature of the image to be detected is obtained, and the initial target feature represents: the initial association relationship between the N candidate objects and each target detection category preset for the image to be detected; then, the obtained fusion feature maps are used to perform feature interaction processing on the initial target feature to obtain the fusion target feature, and based on the fusion target feature, at least one target object belonging to the corresponding target detection category is determined among the N candidate objects to obtain the target detection result of the image to be detected; then, the target detection result is sent to the terminal device 110 through the server 120. Finally, the terminal device 110 performs a visual display.

[0117] Exemplarily, the solution provided in the embodiments of the present application can be applied to target detection application scenarios of various physical objects, such as: application scenarios of lithium battery quality inspection, application scenarios of obstacle recognition, application scenarios of road recognition, etc.

[0118] refer to Figure 2As shown, it is a schematic diagram of an application scenario of target detection of a physical object provided in an embodiment of the present application. In this scenario, the server 120 is used as the background server of the target detection platform, and the terminal device 110 is used as the subject for the target detection platform to interact with the user object. The user can use the target detection platform on the terminal device 110 by sending the image to be detected to the target detection platform; accordingly, after acquiring the image to be detected, the terminal device 110 forwards the image to be detected to the server 120; of course, the terminal device 110 can also collect and process the physical object to obtain the corresponding image to be detected and send it to the server 120; accordingly, the server 120 receives the image to be detected, extracts the corresponding color feature map and depth feature map, performs feature fusion processing on the extracted feature map based on multiple feature map sizes, and obtains at least one fused feature map corresponding to each of the multiple feature map sizes; and obtains the initial target feature of the image to be detected, the initial target feature represents: the initial association relationship between N candidate objects and each target detection category preset for the image to be detected; then, using the obtained fused feature maps, the initial target feature is subjected to feature interaction processing to obtain the fused target feature, and based on the fused target feature, at least one target object belonging to the corresponding target detection category is determined among the N candidate objects to obtain the target detection result of the image to be detected.

[0119] It should be noted that the above-mentioned client may be a mini-program client, a browser client, etc., and the mini-program client and the browser client may be deployed in different backend servers respectively.

[0120] Of course, the method provided in the embodiment of the present application is not limited to Figure 1 or Figure 2 The application scenarios shown can also be used in other possible application scenarios, and the embodiments of the present application are not limited thereto. Figure 1 or Figure 2 The functions that can be implemented by each device in the application scenario shown will be described in the subsequent method embodiments, and will not be described in detail here.

[0121] In addition, the above-mentioned process of the terminal device 110 and the server 120 jointly executing the label generation method of the multimedia information is only a feasible implementation method. In fact, any process of the terminal device 110 and the server 120 executing the method is applicable to the embodiments of the present application, and this article will not go into details one by one.

[0122] In an optional implementation, the terminal device 110 and the server 120 may be directly or indirectly connected in communication through one or more networks 130. The network 130 may be a wired network or a wireless network, for example, a mobile cellular network, or a wireless fidelity (Wireless-Fidelity, WIFI) network, or other possible networks, which are not limited in the embodiment of the present invention.

[0123] It should be noted that in the embodiment of the present application, the number of terminal devices 110 can be one or more, and similarly, the number of servers 120 can be one or more, that is, there is no limitation on the number of terminal devices 110 or servers 120.

[0124] In a possible application scenario, in order to reduce the communication delay of retrieval, servers 120 can be deployed in various regions, or for load balancing, different servers 120 can serve terminal devices 110 in different regions respectively. For example, the terminal device 110 is located at location a and establishes a communication connection with the server 120 serving location a; the terminal device 110 is located at location b and establishes a communication connection with the server 120 serving location b. Multiple servers 120 form a data sharing system to achieve data sharing through blockchain.

[0125] Each server 120 in the data sharing system has a node identifier corresponding to the server 120. Each server 120 in the data sharing system can store the node identifiers of other servers 120 in the data sharing system, so that the generated blocks can be broadcast to other servers 120 in the data sharing system according to the node identifiers of other servers 120. A node identifier list can be maintained in each server 120, and the server 120 name and the node identifier are stored in the node identifier list accordingly. The node identifier can be an Internet Protocol (IP) address for interconnection between networks and any other information that can be used to identify the node.

[0126] Exemplarily, in an embodiment of the present application, when the number of servers is multiple, the multiple servers can form a blockchain, and the servers are nodes on the blockchain; as in the target detection method disclosed in the embodiment of the present application, the various feature map sizes involved, the color feature maps and depth feature maps corresponding to the image to be detected, the fused feature maps, the initial target features, the fused target features, the target detection results, etc. can all be saved on the blockchain.

[0127] In addition, the embodiments of the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, autonomous driving and the like.

[0128] Below, in combination with the application scenarios described above, the method provided by the exemplary implementation of the present application is described with reference to the accompanying drawings. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principle of the present application, and the implementation of the present application is not limited in this respect. It should be noted that the following method can be executed by the above terminal device or server, or by the terminal device and the server together. Here, the server execution is specifically used as an example for illustration.

[0129] See also Figure 3 As shown, it is an implementation flow chart of a method for generating a label of multimedia information provided in an embodiment of the present application. Taking a computing device characterized as a terminal device or a server as an execution subject as an example, the specific implementation process of the method is as follows:

[0130] Step 301: Based on a plurality of preset feature map sizes, extract corresponding color feature maps and depth feature maps for the image to be detected.

[0131] Among them, the preset multiple feature map sizes are different from each other. The smaller the feature map size, the less information the corresponding color feature map and depth feature map retains of the image to be detected. The larger the feature map size, the more information the corresponding color feature map and depth feature map retain of the image to be detected.

[0132] The inventors have found in practice that, regardless of the images to be detected collected for small objects, images to be detected collected for large objects, or images to be detected collected for both small and large objects, the color feature map and depth feature map extracted based on a single feature map size are difficult to capture the global information of large objects while capturing the detail information of small objects. As a result, the subsequent color feature map and depth feature map based on a single feature map size cannot accurately analyze the effective information in the image to be detected, and the target detection results obtained are unstable and have low accuracy.

[0133] Preferably, in an embodiment of the present application, in order to improve the versatility of target detection, the size difference between the preset multiple feature map sizes is greater than the preset difference value, so as to better capture the global features of large objects while capturing the detailed information of small objects, thereby improving the performance of target detection.

[0134] Furthermore, based on a plurality of preset feature map sizes, corresponding color feature maps and depth feature maps are extracted from the image to be detected respectively.

[0135] In an optional implementation, the above-mentioned image to be detected includes: a color map to be detected and a depth map to be detected collected for the same target scene, and then the following operations are performed on the color map to be detected and the depth map to be detected based on a preset plurality of feature map sizes: based on a preset feature map size, the color map to be detected and the initial depth map to be detected are respectively transformed in size to obtain corresponding initial color feature maps and initial depth feature maps, and then, the initial color feature map and the initial depth feature map are respectively interactively processed with pixels within the image to obtain corresponding color feature maps and depth feature maps.

[0136] Among them, the target scene contains at least one object to be detected, the color map to be detected can be an RGB (Red Green Blue) map acquired by a color acquisition device, and the depth map to be detected can be a depth map acquired by a depth acquisition device. Alternatively, the depth map to be detected can be obtained by calculating the distance between two acquired depth maps acquired by the same acquisition device at different acquisition times for the same target scene, or the depth map to be detected can also be obtained by calculating the distance between two acquired depth maps acquired by different acquisition devices at different acquisition positions for the same target scene.

[0137] In an optional specific implementation method, in order to realize the image size transformation processing of the color map to be detected and the depth map to be detected, taking a preset feature map size as an example, based on a feature map size, the two attribute maps to be detected, namely the color map to be detected and the depth map to be detected, are respectively performed the following operations: based on the size difference value between the image size of an attribute map to be detected and a feature map size, an attribute map to be detected is divided into multiple blocks to be detected, and then at least one initial pixel point contained in each of the multiple blocks to be detected is respectively mapped to the corresponding feature space to obtain a corresponding initial attribute feature map, which is an initial color feature map or an initial depth feature map.

[0138] For example, taking the initial color feature map as an example, see Figure 4AAs shown, it is a schematic diagram of a possible process of obtaining an initial color feature map in an embodiment of the present application. If the height of the color map to be detected is H, the width is W, and the number of channels is 3 (RGB three channels), where H and W constitute the image size of the color map to be detected, and a preset feature map size is H / 4 and W / 4, then the size difference between the image size of the color map to be detected and a feature map size is 4, and the color map to be detected is divided into multiple blocks to be detected with 4x4 adjacent pixels as a small block, and then for each block to be detected, each pixel in it is flattened in the channel direction. (flatten), where each image block to be detected has 16 pixels in a single channel after flattening, and the corresponding RGB three channels have 48 pixels. At this time, the height of the color map to be detected is H / 4, the width is W / 4, and the number of channels is 48. In order to ensure that the number of channels of the initial color feature map obtained by transforming different feature map sizes is consistent, at least one initial pixel in each image block to be detected is mapped to the corresponding feature space, and linear transformation processing is implemented for the channel data of each initial pixel, so as to obtain an initial color feature map with a height of H / 4, a width of W / 4, and a number of channels of C (specified number of channels).

[0139] It should be noted that the initial depth feature map can also be obtained by converting the depth map to be detected in the above manner. Optionally, the number of dimensions of the depth map to be detected is changed from 1 to 3, that is, the depth dimension of the depth map to be detected is copied three times, and its processing method is exactly the same as the processing method of the above-mentioned color map to be detected. Of course, the aforementioned dimensional copying operation is not necessary. Based on a similar idea, the corresponding initial color feature map can also be obtained by converting the depth map to be detected.

[0140] In addition, the above process can be based on deep learning ideas, such as applying Swin Transformer to implement the conversion process of the above initial color feature map or initial depth map, and no specific restrictions are made on the specific implementation method here.

[0141] In an optional specific implementation method, in order to obtain a corresponding color feature map and a depth feature map based on each initial color feature map and an initial depth feature map, here, taking a preset feature map size as an example, the following operations are performed on the two initial attribute feature maps, the corresponding initial color feature map and the corresponding initial depth feature map: based on a preset window size, an initial attribute feature map is divided into multiple window tiles, wherein each window tile contains at least one candidate pixel point in an initial attribute feature map, and then, self-attention feature interaction processing is performed on at least one candidate pixel point contained in each of the multiple window tiles, and cross-attention feature interaction processing is performed on at least multiple window tiles to obtain a corresponding attribute feature map, which is a color feature map or a depth feature map.

[0142] For example, taking an initial feature map as an example, see Figure 4B As shown, a schematic diagram of a possible process of obtaining an attribute feature map in an embodiment of the present application is shown. If the preset window size is 2x2 (window unit: pixel), starting from the upper left corner of the initial attribute feature map, each candidate pixel in the initial attribute map is traversed from left to right and from top to bottom, and an initial feature map is divided into multiple window tiles. Self-attention feature interaction processing is performed on the four candidate pixels contained in each window tile, and cross-attention feature interaction processing is performed on two adjacent window tiles to obtain a corresponding attribute feature map.

[0143] In addition, the above process can be based on deep learning ideas, such as applying win Transformer Block to implement the process of obtaining the above attribute feature map, and no specific restrictions are made on the implementation method here.

[0144] Optionally, in the process of obtaining the corresponding attribute feature map through the above-mentioned feature interaction processing, the adjacent window blocks that have undergone feature interaction processing are merged, and the merged results are converted into feature (channel) dimensions, that is, the corresponding channel number is converted into the specified channel number, so as to realize the mining of useful correlation information in the image to be detected.

[0145] It is easy to understand that in the processing of the above window blocks, the idea of ​​transformer is introduced to realize the interactive processing of self-attention (W-MSA) features within the window blocks, and the idea of ​​convolution shift is introduced to realize the interactive processing of cross-attention features between window blocks, which strengthens the mining of the correlation relationship between each pixel point and the adjacent pixel points in the attribute feature map, which helps to improve the accuracy and stability of subsequent target detection results.

[0146] Step 302: Based on the various feature map sizes, feature fusion processing is performed on the corresponding color feature map and depth feature map respectively to obtain at least one fused feature map corresponding to each of the various feature map sizes.

[0147] Generally speaking, for each feature map size, feature fusion processing is performed on a corresponding color feature map and a depth feature map to obtain a fused feature map corresponding to the feature map size; since the color feature map and the depth feature map are generated by mining the correlation between each pixel point in the corresponding image to be detected, the corresponding fused feature map can not only better retain the morphological information of the object in the image to be detected, but also effectively retain the correlation between each pixel point therein, such as the correlation between objects and objects, objects and non-objects, objects and backgrounds, etc.

[0148] In an embodiment of the present application, a feature map fusion method based on pixel level is also provided. In this method, for each feature map size, the following operations are performed respectively: a color feature map and a depth feature map matching a feature map size are obtained, and in the aforementioned color feature map, feature fusion processing is performed on each color pixel based on the K depth pixels in the aforementioned depth feature map to obtain a corresponding first fused feature map; and in the aforementioned depth feature map, feature fusion processing is performed on each depth pixel based on the K color pixels in the aforementioned color feature map to obtain a corresponding second fused feature map, and then the first fused feature map and the second fused feature map are both used as fused feature maps corresponding to the feature map size.

[0149] Among them, the first fused feature map is a fused feature map generated by taking the color feature map as the main component and combining the depth feature map, and the second fused feature map is a fused feature map generated by taking the depth feature map as the main component and combining the color feature map.

[0150] For example, see Figure 5 As shown, it is a schematic diagram of generating the first fused feature map and the second fused feature map corresponding to each of the various feature map sizes in the implementation of the present application. First, multiple color feature maps are respectively taken as the leading ones, which are fused with the corresponding depth feature maps to generate multiple first fused feature maps; second, multiple depth feature maps are respectively taken as the leading ones, which are fused with the corresponding color feature maps to generate multiple second fused feature maps.

[0151] For ease of understanding, the following is divided into two parts, respectively explaining how to generate the first fused feature map and the second fused feature map.

[0152] Part 1: How to generate the first fusion feature map.

[0153] Specifically, in the color feature map, feature fusion processing is performed on each color pixel based on the K depth pixels in the depth feature map to obtain a corresponding first fusion feature map.

[0154] See also Fig. 6A As shown, it is a schematic diagram of the process of obtaining the first fusion feature map in the embodiment of the present application. Taking a single color pixel as an example, for each color pixel, the following operations are performed respectively: based on the association relationship between the color pixel and each depth pixel in the depth feature map, K depth pixels that meet the first association condition and their corresponding first fusion weights are obtained, and based on the obtained K first fusion weights, the K depth pixels and the color pixels are fused to obtain the first fusion point corresponding to the color pixel. Then, based on the first fusion point corresponding to each color pixel, the corresponding first fusion feature map is obtained.

[0155] Among them, each first fusion weight represents: the association relationship between the corresponding depth pixel point and the color pixel point, and the first association condition is used to evaluate the similarity of the association relationship. Of course, the association relationship here includes but is not limited to: the distance similarity relationship and the semantic association relationship between the aforementioned two.

[0156] For example, for a preset image size, the corresponding color feature image F rgb The size and dimension of is h×w×c, and the corresponding depth feature map F depth The size and color of h×w×c, then for a color pixel q corresponding to the first fusion point The relationship between the K depth pixels and their corresponding first fusion weights can be characterized as follows:

[0157]

[0158] in, is the first fusion point corresponding to the color pixel q, is the color pixel q, W qk is the kth first fusion weight corresponding to the color pixel q, is the kth depth pixel corresponding to the color pixel q; For example, Δ qk Represents the position offset of the k-th depth pixel compared to the color pixel.

[0159] Further, in a first optional specific implementation method, a method for obtaining K depth pixels and their corresponding first fusion weights is also proposed. In this method, first, based on the first relationship prediction parameter, first association information between each depth pixel and the color pixel is obtained, and then, among the obtained first association information, K first association information satisfying the first association condition is selected, and the corresponding K depth pixels are obtained, and then, based on the first weight prediction parameter, the influence of each depth pixel on the color pixel is calculated respectively, and the first fusion weight corresponding to each of the K depth pixels is obtained;

[0160] Among them, the first relationship prediction parameter represents: the association relationship between the color pixel and each depth pixel in the depth feature map, and the first weight prediction parameter represents: the association relationship between the color pixel and K depth pixels.

[0161] Exemplarily, the process of obtaining the above-mentioned K depth pixels and their corresponding first fusion weights can be based on deep learning and obtained by using full connection layer prediction. The above-mentioned first relationship prediction parameter and first weight prediction parameter are learning parameters in the full connection layer training process. Then, the association relationship between the K depth pixels and their corresponding first fusion weights and the color pixel q can be characterized as follows:

[0162]

[0163] Among them, W qk is the kth first fusion weight corresponding to the color pixel q, Δ qk is the position offset of the kth depth pixel compared to the color pixel, is the first fusion point corresponding to the color pixel q, and MLP() is the fully connected layer processing method.

[0164] It is worth noting that the sum of the above K first fusion weights satisfies a preset threshold. For example, when the preset threshold is 1, the correlation relationship between the K first fusion weights can be characterized as follows:

[0165]

[0166] Among them, W qk is the kth first fusion weight corresponding to the color pixel q, and K is an integer greater than or equal to 1.

[0167] Part 2: How to generate the second fusion feature map.

[0168] Specifically, in the depth feature map, feature fusion processing is performed on each depth pixel based on the K color pixels in the color feature map to obtain a corresponding second fused feature map.

[0169] See also Figure 6B As shown, it is a schematic diagram of the process of obtaining the second fusion feature map in the embodiment of the present application. Taking a single depth pixel as an example, for each depth pixel, the following operations are performed respectively: based on the association relationship between the depth pixel and each color pixel in the color feature map, K color pixels that meet the second association condition and their corresponding second fusion weights are obtained, and then, based on the obtained K second fusion weights, the K color pixels and the depth pixels are fused to obtain the second fusion point corresponding to the depth pixel. Then, based on the second fusion point corresponding to each depth pixel, the corresponding second fusion feature map is obtained.

[0170] Among them, each second fusion weight represents: the association relationship between the corresponding color pixel and the depth pixel, and the second association condition is used to evaluate the similarity of the association relationship. Of course, the association relationship here includes but is not limited to: the distance similarity relationship and the semantic association relationship between the aforementioned two.

[0171] For example, for a preset image size, the corresponding color feature image F rgb The size and dimension of is h×w×c, and the corresponding depth feature map F depthThe size and color of is h×w×c, then for a depth pixel q corresponding to the first fusion point The correlation relationship between the K color pixels and their corresponding second fusion weights can be characterized as:

[0172]

[0173] in, is the second fusion point corresponding to the depth pixel q, is the depth pixel q, W qk is the kth second fusion weight corresponding to the depth pixel q, is the kth color pixel corresponding to the depth pixel q; For example, Δ qk Represents the position offset of the k-th color pixel compared to the depth pixel.

[0174] Furthermore, in an optional specific implementation method, a method for obtaining K color pixels and their corresponding second fusion weights is proposed. In this method, first, based on the second relationship prediction parameter, the second association information between each color pixel and the depth pixel is obtained, and then, among the obtained second association information, K second association information that meet the second association condition are selected, and the corresponding K color pixels are obtained. Then, based on the second weight prediction parameter, the degree of influence of each color pixel on the depth pixel is calculated respectively, and the second fusion weights corresponding to each of the K color pixels are obtained.

[0175] Among them, the second relationship prediction parameter represents: the association relationship between the depth pixel point and each color pixel point in the color feature map, and the second weight prediction parameter represents: the association relationship between the depth pixel point and K color pixels.

[0176] Exemplarily, the process of obtaining the above-mentioned K color pixels and their corresponding second fusion weights can be based on deep learning and obtained by prediction using a fully connected layer. The above-mentioned second relationship prediction parameter and second weight prediction parameter are learning parameters in the training process of the fully connected layer. Then, the association relationship between the K color pixels and their corresponding second fusion weights and the depth pixel q can be characterized as follows:

[0177]

[0178] Among them, W qk is the kth second fusion weight corresponding to the depth pixel q, Δ qk is the position offset of the kth color pixel compared to the depth pixel, is the second fusion point corresponding to the depth pixel q, and MLP() is the fully connected layer processing method.

[0179] It is worth noting that the sum of the above K second fusion weights satisfies a preset threshold. For example, when the preset threshold is 1, the correlation relationship between the K second fusion weights can be characterized as follows:

[0180]

[0181] Among them, W qk is the kth second fusion weight corresponding to the depth pixel point q, and K is an integer greater than or equal to 1.

[0182] It should also be noted that the above implementation process of this step can adopt deep learning, based on the same implementation idea, and implemented through the DTCA (Deformable Transformer Cross Attention) module, which will not be elaborated here.

[0183] To summarize, this step realizes the fusion of feature maps based on the pixel level, and proposes a feature map fusion method dominated by color feature maps or depth feature maps. Specifically, for a single color pixel or depth pixel, by sampling the corresponding K depth pixels or color pixels, the sampling of sparse space and pixel-level interaction modeling are realized. Moreover, for each color pixel or depth pixel, the corresponding K depth pixels or color pixels are not the same, but the positions of the selected K depth pixels and their corresponding fusion weights (first fusion weight or second fusion weight) are determined based on the correlation relationship between the pixels, so that the fusion process of the feature map is more flexible, can better adapt to different feature map sizes, and effectively fuse the effective information of color and the effective information of depth, so as to improve the stability and accuracy of subsequent target detection results.

[0184] Step 303: Acquire initial target features of the image to be detected, where the initial target features represent: initial association relationships between N candidate objects and target detection categories preset for the image to be detected.

[0185] Among them, the initial target features can be a set of randomly initialized features that can be learned / updated, and the initial association relationship can be preset according to actual conditions, for example, each initial association relationship is preset to be equivalent, and so on.

[0186] For example, the initial target feature may represent a learnable target template object quries after random initialization. in, is a real number, N is the preset N candidate objects, and C is the feature dimension (number of channels) corresponding to each candidate object.

[0187] Step 304: using the obtained fusion feature maps, perform feature interaction processing on the initial target features to obtain fusion target features.

[0188] Among them, the fused target feature representation: the target association relationship between N candidate objects and each target detection category.

[0189] See also Figure 7 As shown, it is a schematic diagram of the process of performing feature interaction processing on the initial target feature in the embodiment of the present application. It can be seen that, taking 4 preset feature map sizes as an example, the corresponding 4 first fusion feature maps and 4 second fusion feature maps are used to perform feature interaction processing on the initial target feature to obtain the fusion target feature.

[0190] Specifically, in the implementation of the present application, each fusion feature map obtained is used to perform iterative update processing on the initial target features in turn to obtain corresponding updated target features.

[0191] For the above-mentioned update processing, in one update process, feature association information between a fused feature map and the initial target feature is obtained, and based on the positive impact of the feature association information on the initial target feature, the initial target feature is updated.

[0192] In an optional implementation method, in the process of updating the above-mentioned initial target features, taking the updating of a single initial feature point in the initial target features as an example, for each initial feature point, the following operations are performed respectively: based on an initial feature point, the association relationship between each fused pixel point in a fused feature map is obtained, J fused pixel points that meet the fusion association conditions and their corresponding fusion association weights are obtained, and then based on the obtained J fusion association weights, the J fused pixel points and the initial feature point are fused to obtain an updated feature point corresponding to the initial feature point.

[0193] Among them, each fusion association weight represents: the association relationship between the corresponding fused pixel point and the initial feature point, and the association relationship represents the distance relationship, semantic relationship, etc. between the corresponding fused pixel point and the initial feature point.

[0194] As an example, multiple first fused feature maps corresponding to multiple preset feature map sizes are used to update the initial target feature. Taking the use of a first fused feature map to update an initial feature point i in the initial target feature as an example, the correlation relationship between the first fused feature map and the initial feature point i can be characterized as follows:

[0195]

[0196] Among them, i∈[1,N], N is the total number of preset candidate targets, Q i is an initial feature point i in the initial target feature, W ijis the jth fusion association weight corresponding to the initial feature point i, and J is an integer greater than or equal to 1, is the initial feature point Q i The corresponding j-th fused pixel (that is, the feature of the j-th fused pixel); For example, ref represents the initial feature point Q i The predicted reference point (also known as the anchor point), Δij represents the position offset of the jth fused pixel point compared to the reference point ref, W ij Characterize an initial feature point Q i Correlation with reference point ref.

[0197] It is worth noting that the sum of the above J fusion association weights satisfies a preset threshold. For example, when the preset threshold is 1, the association relationship between the J fusion association weights can be characterized as follows:

[0198]

[0199] Among them, W ij is the fusion association weight corresponding to the j-th fused pixel point, and K is an integer greater than or equal to 1.

[0200] As another example, multiple second fused feature maps corresponding to multiple preset feature map sizes are used to update the initial target feature. Taking the use of a second fused feature map to update an initial feature point i in the initial target feature as an example, the correlation relationship between the second fused feature map and the initial feature point i can be characterized as follows:

[0201]

[0202] Among them, i∈[1,N], N is the total number of preset candidate targets, Q i is an initial feature point i in the initial target feature, W ij is the jth fusion association weight corresponding to the initial feature point i, and J is an integer greater than or equal to 1, is the initial feature point Q i The corresponding j-th fused pixel (that is, the feature of the j-th fused pixel); For example, ref represents the initial feature point Q i The predicted reference point (also known as the anchor point), Δij represents the position offset of the jth fused pixel point compared to the reference point ref, W ij Characterize an initial feature point Q i Correlation with reference point ref.

[0203] It is worth noting that the sum of the above J fusion association weights satisfies a preset threshold. For example, when the preset threshold is 1, the association relationship between the J fusion association weights can be characterized as follows:

[0204]

[0205] Among them, W ij is the fusion association weight corresponding to the j-th fused pixel point, and K is an integer greater than or equal to 1.

[0206] Subsequently, after the initial target feature is updated based on the first fused feature map and the second fused feature map corresponding to the multiple feature map sizes to obtain the updated target feature, the updated target feature is adjusted based on the object association information between the N candidate objects represented by the updated target feature to obtain the fused target feature.

[0207] In an optional implementation, in the process of adjusting the above-mentioned updated target features, for the N candidate objects represented by the updated target features, based on the object association relationship between each candidate object and the N candidate objects, the object weights corresponding to the N candidate objects are obtained, and then the updated target features are weighted summed using the obtained N object weights to obtain the fused target features.

[0208] Exemplarily, for the updated target feature Q, the matrix inversion Q of the updated target feature is obtained T , update the target feature Q and its matrix inversion Q T Perform dot multiplication (Q·Q T ), and then the result is (Q·Q T ) is normalized, and the normalized result can be used as the object weight corresponding to each of the N candidate objects. Optionally, the normalization is implemented based on an activation function (such as softmax), and the object weight corresponding to each of the N candidate objects can be represented as: softmax(Q·Q T ). Accordingly, the relationship between the fusion target feature and the N object weights and the updated target feature can be represented as:

[0209] Q update =softmax(Q·Q T )·Q

[0210] Among them, Q update To fuse the target features, softmax(Q·Q T ) is the object weight corresponding to each of the N candidate objects, and Q is the updated target feature (which includes at least N candidate objects).

[0211] To sum up, through this step, the initial target features are introduced to perform fusion processing with at least one fused feature map corresponding to each of the multiple feature map sizes, so that the obtained fused target features can characterize the fused feature representations of different feature map sizes (that is, different scales), and more comprehensively capture the detail information and semantic information (context information) of real objects at different sizes (scales), thereby improving the expressiveness and accuracy of target detection.

[0212] Step 305: Based on the fused target features, determine at least one target object belonging to the corresponding target detection category among the N candidate objects to obtain a target detection result of the image to be detected.

[0213] Among them, based on the two trained MLP fully connected layers, the fused target features are respectively input into the two MLP fully connected layers, and based on the two MLP fully connected layers, at least one target object belonging to the corresponding target detection category is respectively determined, and the position information (for example: detection box, etc.) of each at least one target object in the corresponding image to be detected is determined.

[0214] In an optional implementation, the trained MLP fully connected layer pre-stores detection parameters associated with each target detection category, and its internal execution process may include: obtaining detection parameters associated with each target detection category, wherein each detection parameter represents: the degree of association between each historical object and the corresponding target detection category, and then based on the obtained detection parameters, performing feature analysis processing on the fused target features to obtain the detection probability of each of the N candidate objects belonging to each target detection category, and then performing the following operations on the N candidate objects: when the maximum detection probability associated with a candidate object meets the preset detection probability, determining that the candidate object is a target object belonging to the target detection category corresponding to the maximum detection probability, and confirming the position information of the target object in the corresponding image to be detected (for example: detection box).

[0215] It should be noted that the above execution process uses the trained MLP fully connected layer as a possible scenario. The embodiment of the present application does not impose specific restrictions on the execution subject or execution scenario, and it can also be other trained detection models or detection architectures.

[0216] In summary, the embodiments of the present application provide a target detection method, which can be widely used in various target detection scenarios for physical objects to improve the accuracy of target detection and reduce the occurrence of false detection and missed detection.

[0217] like Figure 8As shown, it is a schematic diagram of defect detection provided by an embodiment of the present application. In short, based on the color acquisition device, a color map to be detected is collected for the target scene, and based on the depth acquisition device, a depth map to be detected is collected for the target scene. The color map to be detected and the depth map to be detected are both used as images to be detected, and are input into a dual-stream encoder (i.e., two SwinTransformer encoders). Based on a plurality of preset feature map sizes, corresponding color feature maps and depth feature maps are extracted for the images to be detected, respectively. Based on a plurality of feature map sizes, feature fusion processing is performed on the corresponding color feature maps and depth feature maps, respectively, to obtain at least one fused feature map corresponding to each of the plurality of feature map sizes, and to obtain the initial target features of the image to be detected. The obtained fused feature maps are used to perform feature interaction processing on the initial target features to obtain fused target features; the fused target features are then input into a decoder, and based on the fused target features, at least one target object belonging to the corresponding target detection category is determined among the N candidate objects represented by the fused target features to obtain the target detection result of the image to be detected, and the position of the target object (such as the target lithium battery) having a fault in the image to be detected is marked with a detection frame, and the fault category of the target object is determined.

[0218] See also Fig. 9 As shown, based on the same inventive concept, the embodiment of the present application further provides a target detection device 900, which includes:

[0219] An extraction unit 901 is used to extract corresponding color feature maps and depth feature maps for the image to be detected based on a plurality of preset feature map sizes;

[0220] A fusion unit 902 is used to perform feature fusion processing on the corresponding color feature map and the depth feature map based on multiple feature map sizes, and obtain at least one fused feature map corresponding to each of the multiple feature map sizes;

[0221] The interaction unit 903 is used to obtain the initial target feature of the image to be detected, and the initial target feature represents: the initial association relationship between N candidate objects and each target detection category preset for the image to be detected; using each fusion feature map obtained, the initial target feature is subjected to feature interaction processing to obtain the fusion target feature;

[0222] The obtaining unit 904 is configured to determine, based on the fused target features, at least one target object belonging to a corresponding target detection category among the N candidate objects, so as to obtain a target detection result of the image to be detected.

[0223] Optionally, the image to be detected includes: a color image to be detected and a depth image to be detected collected for the same target scene;

[0224] The extraction unit 901 is specifically used for:

[0225] Based on the preset multiple feature map sizes, perform the following operations on the color map to be detected and the depth map to be detected:

[0226] Based on a preset feature map size, the color map to be detected and the initial depth map to be detected are respectively transformed to obtain a corresponding initial color feature map and an initial depth feature map;

[0227] The initial color feature map and the initial depth feature map are respectively subjected to interactive processing of the pixels in the map to obtain the corresponding color feature map and depth feature map.

[0228] Optionally, the extraction unit 901 is used to transform the image size of the color image to be detected and the depth image to be detected based on a preset feature image size, respectively, to obtain the corresponding initial color feature image and initial depth feature image, specifically for:

[0229] Based on a preset feature map size, the following operations are performed on the two attribute maps to be detected, the color map to be detected and the depth map to be detected:

[0230] Based on a size difference between a size of an attribute map to be detected and a size of a feature map, a property map to be detected is divided into a plurality of blocks to be detected;

[0231] At least one initial pixel point contained in each of the multiple image blocks to be detected is mapped to a corresponding feature space to obtain a corresponding initial attribute feature map, where the initial attribute feature map is an initial color feature map or an initial depth feature map.

[0232] Optionally, the extraction unit 901 is used to perform interactive processing of pixels in the initial color feature map and the initial depth feature map, respectively, to obtain corresponding color feature maps and depth feature maps, specifically for:

[0233] For the two initial attribute feature maps, the initial color feature map and the initial depth feature map, perform the following operations respectively:

[0234] Based on a preset window size, an initial attribute feature map is divided into a plurality of window blocks; wherein each window block contains at least one candidate pixel point in the initial attribute feature map;

[0235] Self-attention feature interaction processing is performed on at least one candidate pixel point contained in each of the multiple window tiles, and cross-attention feature interaction processing is performed at least between the multiple window tiles to obtain a corresponding attribute feature map, which is a color feature map or a depth feature map.

[0236] Optionally, the fusion unit 902 is specifically configured to:

[0237] For each feature map size, perform the following operations:

[0238] Get the color feature map and depth feature map that match the feature map size;

[0239] In the color feature map, based on the K depth pixels in the depth feature map, feature fusion processing is performed on each color pixel to obtain a corresponding first fusion feature map;

[0240] In the depth feature map, based on the K color pixels in the color feature map, feature fusion processing is performed on each depth pixel to obtain a corresponding second fusion feature map;

[0241] The first fused feature map and the second fused feature map are both used as fused feature maps corresponding to the feature map size.

[0242] Optionally, the fusion unit 902 is used to perform feature fusion processing on each color pixel in the color feature map based on K depth pixels in the depth feature map to obtain a corresponding first fusion feature map, specifically for:

[0243] For each color pixel in the color feature map, perform the following operations:

[0244] Based on the association relationship between the color pixel and each depth pixel in the depth feature map, K depth pixels satisfying the first association condition and their corresponding first fusion weights are obtained; wherein each first fusion weight represents: the association relationship between the corresponding depth pixel and the color pixel;

[0245] Based on the obtained K first fusion weights, the K depth pixels and the color pixels are fused to obtain first fusion points corresponding to the color pixels;

[0246] Based on the first fusion point corresponding to each color pixel, a corresponding first fusion feature map is obtained.

[0247] Optionally, the fusion unit 902 is used to obtain K depth pixels satisfying the first association condition and their corresponding first fusion weights based on the association relationship between the color pixels and each depth pixel in the depth feature map, specifically for:

[0248] Based on the first relationship prediction parameter, first association information between each depth pixel and the color pixel is obtained; wherein the first relationship prediction parameter represents: the association relationship between the color pixel and each depth pixel in the depth feature map;

[0249] Select K first association information satisfying the first association condition from among the obtained first association information, and obtain corresponding K depth pixel points;

[0250] Based on the first weight prediction parameter, the influence of each depth pixel on the color pixel is calculated respectively to obtain the first fusion weight corresponding to each of the K depth pixels; wherein the first weight prediction parameter represents: the correlation relationship between the color pixel and the K depth pixels.

[0251] Optionally, the fusion unit 902 is used to perform feature fusion processing on each depth pixel in the depth feature map based on K color pixels in the color feature map to obtain a corresponding second fused feature map, specifically for:

[0252] For each depth pixel in the depth feature map, perform the following operations:

[0253] Based on the association relationship between the depth pixel and each color pixel in the color feature map, K color pixels satisfying the second association condition and their corresponding second fusion weights are obtained; wherein each second fusion weight represents: the association relationship between the corresponding color pixel and the depth pixel;

[0254] Based on the obtained K second fusion weights, the K color pixels and the depth pixels are fused to obtain second fusion points corresponding to the depth pixels;

[0255] Based on the second fusion point corresponding to each depth pixel, a corresponding second fusion feature map is obtained.

[0256] Optionally, the fusion unit 902 is used to obtain K color pixels satisfying the second association condition and their corresponding second fusion weights based on the association relationship between the depth pixel and each color pixel in the color feature map, specifically for:

[0257] Based on the second relationship prediction parameter, second association information between each color pixel and the depth pixel is obtained; wherein the second relationship prediction parameter represents: the association relationship between the depth pixel and each color pixel in the color feature map;

[0258] Select K pieces of second association information satisfying the second association condition from among the obtained pieces of second association information, and obtain corresponding K color pixel points;

[0259] Based on the second weight prediction parameter, the influence of each color pixel on the depth pixel is calculated respectively to obtain the second fusion weights corresponding to the K color pixels respectively; wherein the second weight prediction parameter represents: the correlation relationship between the depth pixel and the K color pixels.

[0260] Optionally, the interaction unit 903 is specifically configured to:

[0261] Using each acquired fusion feature map, the initial target feature is iteratively updated in turn to obtain the corresponding updated target feature; wherein, in one updating process, feature association information between a fusion feature map and the initial target feature is obtained, and the initial target feature is updated based on the positive influence of the feature association information on the initial target feature;

[0262] Based on the object association information between the N candidate objects represented by the updated target feature, the updated target feature is adjusted to obtain the fused target feature.

[0263] Optionally, the interaction unit 903 is used to obtain feature association information between a fused feature map and an initial target feature, and update the initial target feature based on the positive impact of the feature association information on the initial target feature, specifically for:

[0264] For each initial feature point in the initial target feature, the following operations are performed respectively: based on the association relationship between the initial feature point and each fused pixel point in a fused feature map, J fused pixel points that meet the fusion association conditions and their corresponding fusion association weights are obtained; wherein each fusion association weight represents: the association relationship between the corresponding fused pixel point and the initial feature point;

[0265] Based on the obtained J fusion association weights, the J fusion pixel points and the initial feature points are fused to obtain updated feature points corresponding to the initial feature points.

[0266] Optionally, the interaction unit 903 is used to adjust the updated target feature based on the object association information between the N candidate objects represented by the updated target feature to obtain the fused target feature, specifically for:

[0267] For the N candidate objects represented by the updated target features, based on the object association relationship between each candidate object and the N candidate objects, obtain the object weights corresponding to the N candidate objects;

[0268] The obtained N object weights are used to perform weighted sum processing on the updated target features to obtain the fused target features.

[0269] Optionally, the obtaining unit 904 is specifically configured to:

[0270] Obtaining detection parameters associated with each target detection category; wherein each detection parameter represents: the degree of association between each historical object and the corresponding target detection category;

[0271] Based on the acquired detection parameters, feature analysis is performed on the fused target features to obtain the detection probability of each of the N candidate objects belonging to each target detection category;

[0272] For N candidate objects, the following operations are performed respectively: when the maximum detection probability associated with a candidate object meets the preset detection probability, the candidate object is determined to be a target object belonging to the target detection category corresponding to the maximum detection probability.

[0273] The device can be used to execute the methods shown in the various embodiments of the present application. Therefore, for the functions that can be implemented by the various functional modules of the device, reference can be made to the description of the aforementioned embodiments and no further details will be given.

[0274] See also Fig.10 As shown, based on the same technical concept, the embodiment of the present application also provides a computer device 1000, which can be Figure 1 or Figure 2 As shown in the terminal device or server, the computer device 1000 may include a memory 1001 and a processor 1002 .

[0275] The so-called memory 1001 is used to store computer programs executed by the processor 1002. The memory 1001 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the computer device, etc. The processor 1002 may be a central processing unit (CPU), or a digital processing unit, etc. The specific connection medium between the above-mentioned memory 1001 and the processor 1002 is not limited in the embodiments of the present application. The embodiments of the present application are Fig.10 In the embodiment, the memory 1001 and the processor 1002 are connected via a bus 1003. The bus 1003 is connected to the processor 1002 via a bus 1003. Fig.10 The connection between other components is shown by bold lines, and is not intended to be limiting. The so-called bus 1003 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.10 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0276] The memory 1001 may be a volatile memory, such as a random-access memory (RAM); the memory 1001 may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD), or the memory 1001 is any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 1001 may be a combination of the above memories.

[0277] The processor 1002 is used to execute the method executed by the device in each embodiment of the present application when calling the computer program stored in the so-called memory 1001.

[0278] In some possible implementations, various aspects of the method provided in the present application may also be implemented in the form of a program product, which includes a program code. When the so-called program product is run on a computer device, the so-called program code is used to enable the so-called computer device to execute the steps of the method according to the various exemplary implementations of the present application described above in this specification. For example, the so-called computer device can execute the method executed by the device in each embodiment of the present application.

[0279] The so-called program product can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0280] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0281] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A target detection method, characterized in that: include: Based on the preset multiple feature map sizes, the corresponding color feature map and depth feature map are extracted for the image to be detected; Based on the multiple feature map sizes, respectively perform feature fusion processing on the corresponding color feature map and depth feature map to obtain at least one fused feature map corresponding to each of the multiple feature map sizes; Acquire initial target features of the image to be detected, wherein the initial target features represent: initial association relationships between N candidate objects and target detection categories preset for the image to be detected; Using each acquired fusion feature map, the initial target feature is iteratively updated in turn to obtain a corresponding updated target feature; wherein, in one updating process, feature association information between a fusion feature map and the initial target feature is acquired, and based on the positive influence of the feature association information on the initial target feature, the initial target feature is updated; Based on the object association information between the N candidate objects represented by the updated target feature, adjusting the updated target feature to obtain a fused target feature; Based on the fused target features, at least one target object belonging to a corresponding target detection category is determined among the N candidate objects to obtain a target detection result of the image to be detected.

2. The method according to claim 1, characterized in that The image to be detected includes: a color image to be detected and a depth image to be detected collected for the same target scene; Then, based on the preset multiple feature map sizes, the corresponding color feature map and depth feature map are extracted for the image to be detected, including: Based on the preset multiple feature map sizes, the following operations are performed on the color map to be detected and the depth map to be detected: Based on a preset feature map size, respectively performing image size transformation processing on the color map to be detected and the initial depth map to be detected to obtain a corresponding initial color feature map and an initial depth feature map; The initial color feature map and the initial depth feature map are respectively subjected to interactive processing of pixels within the map to obtain corresponding color feature maps and depth feature maps.

3. The method according to claim 2, characterized in that The method of performing image size transformation processing on the color image to be detected and the depth image to be detected based on a preset feature image size to obtain a corresponding initial color feature image and an initial depth feature image includes: Based on a preset feature map size, the following operations are performed on the two attribute maps to be detected, namely, the color map to be detected and the depth map to be detected: Based on a size difference value between a size of an attribute graph to be detected and a size of the feature graph, dividing the attribute graph to be detected into a plurality of image blocks to be detected; At least one initial pixel point contained in each of the multiple image blocks to be detected is mapped to a corresponding feature space to obtain a corresponding initial attribute feature map, where the initial attribute feature map is an initial color feature map or an initial depth feature map.

4. The method according to claim 2, characterized in that The step of performing interactive processing of pixels in the initial color feature map and the initial depth feature map to obtain corresponding color feature maps and depth feature maps includes: The following operations are performed on the two initial attribute feature maps, the initial color feature map and the initial depth feature map, respectively: Based on a preset window size, an initial attribute feature map is divided into a plurality of window blocks; wherein each window block includes at least one candidate pixel point in the initial attribute feature map; Self-attention feature interaction processing is performed on at least one candidate pixel point contained in each of the multiple window tiles, and cross-attention feature interaction processing is performed at least between the multiple window tiles to obtain a corresponding attribute feature map, which is a color feature map or a depth feature map.

5. The method according to claim 1, characterized in that The performing feature fusion processing on the corresponding color feature map and the depth feature map based on the multiple feature map sizes to obtain at least one fused feature map corresponding to each of the multiple feature map sizes includes: For each feature map size, perform the following operations: Obtain a color feature map and a depth feature map that match the size of the feature map; In the color feature map, based on the K depth pixels in the depth feature map, feature fusion processing is performed on each color pixel to obtain a corresponding first fusion feature map; In the depth feature map, based on the K color pixels in the color feature map, feature fusion processing is performed on each depth pixel to obtain a corresponding second fusion feature map; The first fused feature map and the second fused feature map are both used as fused feature maps corresponding to the feature map size.

6. The method according to claim 5, characterized in that In the color feature map, based on the K depth pixels in the depth feature map, feature fusion processing is performed on each color pixel to obtain a corresponding first fusion feature map, including: For each color pixel in the color feature map, perform the following operations respectively: Based on the association relationship between the color pixel and each depth pixel in the depth feature map, K depth pixels satisfying the first association condition and their corresponding first fusion weights are obtained; wherein each first fusion weight represents: the association relationship between the corresponding depth pixel and the color pixel; Based on the obtained K first fusion weights, the K depth pixels and the color pixels are fused to obtain first fusion points corresponding to the color pixels; Based on the first fusion point corresponding to each color pixel, a corresponding first fusion feature map is obtained.

7. The method according to claim 6, characterized in that The obtaining, based on the association relationship between the color pixel and each depth pixel in the depth feature map, K depth pixels satisfying a first association condition and their corresponding first fusion weights includes: Based on the first relationship prediction parameter, first association information between each of the depth pixels and the color pixel is obtained; wherein the first relationship prediction parameter represents: the association relationship between the color pixel and each of the depth pixels in the depth feature map; Select K first association information satisfying the first association condition from among the obtained first association information, and obtain corresponding K depth pixel points; Based on the first weight prediction parameter, the influence degree of each depth pixel on the color pixel is calculated respectively, and the first fusion weight corresponding to each of the K depth pixels is obtained; wherein the first weight prediction parameter represents: the correlation relationship between the color pixel and the K depth pixel.

8. The method according to claim 5, characterized in that In the depth feature map, based on the K color pixels in the color feature map, feature fusion processing is performed on each depth pixel to obtain a corresponding second fusion feature map, including: For each depth pixel in the depth feature map, perform the following operations respectively: Based on the association relationship between the depth pixel and each color pixel in the color feature map, K color pixels satisfying the second association condition and their corresponding second fusion weights are obtained; wherein each second fusion weight represents: the association relationship between the corresponding color pixel and the depth pixel; Based on the obtained K second fusion weights, the K color pixels and the depth pixel are fused to obtain a second fusion point corresponding to the depth pixel; Based on the second fusion point corresponding to each depth pixel point, a corresponding second fusion feature map is obtained.

9. The method according to claim 8, characterized in that The obtaining, based on the association relationship between the depth pixel and each color pixel in the color feature map, K color pixels satisfying the second association condition and their corresponding second fusion weights includes: Based on the second relationship prediction parameter, second association information between each of the color pixels and the depth pixel is obtained; wherein the second relationship prediction parameter represents: the association relationship between the depth pixel and each of the color pixels in the color feature map; Select K pieces of second association information satisfying the second association condition from among the obtained pieces of second association information, and obtain corresponding K color pixel points; Based on the second weight prediction parameter, the influence of each color pixel on the depth pixel is calculated respectively to obtain the second fusion weight corresponding to each of the K color pixels; wherein the second weight prediction parameter represents: the correlation relationship between the depth pixel and the K color pixels.

10. The method according to claim 1, characterized in that The acquiring feature association information between a fused feature map and the initial target feature, and updating the initial target feature based on the positive influence of the feature association information on the initial target feature, includes: For each initial feature point in the initial target feature, the following operations are performed respectively: based on the association relationship between the initial feature point and each fused pixel point in a fused feature map, J fused pixel points that meet the fusion association condition and their corresponding fusion association weights are obtained; wherein each fusion association weight represents: the association relationship between the corresponding fused pixel point and the initial feature point; Based on the obtained J fusion association weights, the J fusion pixel points and the initial feature points are fused to obtain updated feature points corresponding to the initial feature points.

11. The method according to claim 1, characterized in that The step of adjusting the updated target feature based on the object association information between the N candidate objects represented by the updated target feature to obtain the fused target feature includes: For the N candidate objects represented by the updated target feature, based on the object association relationship between each candidate object and the N candidate objects, obtaining the object weight corresponding to each of the N candidate objects; The obtained N object weights are used to perform weighted sum processing on the updated target features to obtain fused target features.

12. The method according to any one of claims 1 to 9, characterized in that: The determining, based on the fused target feature, at least one target object belonging to a corresponding target detection category among the N candidate objects comprises: Acquire detection parameters associated with each target detection category; wherein each detection parameter represents: the degree of association between each historical object and the corresponding target detection category; Based on the acquired detection parameters, feature analysis processing is performed on the fused target features to obtain detection probabilities that each of the N candidate objects belongs to each target detection category; For the N candidate objects, the following operations are performed respectively: when the maximum detection probability associated with each of the candidate objects meets the preset detection probability, the candidate object is determined to be a target object belonging to the target detection category corresponding to the maximum detection probability.

13. A target detection device, characterized in that: include: An extraction unit, used for extracting corresponding color feature maps and depth feature maps for the image to be detected based on a plurality of preset feature map sizes; A fusion unit, configured to perform feature fusion processing on the corresponding color feature map and the depth feature map based on the multiple feature map sizes, to obtain at least one fused feature map corresponding to each of the multiple feature map sizes; The interactive unit is used to obtain the initial target feature of the image to be detected, wherein the initial target feature represents: the initial association relationship between N candidate objects and each target detection category preset for the image to be detected; using each fused feature map obtained, iteratively updating the initial target feature in turn to obtain a corresponding updated target feature; wherein, in one updating process, feature association information between a fused feature map and the initial target feature is obtained, and the initial target feature is updated based on the positive influence of the feature association information on the initial target feature; based on the object association information between the N candidate objects represented by the updated target feature, the updated target feature is adjusted to obtain a fused target feature; An obtaining unit is used to determine at least one target object belonging to a corresponding target detection category among the N candidate objects based on the fused target feature, so as to obtain a target detection result of the image to be detected.

14. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 12 are implemented.

15. A computer storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.

16. A computer program product comprising computer program instructions, characterized in that When the computer program instructions are executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.

Images