A method and apparatus for object detection

Abstract

The application discloses a target detection method and device, and relates to the technical field of computers. A specific implementation of the method comprises the following steps: image features of different modalities of a region to be detected are extracted, and modality-specific features and modality-shared features are extracted from the image features, so that target detection of the region to be detected is performed according to the modality-specific features and the modality-shared features. The implementation can retain specific features of different modalities, and target detection can be realized by combining shared features between different modalities, so that the accuracy of a detection result is improved.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method and apparatus for target detection. Background Technology

[0002] Currently, in the field of image recognition technology, improving recognition performance typically involves data acquisition and processing across multiple modalities. Typical modalities include visible light images and infrared images, which differ significantly in style and each possesses its own advantages. To ensure the accuracy of the final recognition result, data from different modalities needs to be processed comprehensively. A common approach is to fuse the data from different modalities, including feature summation and concatenation, mapping the features of both modalities to the same feature space to reduce intermodal differences and capture shared modal features.

[0003] However, existing fusion methods tend to map different modal features to the same feature space and minimize feature differences to facilitate processing and analysis under a unified representation. This strategy aims to enhance the model's comprehensive understanding of multimodal data by eliminating feature differences between modalities. However, this approach focuses too much on the degree of feature fusion, easily neglecting the inherent characteristics and advantages of different modal features, thus affecting the accuracy of the recognition results. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a method and apparatus for target detection that can simultaneously retain the inherent characteristics of different modalities and the fusion characteristics between modalities, and combine the two for target detection, thereby greatly improving the accuracy of the detection results.

[0005] To achieve the above objectives, according to one aspect of the present invention, a method for target detection is provided, comprising: Extract image features of different modalities in the region to be detected, and extract modality-specific features and modality-shared features based on the image features of the different modalities; Target detection is performed on the region to be detected based on the modality-specific features and the modality-shared features.

[0006] Optionally, the step of extracting modality-specific features and modality-shared features based on the image features of the different modalities includes: Specific features are extracted based on the image features of the different modalities to obtain modality-specific features for each modality; The image features of the different modalities are summed, and the shared features are extracted based on the summation results to obtain the modality shared features.

[0007] Optionally, the step of extracting specific features based on the image features of different modalities to obtain modality-specific features for different modalities includes: The image features of the different modalities are subjected to instance normalization processing respectively, and the frequency domain information of the features of the different modalities is extracted based on the processing results; The modality-specific features of the different modes are obtained by calculating based on the frequency domain information and the preset activation function.

[0008] Optionally, the step of extracting shared features based on the summation result to obtain the modality shared features includes: Based on the summation result, feature division is performed to obtain the first feature and the second feature; The feature dependency is determined based on the first feature, and the importance of the image pixels is calculated based on the second feature; The modality-shared features are calculated based on the feature dependencies and the importance of the image pixels.

[0009] Optionally, the image features are multi-stage image features, and the multi-stages are divided according to the number of different convolutional layers in the feature extraction network; The step of extracting specific features based on the image features of different modalities to obtain modality-specific features of different modalities includes: extracting specific features based on the image features of each stage of the different modalities to obtain modality-specific features of each stage of the different modalities; The step of summing the image features of the different modalities and extracting shared features based on the summation results to obtain the modal shared features includes: summing the image features of each stage of the different modalities to obtain the summation results of each stage, and extracting shared features based on the summation results of each stage to obtain the modal shared features of each stage.

[0010] Optionally, the step of performing target detection on the region to be detected based on the modality-specific features and the modality-shared features includes: Predict the target bounding box and its category based on the modality-specific features and the modality-shared features; The detection target of the region to be detected is determined based on the target bounding box and the category of the target bounding box.

[0011] Optionally, before extracting image features of different modalities of the region to be detected, the method further includes: Image data of the area to be detected is acquired by acquisition devices with different modalities synchronized with parameters, thus obtaining image data of different modalities; The extraction of image features of different modalities in the region to be detected includes: Feature extraction is performed on the image data of different modalities to obtain the image features of the region to be detected in different modalities.

[0012] According to another aspect of the present invention, an apparatus for target detection is provided, comprising: The extraction module is used to extract image features of different modalities in the region to be detected, and to extract modality-specific features and modality-shared features based on the image features of the different modalities. The detection module is used to perform target detection on the region to be detected based on the modality-specific features and the modality-shared features.

[0013] According to another aspect of the present invention, an electronic device is provided, comprising: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the target detection method provided in the embodiments of the present invention.

[0014] According to another aspect of the present invention, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the target detection method provided in the embodiments of the present invention.

[0015] According to another aspect of the present invention, a computer program product is provided, including a computer program that, when executed by a processor, implements the target detection method provided in the embodiments of the present invention.

[0016] One embodiment of the above invention has the following advantages or beneficial effects: it can identify targets in a region by combining shared features between different modalities while retaining the inherent advantages of features from different modalities, thus ensuring the accuracy of the detection results. Specifically, by extracting modality-specific features to achieve the extraction of advantageous features from different modal data, an accurate data foundation is provided for target detection. At the same time, shared features between modalities can integrate the commonalities of features from different modalities, further improving the accuracy of the detection results.

[0017] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0018] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein: Figure 1 This is a schematic diagram of the main steps of a target detection method according to an embodiment of the present invention; Figure 2 A schematic diagram of the structure of the modality-specific feature and modality-shared feature coordinator provided in an embodiment of the present invention; Figure 3 This is a structural diagram of the target detection model provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the main modules of a target detection device according to an embodiment of the present invention; Figure 5 This is an exemplary system architecture diagram in which embodiments of the present invention can be applied; Figure 6 This is a schematic diagram of the structure of a computer system suitable for implementing terminal devices or servers of the present invention. Detailed Implementation

[0019] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0020] It should be noted that the technical solutions disclosed in this invention, regarding the collection, updating, analysis, processing, use, transmission, and storage of user personal information, all comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken to prevent unauthorized access to user personal information data and to safeguard user personal information security, network security, and national security.

[0021] It should be noted that the collection, use, storage, sharing and transfer of user personal information involved in the technical solution of the present invention all comply with the provisions of relevant laws and regulations, and require notification to users and obtaining their consent or authorization. When applicable, user personal information is subjected to de-identification and / or anonymization and / or encryption technical processing.

[0022] Currently, in scenarios such as object recognition and target detection, it is necessary to collect scene data and extract its features for computation to achieve the purpose of object recognition / detection. With technological advancements, it has been found that collecting data from different modalities during data acquisition and then processing it through fusion and other methods before recognition / detection yields more accurate results. For example, common modal data include visible light images and infrared images, which have significant differences, making direct fusion challenging. Existing methods typically map features from both modalities to the same feature space through feature summation and concatenation to reduce intermodal differences and capture shared features. However, existing methods overemphasize feature fusion, often neglecting the unique information and distinctive feature expressions carried by different modalities. In practical applications, each modality often possesses its own specificity and advantages. For example, thermal imaging performs better than visible light in low-light environments, while visible light excels in color and detail representation. Therefore, simply pursuing feature consistency may lead to the neglect of these modal-specific information, failing to fully leverage the unique advantages of each modality.

[0023] To fully consider the shortcomings of the above methods, this invention provides a target detection method that can collect feature data from different modalities during the feature acquisition stage, while simultaneously retaining both the uniqueness of the feature data from each modality and the fusion of feature data between modalities. In the final target detection stage, both types of data are combined for accurate detection, thereby improving the accuracy of the detection results. This invention can be used in target detection scenarios based on multimodal data in different scenarios. The following mainly focuses on the recognition scenario of autonomous vehicles under all-weather, all-time visual perception. Specifically, it focuses on pedestrian detection scenarios based on autonomous vehicle driving, including extreme environmental scenarios. These environmental scenarios include nighttime driving scenarios, severe weather scenarios (fog, rain, snow, smoke, etc., which reduce visibility), and strong light interference scenarios (glare scenarios such as facing oncoming vehicle high beams, direct sunlight / sunset, etc.).

[0024] In the above scenarios, data acquired by infrared images in extreme environments is more effective than data acquired by visible light images, while data acquired by visible light images in non-extreme environments is more effective.

[0025] Figure 1 This is a schematic diagram of the main flow of a target detection method according to an embodiment of the present invention. Figure 1 As shown, the target detection method mainly includes steps S101 to S102.

[0026] Step S101: Extract image features of different modalities in the region to be detected, and extract modality-specific features and modality-shared features based on the image features of different modalities.

[0027] Before target detection, data acquisition is necessary, specifically acquiring image features of the target area in different modalities. These image features are determined using data acquired by different modal data acquisition devices. Specifically, visible light cameras and infrared cameras can be used to simultaneously acquire image data of the target area, and the acquired data is stored. To ensure the synchronization of data acquired by different modal data acquisition devices, the devices need to be calibrated to maintain synchronized hardware settings, including pose, field of view, exposure time, and frame rate. This hardware synchronization resolves content differences between image data from different modalities, including reducing ensemble distortion while maintaining the reliability of data acquired by different modal devices. The data acquired in this way consists of two sets of image data that are strictly synchronized in time and spatially aligned. Preliminary feature extraction is then performed based on this image data to obtain the image features of different modalities.

[0028] After obtaining image features from different modalities, modality-specific features and modality-shared features can be extracted based on these features. Modality-specific features retain the advantageous features of different modalities, while modality-shared features are extracted based on the fusion of modality features. Combining these two types of features enables better object detection. Modality-specific features can be extracted separately based on the image features of different modalities. However, for modality-shared features, it is necessary to first fuse the image features of different modalities, and then extract features based on the fusion result.

[0029] In one embodiment, the calibration of the device is an offline process for precisely measuring the intrinsic and extrinsic parameters of the camera. In the application scenario of this invention, this mainly refers to "stereo calibration" and "image correction." The process involves using a special calibration plate (e.g., a checkerboard pattern with internal heating elements, or a checkerboard pattern with a temperature difference from the background at room temperature) that is clearly visible in both modalities. The calibration plate is placed at different positions and angles, allowing the dual-modal cameras to simultaneously acquire multiple sets of images. An algorithm is used to detect the pixel coordinates of the calibration plate corner points in both sets of images. Calculations include: Intrinsic parameter calibration: determining the characteristics of each camera, such as focal length, optical center, and distortion coefficient. This helps correct lens distortion. Extrinsic parameter calibration (core): calculating the rotation matrix and translation vector of the infrared camera relative to the visible light camera—that is, the precise relative pose—using matched corner point pairs. Simultaneously, the difference in field of view between the two cameras is also determined.

[0030] In one embodiment, data preprocessing is required for the collected data from different modalities before subsequent feature extraction operations can be performed. Data preprocessing includes image denoising, distortion correction, color adjustment, image alignment, and camera parameter recording and calibration. Image denoising: State-of-the-art image denoising algorithms, such as convolutional neural networks (CNNs) and adaptive filtering techniques, are employed to effectively remove noise from the image. These algorithms intelligently identify and eliminate random and texture noise while preserving edge details and texture information. This step not only improves the visual clarity of the image but also enhances the accuracy of subsequent image processing algorithms. Distortion correction: A precise lens distortion model is used to correct lens distortion in the camera. Common problems such as fisheye effect and barrel distortion are corrected to ensure accurate image geometry. The corrected image accurately reflects the spatial relationships of the shooting scene, greatly improving the accuracy of measurement and analysis. Color adjustment: Color correction algorithms are used to adjust the colors of images captured by different cameras. Techniques such as white balance adjustment, color mapping, and gamma correction are used to ensure color consistency in the images. This process eliminates color deviations caused by differences in camera sensors and variations in ambient lighting, enabling seamless visual fusion of images in multi-camera systems. Image Alignment: Advanced feature matching and optical flow algorithms achieve precise image alignment. By identifying and matching feature points in the images, spatial alignment of images captured by different cameras is achieved. This not only ensures the accuracy of multi-view image fusion but also provides a reliable data foundation for stereo vision and 3D reconstruction. Camera Parameter Recording and Calibration: Detailed internal and external parameters of each intersection camera are recorded, including focal length, aperture, angle of view, and installation position. These parameters are precisely calibrated to ensure consistency and accuracy in image processing under different environmental conditions. Regular calibration and parameter updates allow adaptation to any physical or environmental changes in the cameras. After these comprehensive preprocessing steps, images and camera parameters are carefully prepared, ensuring high-quality and consistent input data. This lays a solid foundation for subsequent analysis and processing, contributing to improved overall system reliability and accuracy. Whether for real-time monitoring, intelligent traffic analysis, or complex computer vision tasks, these preprocessing steps are crucial, ensuring stable system operation in various complex environments.

[0031] Step S102: Target detection is performed on the region to be detected based on modality-specific features and modality-shared features.

[0032] After obtaining modality-specific features and modality-shared features, target detection can be performed on the region to be detected based on the data from these two features. During detection, the main focus is on detecting the bounding box data of the target, and the target detection result is determined based on the bounding box data, such as whether a pedestrian is present. To perform target detection, the data from the two features can be input into a series of convolutional layers for computation, and the detection results are output.

[0033] The target detection method provided by the embodiments of the present invention can comprehensively identify targets in a region by combining shared features between different modalities while retaining the inherent advantages of features from different modalities, thus ensuring the accuracy of the detection results. Specifically, by extracting modality-specific features, the advantageous features of different modal data are extracted, providing an accurate data foundation for target detection. Simultaneously, the shared features between modalities can integrate the commonalities of features from different modalities, further improving the accuracy of the detection results.

[0034] In one embodiment, modality-specific features and modality-shared features are extracted based on image features of different modalities, including: extracting specific features based on image features of different modalities to obtain modality-specific features of different modalities; summing the features of image features of different modalities, and extracting shared features based on the summation result to obtain modality-shared features.

[0035] When extracting modality-specific features, to avoid the influence of image features from different modalities, different feature extractors can be used to perform the extraction operation. Taking visible light image features as an example, a corresponding modality-specific RGB (a color standard) feature extractor can be set up, and feature extraction can be performed through this extractor. For infrared image features, a corresponding modality-specific infrared feature extractor can be set up, and feature extraction can be performed through this extractor.

[0036] When extracting shared features, it is necessary to first sum the image features of different modalities. This summation operation is pixel-level and involves dividing the summation result into two branches according to channels. One branch is used to learn the dependencies between channels, and the other branch is used to learn the importance of pixels. The two branches are then combined to extract modal shared features.

[0037] In one embodiment, feature fusion can be performed based on image features of different modalities, and then modality-shared features can be extracted based on the fusion results.

[0038] In one embodiment, to ensure the uniqueness of data from different modalities when extracting image features, different neural networks are needed for feature extraction to avoid mutual interference between data from different modalities. For example, two ResNet50 networks (50-layer deep residual networks) with non-shared weights can be used to extract features from visible light and infrared image data to obtain image features from different modalities. The ResNet50 network consists of five stages, each composed of multiple residual blocks. Each residual block contains three convolutional layers, with the last convolutional layer having a stride of 2, used to generate multi-scale features for different modalities. Figure 2 The diagram shows a schematic representation of the modality-specific feature and modality-shared feature coordinator provided in an embodiment of the present invention. The two ResNet50 networks in the diagram can adapt to the data characteristics of different modalities, better generating feature data for the corresponding modality. Specifically, the generated multi-scale features can be represented as follows:

[0039] in, This represents the multi-scale feature map (multi-scale feature data) obtained after passing an RGB image through ResNet50, where the subscript number indicates which stage the feature map was obtained in. This represents the input from an RGB camera. Similarly, when the input is an infrared image, we can obtain... .

[0040] The target detection method provided by the embodiments of the present invention can extract features of data of different modalities through different feature extraction networks, so as to ensure the uniqueness of features of different modalities, avoid the influence between data of different modalities, and preserve the reliability of the original data.

[0041] In one embodiment, specific features are extracted based on image features of different modalities to obtain modality-specific features of different modalities, including: performing instance normalization processing on image features of different modalities respectively, and extracting frequency domain information of features of different modalities based on the processing results; calculating based on frequency domain information and preset activation functions to obtain modality-specific features of different modalities.

[0042] When performing modality-specific feature extraction, the image features of different modalities can first be normalized. Taking modality-specific feature extraction of visible light images as an example, it can be performed using a modality-specific RGB feature extractor. Specifically, firstly, as... Figure 2 The input feature map shown Instance normalization allows the model to focus more on the detailed features of each sample, rather than relying on global statistics for the entire batch, denoted as... Here, H, W, and C represent the height, width, and number of channels of the feature map, respectively. This effectively removes irrelevant background information, resulting in a feature map that focuses more on local details and further emphasizes the texture and structure of the foreground object. Then, channel attention with different frequency distributions is applied... Modeling is performed to obtain the mask. The detailed structure of channel attention will first consider the input... Divide into n sub-vectors, represented as ,in, ,and For each subvector, a corresponding two-dimensional discrete cosine transform (DCT) frequency component is assigned based on its features. Frequency domain information is extracted, compressing spatial domain details into lower-dimensional frequency components, which can be represented as:

[0043] in, Indicates the first This involves several subvectors. This not only preserves important frequency information but also highlights key details through compression. By concatenating different subvectors... The overall frequency vector can be obtained. The frequency components are processed through a fully connected (FC) layer to obtain channel attention weights, which can be represented as:

[0044] in, This represents the sigmoid activation function. This represents the output after the frequency component is activated. It can then be obtained again through the FC layer. Then through with Multiplication yields mode-specific RGB feature maps. , can be represented as:

[0045] Similarly, the modality-specific RGB feature extractor has the same structure as the modality-specific infrared feature extractor and can obtain modality-specific infrared features. .

[0046] In one embodiment, extracting shared features based on the summation result to obtain modal shared features includes: performing feature segmentation based on the summation result to obtain a first feature and a second feature; determining feature dependencies based on the first feature and calculating the importance of image pixels based on the second feature; and calculating modal shared features based on the feature dependencies and the importance of image pixels.

[0047] When extracting shared features, it is necessary to combine the features of two images for comprehensive calculation. This can be achieved using a modality-sharing feature extractor. Figure 2The feature maps from different modalities shown are first summed at the pixel level, and then divided into... Subvectors Next, along the passage... It is divided into two branches, with the input of one branch being... One branch is used to capture the interdependencies between different channels, i.e., feature dependencies. The input to the other branch is... This is used to establish spatial relationships between different pixels, i.e., the importance of image pixels. For cross-channel dependency establishment, channel statistics are first obtained through Global Average Pooling (GAP). Subsequently, channel attention weights are generated using the Sigmoid activation function, and then multiplied by... Obtaining the weighted feature map enhances the model's ability to selectively focus on features from different channels, which can be represented as:

[0048] in, This represents the output of channel attention. For spatial attention, first... By employing group normalization to obtain the spatial statistical norm, and then enhancing the feature representation through an FC layer, it can be expressed as:

[0049] Finally, all sub-vectors are concatenated along the channel dimension, and the channel shuffling operator is used to realize cross-group information flow, resulting in... To fully utilize multi-scale shared features, for the second and subsequent modality sharing, we use the feature map from the previous modality sharing. As input and combined with the shared features obtained at the current stage Combine them. First, After a convolution operation with a kernel size of 1, an upsampling operation is performed to adjust its shape to match. The shape. Then, By mapping to a new feature space through a multilayer perceptron (MLP), thus obtaining... and Subsequently, execution and Matrix multiplication is performed between the matrix components, followed by a softmax operation to obtain the attention weights. Finally, element-wise multiplication is used to... and and Serial feature map Multiply, we get , can be represented as:

[0050] in, and Generated in series and based on .

[0051] In one embodiment, the image features are multi-stage image features, and the multi-stage is divided according to the number of different convolutional layers in the feature extraction network; Specific features are extracted based on the image features of different modalities to obtain modality-specific features for each modality, including: specific features are extracted based on the image features of each stage of different modalities to obtain modality-specific features for each stage of different modalities; The image features of different modalities are summed, and shared features are extracted based on the summation results to obtain modal shared features. This includes summing the image features of each stage of different modalities to obtain the summation results of each stage, and extracting shared features based on the summation results of each stage to obtain modal shared features for each stage.

[0052] To improve recognition performance, image features extracted can be multi-stage features, with each stage corresponding to a different number of convolutional layers. For example, five stages can be set, each stage consisting of multiple residual blocks, each residual block containing three convolutional layers, and the last convolutional layer having a stride of 2, generating multi-scale features. Therefore, when extracting specific features for different modalities, extraction can be performed according to different stages to obtain modality-specific features for each stage, which can then be aggregated to obtain the modality-specific features for that modality.

[0053] Correspondingly, the modal shared features generated based on the summation results are also generated according to different stages. First, the summation result is the sum of the image features of each stage, and then the modal shared feature extraction results of the previous stage are combined with the modal shared feature extraction results of the current stage in a cascade manner to obtain the final modal shared features output by the last stage.

[0054] In one embodiment, target detection is performed on the region to be detected based on modality-specific features and modality-shared features, including: predicting target bounding boxes and target bounding box categories based on modality-specific features and modality-shared features; and determining the detection target in the region to be detected based on the target bounding boxes and target bounding box categories.

[0055] When identifying targets based on modality-specific and modality-shared features, these two features can be input into the prediction unit to obtain the output target bounding box and class probability. The class of the target bounding box is then determined based on the class probability to determine whether there are obstacles such as pedestrians. In addition, the confidence level corresponding to the above results can also be output. Only results that meet a certain confidence threshold are considered reliable; otherwise, the output results are discarded.

[0056] In one embodiment, modality-specific and modality-shared features can be concatenated before target identification and then used as input to a multi-scale feature interaction neck network for better target detection. The neck network typically uses a Feature Pyramid Network (FPN) and a Path Aggregation Network (PANet) to fuse multi-scale features. The FPN combines low-resolution, semantically strong features with high-resolution, semantically weak features via a bottom-up path, improving the model's ability to detect targets at different scales. PANet further enhances feature propagation and information flow, improving the model's localization accuracy and classification ability through top-down path aggregation. This design can more effectively handle different scales and complex backgrounds in images, improving detection accuracy and robustness.

[0057] In one embodiment, before extracting image features of different modalities of the region to be detected, the method further includes: acquiring image data of the region to be detected through acquisition devices of different modalities with synchronized parameters to obtain image data of different modalities; extracting image features of different modalities of the region to be detected includes: extracting features from the image data of different modalities to obtain image features of different modalities of the region to be detected.

[0058] During the data acquisition phase, to ensure the synchronization of data from different modalities, multiple acquisition devices with synchronized parameters are required to acquire image data of the area to be detected. The number of acquisition devices is the same as the number of modalities. Parameter synchronization includes the synchronization of parameters such as pose, field of view, exposure time, and frame rate. During acquisition, different devices are used, including using a visible light camera to acquire visible light image data of the area to be detected, and using an infrared camera to acquire infrared image data of the area to be detected, and image features are extracted from the corresponding image data.

[0059] The target detection method provided by the embodiments of the present invention can take into account the special characteristics of data from different modalities, retain the unique features of data from different modalities during target detection, and further combine shared features between different modalities for target detection, which helps to improve the accuracy of detection results. Specifically, through a novel modality-specific and modality-shared feature coordinator, modality-specific features and modality-shared features from different modalities can be extracted simultaneously, achieving more accurate multimodal fusion. This coordinator, through a specially designed network structure, can effectively integrate the common information of multimodal data while maintaining the independent features of each modality, contributing to fast and accurate target detection operations.

[0060] like Figure 3 The diagram shows the structure of the target detection model provided in this embodiment of the invention. Visible light and infrared image data undergo multi-stage feature extraction to obtain modality-specific features for different modalities. Features from each stage are summed to extract modality-shared features for each stage, and multiple stages are cascaded to obtain the final modality-shared features. The obtained modality-specific and modality-shared features are input into the Neck for processing, and the processing result is input into the Head unit for detection to obtain the target detection result. The extraction of modality-specific and modality-shared features can be achieved through a Modality-Specific and Modality-Shared Module (MSMH).

[0061] Figure 4 This is a schematic diagram of the main modules of the target detection device provided in an embodiment of the present invention. Figure 4 As shown, the target detection device 400 mainly includes an extraction module 401 and a detection module 402.

[0062] The extraction module 401 is used to extract image features of different modalities in the region to be detected, and to extract modality-specific features and modality-shared features based on the image features of different modalities; The detection module 402 is used to perform target detection on the region to be detected based on modality-specific features and modality-shared features.

[0063] The target detection apparatus provided by the embodiments of the present invention can comprehensively identify targets in a region by combining shared features between different modalities while retaining the inherent advantages of features of different modalities, thus ensuring the accuracy of the detection results. Specifically, by extracting modality-specific features to achieve the extraction of advantageous features from different modal data, an accurate data foundation is provided for target detection. At the same time, shared features between modalities can integrate the commonalities of features from different modalities, further improving the accuracy of the detection results.

[0064] In one embodiment, the extraction module 401 is further configured to: extract specific features based on the image features of different modalities to obtain modality-specific features of different modalities; sum the features of the image features of different modalities, and extract shared features based on the summation result to obtain modality-shared features.

[0065] In one embodiment, the extraction module 401 is further configured to: perform instance normalization processing on image features of different modalities respectively, and extract frequency domain information of features of different modalities according to the processing results; and calculate based on the frequency domain information and a preset activation function to obtain modality-specific features of different modalities.

[0066] In one embodiment, the extraction module 401 is further configured to: perform feature segmentation based on the summation result to obtain a first feature and a second feature; determine feature dependencies based on the first feature and calculate the importance of image pixels based on the second feature; and calculate modality-shared features based on the feature dependencies and the importance of image pixels.

[0067] In one embodiment, the image features are multi-stage image features, and the multi-stage is divided according to the number of different convolutional layers in the feature extraction network; the extraction module 401 is further configured to: extract specific features according to the image features of each stage of different modalities to obtain modality-specific features of each stage of different modalities; sum the features of the image features of each stage of different modalities to obtain the summation result of each stage, and extract shared features according to the summation result of each stage to obtain modality-shared features of each stage.

[0068] In one embodiment, the detection module 402 is further configured to: predict the target bounding box and the category of the target bounding box based on modality-specific features and modality-shared features; and determine the detection target of the region to be detected based on the target bounding box and the category of the target bounding box.

[0069] In one embodiment, the target detection apparatus 400 further includes: an acquisition module 403 (not shown in the figure), used to acquire image data of the area to be detected through acquisition devices of different modalities synchronized with parameters, to obtain image data of different modalities; and an extraction module 401, used to extract features from the image data of different modalities respectively, to obtain image features of different modalities of the area to be detected.

[0070] The target detection apparatus provided by the embodiments of the present invention can take into account the specific characteristics of data from different modalities, retain the unique features of data from different modalities during target detection, and further combine shared features between different modalities for target detection, which helps to improve the accuracy of detection results. Specifically, through a novel modality-specific and modality-shared feature coordinator, modality-specific features and modality-shared features from different modalities can be extracted simultaneously, achieving more accurate multimodal fusion. This coordinator, through a specially designed network structure, can effectively integrate the common information of multimodal data while maintaining the independent features of each modality, contributing to fast and accurate target detection operations.

[0071] Figure 5 An exemplary system architecture 500 is shown, in which the target detection method or apparatus of embodiments of the present invention can be applied.

[0072] like Figure 5 As shown, system architecture 500 may include terminal devices 501, 502, and 503, a network 504, and a server 505. Network 504 serves as the medium for providing communication links between terminal devices 501, 502, and 503 and server 505. Network 504 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0073] Users can use terminal devices 501, 502, and 503 to interact with server 505 via network 504 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 501, 502, and 503, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0074] Terminal devices 501, 502, and 503 can be various electronic devices with displays that support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0075] Server 505 can be a server that provides various services, such as a backend management server that supports shopping websites browsed by users using terminal devices 501, 502, and 503 (for example only). The backend management server can analyze and process data such as received target detection requests, and feed back the processing results (such as detection results - for example only) to the terminal devices.

[0076] It should be noted that the target detection method provided in this embodiment of the invention is generally executed by server 505, and correspondingly, the target detection device is generally set in server 505.

[0077] It should be understood that Figure 5The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0078] The following is for reference. Figure 6 It shows a schematic diagram of the structure of a computer system 600 suitable for implementing terminal devices or servers of the present invention. Figure 6 The terminal device or server shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0079] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 602 or programs loaded from storage section 608 into random access memory (RAM) 603. The RAM 603 also stores various programs and data required for the operation of the system 600. The CPU 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0080] The following components are connected to I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to I / O interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 610 as needed so that computer programs read from it can be installed into storage section 608 as needed.

[0081] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by central processing unit (CPU) 601, it performs the functions defined above in the system of this invention.

[0082] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0083] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0084] The units or modules described in the embodiments of the present invention can be implemented in software or hardware. The described units or modules can also be housed in a processor; for example, a processor can be described as including an extraction module and a detection module. The names of these units or modules do not necessarily limit the specific unit or module itself. For example, an extraction module can also be described as "a module for extracting image features of different modalities of a region to be detected, and extracting modality-specific features and modality-shared features based on the image features of different modalities."

[0085] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, which, when executed by the device, cause the device to include: Extract image features of different modalities in the region to be detected, and extract modality-specific features and modality-shared features based on the image features of different modalities; Target detection is performed on the region to be detected based on modality-specific features and modality-shared features.

[0086] According to the technical solution of the present invention, the target in the region can be comprehensively identified by combining the shared features of different modalities while retaining the inherent advantages of the features of different modalities, thus ensuring the accuracy of the detection results. Specifically, by extracting modality-specific features to achieve the extraction of advantageous features from different modal data, an accurate data foundation is provided for target detection. At the same time, the shared features of different modalities can integrate the commonalities of features from different modalities, further improving the accuracy of the detection results.

[0087] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for target detection, characterized in that, include: Extract image features of different modalities in the region to be detected, and extract modality-specific features and modality-shared features based on the image features of the different modalities; Target detection is performed on the region to be detected based on the modality-specific features and the modality-shared features.

2. The method according to claim 1, characterized in that, The step of extracting modality-specific features and modality-shared features based on the image features of the different modalities includes: Specific features are extracted based on the image features of the different modalities to obtain modality-specific features for each modality; The image features of the different modalities are summed, and the shared features are extracted based on the summation results to obtain the modality shared features.

3. The method according to claim 2, characterized in that, The extraction of specific features based on the image features of different modalities to obtain modality-specific features for different modalities includes: The image features of the different modalities are subjected to instance normalization processing respectively, and the frequency domain information of the features of the different modalities is extracted based on the processing results; The modality-specific features of the different modes are obtained by calculating based on the frequency domain information and the preset activation function.

4. The method according to claim 2, characterized in that, The extraction of shared features based on the summation result to obtain the modality shared features includes: Based on the summation result, feature division is performed to obtain the first feature and the second feature; The feature dependency is determined based on the first feature, and the importance of the image pixels is calculated based on the second feature; The modality-shared features are calculated based on the feature dependencies and the importance of the image pixels.

5. The method according to claim 2, characterized in that, The image features are multi-stage image features, and the multi-stages are divided according to the number of different convolutional layers in the feature extraction network; The step of extracting specific features based on the image features of different modalities to obtain modality-specific features of different modalities includes: extracting specific features based on the image features of each stage of the different modalities to obtain modality-specific features of each stage of the different modalities; The step of summing the image features of the different modalities and extracting shared features based on the summation results to obtain the modal shared features includes: summing the image features of each stage of the different modalities to obtain the summation results of each stage, and extracting shared features based on the summation results of each stage to obtain the modal shared features of each stage.

6. The method according to claim 1, characterized in that, The step of performing target detection on the region to be detected based on the modality-specific features and the modality-shared features includes: Predict the target bounding box and its category based on the modality-specific features and the modality-shared features; The detection target of the region to be detected is determined based on the target bounding box and the category of the target bounding box.

7. The method according to any one of claims 1-6, characterized in that, Before extracting image features of different modalities in the region to be detected, the method further includes: Image data of the area to be detected is acquired by acquisition devices with different modalities synchronized with parameters, thus obtaining image data of different modalities; The extraction of image features of different modalities in the region to be detected includes: Feature extraction is performed on the image data of different modalities to obtain the image features of the region to be detected in different modalities.

8. A target detection device, characterized in that, include: The extraction module is used to extract image features of different modalities in the region to be detected, and to extract modality-specific features and modality-shared features based on the image features of the different modalities. The detection module is used to perform target detection on the region to be detected based on the modality-specific features and the modality-shared features.

9. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-7.

10. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-7.

Images