Object category detection method and device based on multiple modes and computer storage medium

Abstract

The invention discloses an object category detection method and device based on multiple modalities and a computer storage medium, and the method comprises the steps: obtaining a first modal image which comprises a to-be-detected object; performing modal conversion processing on the first modal image to obtain a second modal image; filtering the illumination information in the second modal image to obtain a filtered second modal image; and performing category detection processing based on the first modal image and the filtered second modal image to obtain a target category of the to-be-detected object. Therefore, the accuracy of object category detection is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a multimodal object category detection method, an electronic device and a computer-readable storage medium. Background Art

[0002] With the rapid development of deep learning technology, object recognition technology provides convenience and intelligence for smart devices and interactive smart products with its remarkable accuracy. A secure object recognition system can defend against attacks such as photo printing, video replay, and 3D bionic masks. In order to ensure the security of the object recognition system, it is particularly important to perform object category detection before object recognition.

[0003] Currently, the image to be detected is collected by a camera and feature analysis is performed to determine the category of the object in the image to be detected. However, the above method has the problem of low detection accuracy.

[0004] Therefore, there is an urgent need for an object category detection method that can improve the accuracy of object category detection. Summary of the invention

[0005] The main technical problem solved by the present application is to provide a multimodal object category detection method, an electronic device and a computer-readable storage medium, which can improve the detection accuracy of object categories.

[0006] To solve the above technical problems, the present application provides a multi-modal object category detection method, the method comprising: acquiring a first modal image, the first modal image comprising an object to be detected; performing modal conversion processing on the first modal image to obtain a second modal image; filtering the illumination information in the second modal image to obtain a filtered second modal image; performing category detection processing based on the first modal image and the filtered second modal image to obtain a target category of the object to be detected.

[0007] In one embodiment, the step of performing modal conversion processing on the first modal image to obtain the second modal image includes: inputting the first modal image into a target category detection model to obtain the second modal image output by the modal conversion module of the target category detection model, and / or; the step of performing category detection processing based on the first modal image and the filtered second modal image to obtain the target category of the object to be detected includes: inputting the first modal image and the filtered second modal image into the target category detection model to obtain the target category of the object to be detected determined by the category detection module of the target category detection model.

[0008] In one embodiment, the step of inputting the first modal image into the target category detection model to obtain the second modal image output by the modal conversion module of the target category detection model includes: performing feature extraction processing on the first modal image by the modal conversion module to obtain the latent spatial features of each pixel in the first modal image; performing modal conversion processing according to the latent spatial features of each pixel in the first modal image to obtain the second modal image, wherein the second modal image includes the latent spatial features of each pixel; and performing upsampling processing on the latent spatial features of each pixel in the second modal image to obtain the pixel spatial features of each pixel in the second modal image.

[0009] In one embodiment, the step of inputting the first modal image and the filtered second modal image into the target category detection model to obtain the target category of the object to be detected determined by the category detection module of the target category detection model includes: performing modal fusion processing on the first modal image and the filtered second modal image by the category detection module of the target category detection model to obtain a modal fusion image; determining the detection result output by the category detection module according to the modal fusion features of the modal fusion image; and determining that the category of the object to be detected is the target category in response to the detection result indicating that the probability that the category of the object to be detected belongs to the target category is greater than a preset probability threshold.

[0010] In one embodiment, the step of filtering the illumination information in the second modal image to obtain a filtered second modal image includes: filtering the illumination information of each pixel in the second modal image in a first direction to obtain first filtered data of each pixel in the first direction; filtering the illumination information of each pixel in the second modal image in a second direction to obtain second filtered data of each pixel in the second direction; performing pixel convolution operation based on the first filtered data and the second filtered data of each pixel in the second modal image to obtain the filtered second modal image.

[0011] In one embodiment, the step of filtering the illumination information of each pixel in the second modality image in the first direction to obtain first filtered data of each pixel in the first direction includes: obtaining the horizontal coordinate and the vertical coordinate of each pixel in the second modality image; in response to the sum of the squares between the horizontal coordinate and the vertical coordinate of the pixel being greater than a preset threshold, obtaining the arc tangent value of the coordinate ratio between the vertical coordinate and the horizontal coordinate of the pixel; and determining the ratio between the cosine value of the arc tangent value of the coordinate ratio and the sum of the squares as the first filtered data of the pixel in the first direction.

[0012] In one embodiment, the method further includes: obtaining a first modality image sample set to be trained, the first modality image sample set to be trained including first modality image samples to be trained and sample labels of each first modality image sample, the sample label representing a sample category of a sample object in the first modality image sample; inputting the first modality image samples in the first modality image sample set to be trained into a modality conversion module of a category detection model to be trained, and obtaining second modality image sample data output by the modality conversion module of the category detection model to be trained, the second modality image sample data including second modality image samples; and The method comprises the following steps: determining a modal conversion loss value based on the sample label of each first modal image sample and the corresponding second modal image sample data; inputting the second modal image samples and the filtered second modal image samples into the category detection module of the category detection model to be trained, and obtaining the predicted probability that the category of the sample object in the first modal image samples output by the category detection module of the category detection model to be trained is the target category; determining a category detection loss value based on the sample label of each first modal image sample and the predicted probability; and training the category detection model to be trained based on the modal conversion loss value and / or the category detection loss value until the target category detection model that meets the requirements is obtained.

[0013] In one embodiment, the second modality image sample data also includes feature embeddings of the first modality image samples, and the step of determining the modality conversion loss value based on the sample labels of each first modality image sample and the corresponding second modality image sample data includes: determining a potential feature contrast loss value based on the feature embeddings of each first modality image sample and the number of sample labels of the target category that characterizes the category of the sample object of the corresponding image sample in the first modality image sample set; determining a pixel-level loss value based on the second modality image samples, the filtered second modality image samples and the number of sample labels; determining the modality conversion loss value based on the potential feature contrast loss value and / or the pixel-level loss value.

[0014] To solve the above technical problems, the present application provides an electronic device, including a memory and a processor, wherein the memory stores program instructions, and the processor retrieves the program instructions from the memory to execute the above multimodal-based object category detection method.

[0015] In order to solve the above technical problems, the present application provides a computer-readable storage medium including program data stored therein, and the program data is used to implement the above multimodal-based object category detection method when executed by a processor.

[0016] In the above scheme, the acquired first modality image is subjected to modality conversion processing to obtain the second modality image; and the illumination information in the second modality image is filtered to obtain the filtered second modality image; and then the category detection processing is performed based on the first modality image and the filtered second modality image to obtain the target category of the object to be detected. On the one hand, object category detection can be performed by multi-modal images to improve the accuracy of object category detection; on the other hand, by filtering the illumination information in the second modality image, the influence of illumination information on object category detection can be suppressed, further improving the accuracy of object category detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work, among which:

[0018] Figure 1 is a flowchart of an exemplary embodiment of a multimodal object category detection method shown in the present application;

[0019] Figure 2 is a schematic diagram showing the effect of an exemplary embodiment of modal image filtering processing in a multimodal object category detection method shown in the present application;

[0020] Figure 3 is a flowchart of another exemplary embodiment of a multimodal object category detection method shown in the present application;

[0021] Figure 4 It is a simplified structural diagram of a target category detection model in a multimodal object category detection method shown in the present application;

[0022] Figure 5 is a flowchart of another exemplary embodiment of a multimodal object category detection method shown in the present application;

[0023] Figure 6 is a flowchart of another exemplary embodiment of a multimodal object category detection method shown in the present application;

[0024] Figure 7 yes Figure 1 A flow chart of an exemplary embodiment of step S130 in the multimodal object category detection method is shown;

[0025] Figure 8It is a schematic diagram of the training process of the target category detection model in the multimodal object category detection method shown in the present application;

[0026] Fig. 9 is a schematic diagram of a training framework of a category detection model to be trained in a multimodal object category detection method shown in the present application;

[0027] Fig.10 is a block diagram of a category detection device shown in an exemplary embodiment of the present application;

[0028] Fig.11 It is a structural schematic diagram of an embodiment of an electronic device provided by the present application;

[0029] Fig.12 It is a structural schematic diagram of an embodiment of a computer-readable storage medium provided by the present application. DETAILED DESCRIPTION

[0030] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It will be appreciated that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only some but not all structures related to the present application are shown in the drawings. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the art without making creative work are within the scope of protection of the present application.

[0031] First of all, it should be noted that with the rapid development of deep learning technology, object recognition technology has provided convenience and intelligence for smart devices and interactive smart products with its remarkable accuracy. A secure object recognition system can defend against attacks such as photo printing, video replay, and 3D bionic masks. In order to ensure the security of the object recognition system, it is particularly important to perform object category detection processing before object recognition. Furthermore, object category detection is mainly to prevent the problem of reduced security of object recognition due to object recognition through non-target objects such as photos and videos. In practical applications, object category detection technology is widely used in various scenarios where object verification is required, such as transportation, finance, social networks, etc.

[0032] Currently, the image to be detected is collected by a camera and feature analysis is performed to determine the category of the object in the image to be detected. However, the above method has the problem of low detection accuracy.

[0033] Based on this, the present application proposes a multimodal object category detection method, an electronic device, and a computer-readable storage medium. Figure 1 , Figure 1It is a flowchart of an exemplary embodiment of a multimodal object category detection method shown in the present application.

[0034] The execution subject of the multimodal object category detection method can be a terminal device or a server or other processing device, wherein the terminal device can be a user equipment (UE), a computer, a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The execution subject of the multimodal object category detection method can also be a category detection device. In some possible implementations, the multimodal object category detection method can be implemented by a processor calling a computer-readable instruction stored in a memory.

[0035] Specifically, the multimodal object category detection method of this embodiment includes the following steps:

[0036] Step S110: acquiring a first modality image, where the first modality image includes an object to be detected.

[0037] The first modality image may be a depth image, a grayscale image, an RGB image, etc.

[0038] As for the method of acquiring the first modal image, as an example, the category detection device can collect the depth image through a depth camera, and use the collected depth image as the first modal image. As another example, the category detection device can also collect RGB images through a color camera, and use the collected RGB images as the first modal image. As yet another example, the category detection device can collect binocular images through a binocular camera, and perform matching processing based on the left eye image and the right eye image in the binocular image to obtain the first modal image. Specifically, the category detection device can extract the target left eye image from the left eye image according to the position of the object to be detected in the left eye image; extract the target right eye image from the right eye image according to the position of the object to be detected in the right eye image; and match the target left eye image and the target right eye image to obtain the first modal image. It should be noted that the first modal image in this example can be a depth image.

[0039] The object to be detected can be an object, an animal, a person, etc.

[0040] Step S120: performing modality conversion processing on the first modality image to obtain a second modality image.

[0041] Modality conversion processing refers to converting an image in one form into an image in another form, for example, converting a depth image into an RGB image, converting a depth image into an infrared image, or converting an RGB image into an infrared image.

[0042] The second modality image may be an infrared image, an RGB image, or the like.

[0043] The category detection device performs modality conversion processing on the first modality image to obtain a second modality image. Exemplarily, the category detection device can obtain a preset mapping relationship between the first modality image and the second modality image, and map each pixel in the first modality image through the preset mapping relationship to obtain the second modality image. As another example, the category detection device can also perform deep filtering processing on the first modality image to obtain a filtered first modality image, and then perform morphological processing on the filtered first modality image to obtain the second modality image.

[0044] Step S130 , filtering the illumination information in the second modality image to obtain a filtered second modality image.

[0045] In order to suppress the influence of illumination information on object category detection, the category detection device of the embodiment of the present application filters the illumination information in the second modal image to obtain a filtered second modal image. As an exemplary embodiment, the category detection device can compare the size relationship between the illumination intensity of each pixel in the second modal image and the preset intensity threshold to obtain a comparison result, and adjust the illumination intensity of each pixel in the second modal image according to the comparison result to obtain a filtered second modal image. Specifically, the category detection device can lower the illumination intensity of the pixels in the second modal image whose illumination intensity is greater than the preset intensity threshold. As another example, the category detection device can divide the second modal image into blocks to obtain a plurality of image blocks, adjust the illumination intensity of each pixel in the corresponding image block according to the mean illumination intensity of each image block, obtain each image block after adjusting the illumination intensity, and splice each image block after adjusting the illumination intensity to obtain a filtered second modal image. As another exemplary embodiment, the category detection device can extract modal descriptors from the second modal image to obtain a second modal image. For details, please refer to Figure 2 , Figure 2 The infrared spectrum image IR (Infrared Spectrum) is the first modal image, and the PLGF image (Pattern of local gravitational force) is the second modal image. The PLGF image is obtained by extracting the PLGF descriptor from the IR image.

[0046] It should be noted that, in other embodiments, the category detection device may also filter the noise information in the second modality image to obtain a filtered second modality image.

[0047] Step S140: performing category detection processing based on the first modality image and the filtered second modality image to obtain a target category of the object to be detected.

[0048] The category detection device can perform image fusion processing on the first modality image and the filtered second modality image to obtain a fused image, and perform feature matching based on the fused image features and the preset category features to obtain the target category of the object to be detected in the fused image. Specifically, the category detection device can calculate the feature similarity between the fused image features and the preset category features, and in response to the feature similarity being greater than a similarity threshold, the corresponding preset category is used as the target category of the object to be detected.

[0049] It can be seen that the multimodal object category detection method of the embodiment of the present application performs modality conversion processing on the acquired first modality image to obtain a second modality image; and filters the illumination information in the second modality image to obtain a filtered second modality image; then performs category detection processing based on the first modality image and the filtered second modality image to obtain the target category of the object to be detected. On the one hand, object category detection can be improved by performing object category detection using multimodal images; on the other hand, filtering the illumination information in the second modality image can suppress the influence of illumination information on object category detection, further improving the accuracy of object category detection.

[0050] See also Figure 3 , Figure 3 FIG. 1 is a flow chart of another exemplary embodiment of the multimodal object category detection method shown in the present application. Specifically, the multimodal object category detection method in the embodiment of the present application further includes the following steps:

[0051] Step S310: acquiring a first modality image, where the first modality image includes an object to be detected.

[0052] The detailed description of step S310 in the embodiment of the present application can refer to step S110 in the above embodiment, which will not be repeated here.

[0053] Step S320: input the first modality image into the target category detection model to obtain the second modality image output by the modality conversion module of the target category detection model.

[0054] The modality conversion module can be understood as a subnetwork model in the target category detection model, which is used to convert the first modality image into the second modality image.

[0055] The category detection device inputs the first modality image into the target category detection model to obtain the second modality image output by the modality conversion module of the target category detection model. Exemplarily, the category detection device may perform feature extraction processing on the first modality image through the modality conversion module to obtain the latent spatial features of each pixel in the first modality image; perform modality conversion processing based on the latent spatial features of each pixel in the first modality image to obtain the second modality image, the second modality image includes the latent spatial features of each pixel; upsample the latent spatial features of each pixel in the second modality image to obtain the pixel spatial features of each pixel in the second modality image. For details, please refer to Figure 4 , Figure 4 It is a simple structural diagram of the target category detection model in the embodiment of the present application. The modality conversion module in the target category detection model may include an encoder, a feature extractor and a decoder. Among them, the encoder is composed of 3 convolutional layers, instance normalization and ReLU activation function, which is used to extract features from the pixel space of the first modality image to the latent space features, and obtain the latent space features of each pixel in the first modality image. The feature extractor performs modality conversion processing on the latent space features of each pixel in the first modality image based on the residual structure of ResNet to obtain a second modality image. The feature extractor can also be MobileNet, Xception, Inception, etc. The decoder is symmetrical with the encoder structure, and is used to upsample the latent space features of each pixel in the second modality image to obtain a second modality image including the pixel space features of each pixel.

[0056] Step S330: filtering the illumination information in the second modality image to obtain a filtered second modality image.

[0057] The detailed description of step S330 in the embodiment of the present application can refer to step S130 in the above embodiment, which will not be repeated here.

[0058] Step S340: input the first modality image and the filtered second modality image into the target category detection model to obtain the target category of the object to be detected determined by the category detection module of the target category detection model.

[0059] The category detection device inputs the first modality image and the filtered second modality image into the target category detection model to obtain the target category of the object to be detected determined by the category detection module of the target category detection model. Exemplarily, the category detection device performs modality fusion processing on the first modality image and the filtered second modality image through the category detection module of the target category detection model to obtain a modality fusion image; determines the detection result output by the category detection module according to the modality fusion features of the modality fusion image; and determines that the category of the object to be detected is the target category in response to the detection result indicating that the probability that the category of the object to be detected belongs to the target category is greater than a preset probability threshold. For details, please refer to Figure 4 , the category detection module may include a modality fusion module and a category detector. Figure 4 As shown, the modality fusion module in the category detection module can be used to perform modality fusion processing on the first modality image and the filtered second modality image; the lightweight network function MobileNet in the category detector is used to extract features from the modality fusion image to obtain fusion features; and the fusion features are flattened to obtain fusion features after the expansion operation; and then the fusion features after the expansion operation are activated by the fully connected layer and Sigmoid function in the category detector to obtain the detection result. The detection result represents the probability that the category of the object to be detected belongs to the target category.

[0060] It can be seen that the multimodal object category detection method of the embodiment of the present application obtains a first modal image, the first modal image includes an object to be detected; the first modal image is input into the target category detection model to obtain a second modal image output by the modal conversion module of the target category detection model; the illumination information in the second modal image is filtered to obtain a filtered second modal image; the first modal image and the filtered second modal image are input into the target category detection model to obtain the target category of the object to be detected determined by the category detection module of the target category detection model. On the one hand, the first modal image is modally converted by the modal conversion module to obtain the second modal image, thereby increasing the difference between the first modal image and the second modal image; when the modal fusion image of the first modal image and the filtered second modal image with large differences is input into the category detector for category detection, the detection accuracy of the category detection module is improved; on the other hand, by filtering the illumination information in the second modal image, the influence of the illumination information on the object category detection can be suppressed, thereby further improving the accuracy of the object category detection.

[0061] See also Figure 5 , Figure 5 FIG. 1 is a flow chart of another exemplary embodiment of the multimodal object category detection method shown in the present application. Specifically, the multimodal object category detection method of the embodiment of the present application includes the following steps:

[0062] Step S510: acquiring a first modality image, where the first modality image includes an object to be detected.

[0063] The detailed description of step S510 in the embodiment of the present application can refer to step S110 in the above embodiment, which will not be repeated here.

[0064] Step S520: input the first modality image into the target category detection model to obtain the second modality image output by the modality conversion module of the target category detection model.

[0065] The detailed description of step S520 in the embodiment of the present application can refer to step S320 in the above embodiment, which will not be repeated here.

[0066] Step S530: filtering the illumination information in the second modality image to obtain a filtered second modality image.

[0067] The detailed description of step S530 in the embodiment of the present application can refer to step S130 in the above embodiment, which will not be repeated here.

[0068] Step S540: performing category detection processing based on the first modality image and the filtered second modality image to obtain a target category of the object to be detected.

[0069] The detailed description of step S540 in the embodiment of the present application can refer to step S140 in the above embodiment, which will not be repeated here.

[0070] It can be seen that the multimodal object category detection method of the embodiment of the present application obtains a first modal image, the first modal image includes an object to be detected; the first modal image is input into the target category detection model to obtain a second modal image output by the modal conversion module of the target category detection model; the illumination information in the second modal image is filtered to obtain a filtered second modal image; based on the first modal image and the filtered second modal image, category detection is performed to obtain the target category of the object to be detected. On the one hand, by filtering the illumination information in the second modal image, the influence of the illumination information on the object category detection can be suppressed, and the accuracy of the object category detection can be further improved; on the other hand, the first modal image is modally converted by the modal conversion module to obtain the second modal image, the difference between the first modal image and the second modal image is increased, and the object category determination is conveniently performed based on the modal fusion image determined by the first modal image and the filtered second modal image, thereby improving the accuracy of the object category detection.

[0071] See also Figure 6 , Figure 6FIG. 1 is a flow chart of another exemplary embodiment of the multimodal object category detection method shown in the present application. Specifically, the multimodal object category detection method of the embodiment of the present application includes the following steps:

[0072] Step S610: acquiring a first modality image, where the first modality image includes an object to be detected.

[0073] Step S620: performing modality conversion processing on the first modality image to obtain a second modality image.

[0074] Step S630: filtering the illumination information in the second modality image to obtain a filtered second modality image.

[0075] The detailed description of steps S610 to S630 in the embodiment of the present application can refer to steps S110 to S130 in the above embodiment, which will not be repeated here.

[0076] Step S640: input the first modality image and the filtered second modality image into the target category detection model to obtain the target category of the object to be detected determined by the category detection module of the target category detection model.

[0077] The detailed description of step S640 in the embodiment of the present application can refer to step S340 in the above embodiment, which will not be repeated here.

[0078] It can be seen that the multimodal object category detection method of the embodiment of the present application obtains a first modal image, the first modal image includes an object to be detected; performs modal conversion processing on the first modal image to obtain a second modal image; performs filtering processing on the illumination information in the second modal image to obtain a filtered second modal image; inputs the first modal image and the filtered second modal image into the target category detection model to obtain the target category of the object to be detected determined by the category detection module of the target category detection model. On the one hand, by filtering the illumination information in the second modal image, the influence of the illumination information on the object category detection can be suppressed, and the accuracy of the object category detection can be further improved; on the other hand, by performing object category detection on the modal fusion image of the first modal image and the filtered second modal image through the category detection module of the target category detection model, the accuracy of the object category detection can be improved.

[0079] Based on the above embodiments, the embodiments of the present application adopt Figure 7 The flowchart details how to filter the illumination information in the second modality image. Figure 7 , Figure 7 yes Figure 1The flowchart of an exemplary embodiment of step S130 in the multi-modal object category detection method is shown. Specifically, step S130 filters the illumination information in the second modality image to obtain the filtered second modality image, including the following steps:

[0080] Step S710: filtering the illumination information of each pixel in the second modality image in the first direction to obtain first filtered data of each pixel in the first direction.

[0081] The first direction may be a horizontal coordinate direction of the pixel points, or a vertical coordinate direction of the pixel points.

[0082] The category detection device performs filtering processing on the illumination information of each pixel point in the second modality image in the first direction to obtain the first filtering data of each pixel point in the first direction. Exemplarily, the category detection device can obtain the horizontal coordinate and the vertical coordinate of each pixel point in the second modality image; determine whether the sum of the squares between the horizontal coordinate and the vertical coordinate of each pixel point in the second modality image is greater than a preset threshold; in response to the sum of the squares between the horizontal coordinate and the vertical coordinate of the pixel point being greater than the preset threshold, obtain the arc tangent value of the coordinate ratio between the vertical coordinate and the horizontal coordinate of the pixel point; determine the ratio between the cosine value of the arc tangent value of the coordinate ratio and the square sum as the first filtering data of the pixel point in the first direction. In response to the sum of the squares between the horizontal coordinate and the vertical coordinate of the pixel point being less than or equal to the preset threshold, determine the preset value as the first filtering data of the pixel point in the first direction. The preset threshold can be 0. The preset value can also be 0.

[0083] Specifically, the first filtering data of each pixel point in the first direction satisfies the following formula:

[0084]

[0085] Among them, x represents the horizontal coordinate of the pixel point, y represents the vertical coordinate of the pixel point, and otherwise represents that the sum of the squares of the horizontal and vertical coordinates of the pixel point is less than the preset threshold, that is, (x 2 +y 2 )<0, The first filtering data representing each pixel point in the first direction.

[0086] Step S720: filtering the illumination information of each pixel in the second modality image in the second direction to obtain second filtered data of each pixel in the second direction.

[0087] The second direction may be the vertical coordinate direction of the pixel point, or the horizontal coordinate direction of the pixel point. It should be noted that the first direction and the second direction are different coordinate directions of the pixel point.

[0088] The category detection device performs filtering processing on the illumination information of each pixel in the second modality image in the second direction to obtain the second filtering data of each pixel in the second direction. Exemplarily, the category detection device can obtain the horizontal coordinate and the vertical coordinate of each pixel in the second modality image; determine whether the sum of the squares between the horizontal coordinate and the vertical coordinate of each pixel in the second modality image is greater than a preset threshold; in response to the sum of the squares between the horizontal coordinate and the vertical coordinate of the pixel being greater than the preset threshold, obtain the arc tangent value of the coordinate ratio between the vertical coordinate and the horizontal coordinate of the pixel; determine the ratio between the sine value of the arc tangent value of the coordinate ratio and the square sum as the second filtering data of the pixel in the second direction. In response to the sum of the squares between the horizontal coordinate and the vertical coordinate of the pixel being less than or equal to the preset threshold, determine the preset value as the second filtering data of the pixel in the second direction. The preset threshold can be 0. The preset value can also be 0.

[0089] Specifically, the second filtering data of each pixel point in the second direction satisfies the following formula:

[0090]

[0091] Among them, x represents the horizontal coordinate of the pixel point, y represents the vertical coordinate of the pixel point, and otherwise represents that the sum of the squares of the horizontal and vertical coordinates of the pixel point is less than the preset threshold, that is, The second filtering data representing each pixel point in the second direction.

[0092] Step S730, performing a pixel convolution operation according to the first filtered data and the second filtered data of each pixel point in the second modality image to obtain a filtered second modality image.

[0093] Specifically, the filtered second modality image satisfies the following formula:

[0094]

[0095] in, represents the filtered second modality image, I represents the second modality image, e represents the pixel-by-pixel convolution operation, The first filtering data representing each pixel point in the first direction, The second filtering data representing each pixel in the second direction.

[0096] It can be seen that the multimodal object category detection method of the embodiment of the present application performs filtering processing on the illumination information of each pixel in the second modality image in the first direction to obtain the first filtered data of each pixel in the first direction; performs filtering processing on the illumination information of each pixel in the second modality image in the second direction to obtain the second filtered data of each pixel in the second direction; performs pixel convolution operation based on the first filtered data and the second filtered data of each pixel in the second modality image to obtain the filtered second modality image. In this way, the influence of illumination information on object category detection can be suppressed, and the accuracy of object category detection is improved.

[0097] As an example, if a target category detection model is used for category detection, and the target category detection model includes a modality conversion module and a category detection module, that is, in the embodiment of the present application, the modality conversion module in the target category detection model is used to perform modality conversion processing on the first modality image, and the category detection module in the target category detection model is used to perform category detection processing on the first modality image and the filtered second modality image. The training process of the target category detection model can be referred to Figure 8 , Figure 8 : is a schematic diagram of the training process of the target category detection model in the multimodal object category detection method shown in this application. Specifically, the training process of the target category detection model includes the following steps:

[0098] Step S810, obtaining a first modality image sample set to be trained, wherein the first modality image sample set to be trained includes first modality image samples to be trained and sample labels of each first modality image sample, wherein the sample label represents a sample category of a sample object in the first modality image sample.

[0099] The first modal image sample set includes first modal image samples and sample labels of each first modal image sample. The sample label characterizes the sample category of the sample object in the first modal image sample. It should be further noted that, in order to better train the category detection model, the first modal image sample in the embodiment of the present application specifically includes positive samples and negative samples. It can be understood that the sample category of the positive sample is the same as the target category, and the sample category of the negative sample is different from the target category. Exemplarily, the sample category of the sample object is puppy, the target category is piglet, and the sample category puppy of the sample object is different from the target category piglet, then the sample object of the example is a negative sample. Another exemplary example is that the category of the sample object is mid-size sedan, the target category is mid-size sedan, and the sample category mid-size sedan of the sample object is the same as the target category mid-size sedan, then the sample object of the example is a positive sample.

[0100] Step S820, inputting the first modality image samples in the first modality image sample set to be trained into the modality conversion module of the category detection model to be trained, and obtaining the second modality image sample data output by the modality conversion module of the category detection model to be trained, wherein the second modality image sample data includes the second modality image samples.

[0101] The category detection model to be trained in the embodiment of the present application includes a modality conversion module and a category detection module. For details, please refer to Fig. 9 , Fig. 9 A schematic diagram of the training framework of the category detection model to be trained is shown in FIG. Fig. 9 As shown, the category detection device inputs the first modality image samples in the first modality image sample set to be trained into the modality conversion module of the category detection model to be trained, and obtains the second modality image sample data output by the modality conversion module to be trained. Exemplarily, the second modality image sample data may include the second modality image samples output by the decompressor of the modality conversion module. In another example, the second modality image sample data may also include the second modality image samples output by the decompressor of the modality conversion module and the feature embedding of the first modality image samples output by the feature map of the modality conversion module. It should be noted that the feature map in the modality conversion module may include two convolutional layers for reducing the latent space features of each pixel in the first modality image to a low-dimensional space.

[0102] Step S830: determining a modality conversion loss value according to the sample label of each first modality image sample and the corresponding second modality image sample data.

[0103] Exemplarily, the modality conversion loss value includes a potential feature contrast loss value, and may also include a pixel-level loss value, or include a potential feature contrast loss value and a pixel-level loss value.

[0104] The category detection device may determine the modal conversion loss value according to the sample label of each first modal image sample and the corresponding second modal image sample data. As an example, the category detection device may determine the potential feature contrast loss value according to the feature embedding of each first modal image sample and the number of sample labels in the first modal image sample set that characterize the category of the sample object of the corresponding image sample as the target category; and determine the potential feature contrast loss value as the modal conversion loss value. As another example, the category detection device may determine the pixel-level loss value according to the second modal image sample, the filtered second modal image sample and the number of sample labels; and determine the pixel-level loss value as the modal conversion loss value. As another example, the category detection device may determine the potential feature contrast loss value according to the feature embedding of each first modal image sample and the number of sample labels in the first modal image sample set that characterize the category of the sample object of the corresponding image sample as the target category; determine the pixel-level loss value according to the second modal image sample, the filtered second modal image sample and the number of sample labels; and then use the loss sum between the potential feature contrast loss value and the pixel-level loss value as the modal conversion loss value.

[0105] Specifically, the pixel-level loss value is determined to satisfy the following formula according to the second modality image samples, the filtered second modality image samples, and the number of sample labels:

[0106]

[0107] Among them, L pixel Represents the pixel-level loss value, Characterize the first modality image, Represent the second modality image, y n is a binary control factor. When the category of the sample object in the first modality image sample is consistent with the target category, then y n is 1. When the category of the sample object in the first modality image sample is inconsistent with the target category, y n is 0, N g It represents the number of first modality image samples whose object category is the same as the target category in the first modality image sample set, which can be understood as the number of positive samples in the first modality image sample set. N represents the total number of first modality image samples in the first modality image sample set.

[0108] Specifically, according to the feature embedding of each first modality image sample and the number of sample labels of the target category representing the sample object of the corresponding image sample in the first modality image sample set, the potential feature contrast loss value is determined to satisfy the following formula:

[0109]

[0110] Among them, L latentrepresents the contrast loss value of the potential feature, G, A represent the positive sample set and negative sample set in the first modality image sample set, respectively, Z n Feature embedding that characterizes positive samples, Z g Represents Z n Feature embedding of matched positive samples, Z a Represents Z n Feature embedding of matched negative samples, · Calculate the inner product of two feature embeddings, τ is a hyperparameter.

[0111] Step S840: Input the second modality image samples and the filtered second modality image samples into the category detection module of the category detection model to be trained, and obtain the predicted probability that the category of the sample object in the first modality image samples output by the category detection module of the category detection model to be trained is the target category.

[0112] The filtered second modality image samples refer to those obtained after filtering the second modality image samples. For the filtering method, reference may be made to step S130, which is not limited in the embodiment of the present application.

[0113] The category detection device inputs the second modality image samples and the filtered second modality image samples into the category detection module of the category detection model to be trained, and obtains the predicted probability that the category of the sample object in the first modality image sample output by the category detection module is the target category.

[0114] Step S850: determining a category detection loss value according to the sample label and prediction probability of each first modality image sample.

[0115] Specifically, the category detection loss value is determined according to the sample label and prediction probability of each first modality image sample to satisfy the following formula:

[0116]

[0117] Among them, L bce represents the category detection loss value, N represents the total number of first modality image samples in the first modality image sample set, and d n The predicted probability that the category of the sample object in the first modality image sample output by the category detection module is the target category, y n is a binary control factor. When the sample label represents the category of the sample object in the first modality image sample and the target category is consistent, then y n is 1, when the sample label represents that the category of the sample object in the first modality image sample is inconsistent with the target category, then y n is 0.

[0118] Step S860, training the category detection model to be trained according to the modality conversion loss value and the category detection loss value until a target category detection model that meets the requirements is obtained.

[0119] The category detection device trains the category detection model to be trained according to the modal conversion loss value and the category detection loss value to obtain the target category detection model. As an example, the category detection device trains the category detection model to be trained according to the potential feature contrast loss value and the category detection loss value in the modal conversion loss value until the target category detection model that meets the requirements is obtained. As another example, the category detection device trains the category detection model to be trained according to the pixel-level loss value and the category detection loss value in the modal conversion loss value until the target category detection model that meets the requirements is obtained. As yet another example, the category detection device trains the category detection model to be trained according to the pixel-level loss value and the potential feature contrast loss value and the category detection loss value in the modal conversion loss value until the target category detection model that meets the requirements is obtained.

[0120] For a method of training a to-be-trained category detection model according to the pixel-level loss value, the potential feature contrast loss value, and the category detection loss value in the modal conversion loss value until a target category detection model that meets the requirements is obtained, as an example, the category retrieval device may calculate the sum of the losses among the pixel-level loss value, the potential feature contrast loss value, and the category detection loss value, and train the to-be-trained category detection model with the goal of reducing the sum of the losses among the pixel-level loss value, the potential feature contrast loss value, and the category detection loss value, until the determined sum of the losses between the modal conversion loss value and the category detection loss value is less than a threshold value, thereby obtaining the target category detection model. As another example, the category retrieval device may perform weighted summation processing according to the pixel-level loss value, the potential feature contrast loss value, and the category detection loss value to obtain a weighted loss value, and train the to-be-trained category detection model with the goal of reducing the weighted loss value, until the determined weighted loss value is less than a threshold value, thereby obtaining the target category detection model.

[0121] Specifically, the weighted loss value based on the pixel-level loss value, the potential feature comparison loss value, and the category detection loss value satisfies the following formula:

[0122] L=λ1L pixel +λ2L latent +λ3L bce

[0123] Among them, λ1, λ2, and λ3 are preset loss weight adjustment factors.

[0124] It can be seen that the multimodal object category detection method of the embodiment of the present application obtains a first modality image sample set to be trained, the first modality image sample set to be trained includes the first modality image samples to be trained and the sample labels of each first modality image sample, the sample label represents whether the category of the sample object in the first modality image sample is the target category; the first modality image samples in the first modality image sample set to be trained are input into the modality conversion module of the category detection model to be trained, and the second modality image sample data output by the modality conversion module to be trained is obtained, and the second modality image sample data includes the second modality image samples; the modality conversion loss value is determined according to the sample label of each first modality image sample and the corresponding second modality image sample data; the second modality image samples and the filtered second modality image samples are input into the category detection module of the category detection model to be trained, and the predicted probability that the category of the sample object in the first modality image sample output by the category detection module is the target category is obtained; the category detection loss value is determined according to the sample label and the predicted probability of each first modality image sample; the category detection model to be trained is trained according to the modality conversion loss value and / or the category detection loss value to obtain the target category detection model. On the one hand, using the first modality image samples and the second modality image samples with large differences to train the category detection model to be trained can improve the discrimination of the target category detection model for the first modality image samples and the second modality image samples, and enhance the detection generalization of the target category detection model under different modality images; on the other hand, using the filtered second modality image samples to train the category detection model to be trained can filter out the illumination information of the image and avoid the influence of illumination information on category detection.

[0125] As another example, if a target category detection model is used for category detection, and the target category detection model only includes a modal conversion module, that is, in an embodiment of the present application, the modal conversion module of the target category detection model is used to perform modal conversion processing on the first modal image, and category detection processing is performed on the first modal image and the filtered second modal image based on other methods other than the target category detection model. The training process of the modal conversion module in the target category detection model in the embodiment of the present application can refer to steps S810 to S830 in the above embodiment, and then the modal conversion loss value determined in step S830 becomes smaller as the target to train the modal conversion module until a modal conversion module that meets the requirements is obtained, that is, the modal conversion module in the target category detection model.

[0126] As another example, if a target category detection model is used for category detection, and the target category detection model only includes a category detection module, that is, in the embodiment of the present application, other methods other than the target category detection model are used to perform modal conversion processing on the first modal image, and the category detection module in the target category detection model is used to perform category detection processing on the first modal image and the filtered second modal image. The training process of the category detection module in the target category detection model in the embodiment of the present application can refer to steps S810, S840 to S850 in the above embodiment, and then the category detection loss value determined in step S850 is reduced as the target training category detection module until a category detection module that meets the requirements is obtained, that is, the category detection module in the target category detection model.

[0127] Fig.10 FIG. 1 is a block diagram of a category detection device shown in an exemplary embodiment of the present application. Fig.10 As shown, the exemplary category detection device 1000 includes: an acquisition module 1010, a modality conversion module 1020, a filtering module 1030 and a category detection module 1040. Specifically:

[0128] The acquisition module 1010 is used to acquire a first modality image, where the first modality image includes an object to be detected.

[0129] The modality conversion module 1020 is configured to perform modality conversion processing on the first modality image to obtain a second modality image.

[0130] The filtering module 1030 is used to filter the illumination information in the second modality image to obtain a filtered second modality image.

[0131] The category detection module 1040 is used to perform category detection processing based on the first modality image and the filtered second modality image to obtain the target category of the object to be detected.

[0132] In this exemplary category detection device, the acquired first modality image is subjected to modality conversion processing to obtain a second modality image; and the illumination information in the second modality image is filtered to obtain a filtered second modality image; then, category detection processing is performed based on the first modality image and the filtered second modality image to obtain the target category of the object to be detected. On the one hand, object category detection can be improved by performing object category detection using multi-modality images; on the other hand, by filtering the illumination information in the second modality image, the influence of illumination information on object category detection can be suppressed, further improving the accuracy of object category detection.

[0133] The functions of each module can be found in the embodiment of the multimodal object category detection method, which will not be described in detail here.

[0134] In order to implement the multimodal object category detection method of the above embodiment, the present application proposes another electronic device, which is specifically referred to as Fig.11 , Fig.11 It is a structural schematic diagram of an embodiment of an electronic device provided by the present application.

[0135] The electronic device 1100 includes a memory 1101 and a processor 1102 , wherein the memory 1101 and the processor 1102 are coupled.

[0136] The memory 1101 is used to store program data, and the processor 1102 is used to execute the program data to implement the multi-modal object category detection method of the above embodiment.

[0137] In this embodiment, the processor 1102 may also be referred to as a CPU (Central Processing Unit). The processor 1102 may be an integrated circuit chip having signal processing capabilities. The processor 1102 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. A general-purpose processor may be a microprocessor or the processor 1102 may also be any conventional processor, etc.

[0138] The present application also provides a computer-readable storage medium, such as Fig.12 As shown, the computer-readable storage medium 1200 is used to store program data 1201. When the program data 1201 is executed by the processor, it is used to implement the multi-modal object category detection method in the method embodiment of the present application.

[0139] The method involved in the embodiment of the object category detection method based on multimodality of the present application, when implemented in the form of a software functional unit and sold or used as an independent product, can be stored in a device, such as a computer-readable storage medium. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk.

[0140] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A multimodal object category detection method, characterized in that: The method comprises: Acquire a first modality image, where the first modality image includes an object to be detected; Performing modality conversion processing on the first modality image to obtain a second modality image; filtering the illumination information in the second modality image to obtain a filtered second modality image; A category detection process is performed based on the first modality image and the filtered second modality image to obtain a target category of the object to be detected.

2. The multimodal object category detection method according to claim 1, characterized in that: The step of performing modality conversion processing on the first modality image to obtain the second modality image includes: inputting the first modality image into an object category detection model to obtain a second modality image output by a modality conversion module of the object category detection model, and / or; The step of performing category detection processing based on the first modality image and the filtered second modality image to obtain the target category of the object to be detected includes: The first modality image and the filtered second modality image are input into the target category detection model to obtain the target category of the object to be detected determined by the category detection module of the target category detection model.

3. The multimodal object category detection method according to claim 2, characterized in that: The step of inputting the first modality image into the target category detection model to obtain the second modality image output by the modality conversion module of the target category detection model includes: Performing feature extraction processing on the first modality image by the modality conversion module to obtain latent spatial features of each pixel in the first modality image; Performing modality conversion processing according to the latent spatial features of each pixel point in the first modality image to obtain the second modality image, wherein the second modality image includes the latent spatial features of each pixel point; Upsampling is performed on the latent spatial features of each pixel in the second modality image to obtain pixel spatial features of each pixel in the second modality image.

4. The multimodal object category detection method according to claim 2, characterized in that: The step of inputting the first modality image and the filtered second modality image into the target category detection model to obtain the target category of the object to be detected determined by the category detection module of the target category detection model comprises: Performing modality fusion processing on the first modality image and the filtered second modality image through a category detection module of the target category detection model to obtain a modality fusion image; Determining the detection result output by the category detection module according to the modal fusion feature of the modal fusion image; In response to the detection result indicating that the probability that the category of the object to be detected belongs to the target category is greater than a preset probability threshold, it is determined that the category of the object to be detected is the target category.

5. The multimodal object category detection method according to claim 1, characterized in that: The step of filtering the illumination information in the second modality image to obtain a filtered second modality image includes: Performing filtering processing on illumination information of each pixel point in the second modal image in the first direction to obtain first filtering data of each pixel point in the first direction; Performing filtering processing on illumination information of each pixel point in the second modal image in the second direction to obtain second filtered data of each pixel point in the second direction; A pixel convolution operation is performed according to the first filtered data and the second filtered data of each pixel point in the second modality image to obtain the filtered second modality image.

6. The multimodal object category detection method according to claim 5, characterized in that: The step of filtering the illumination information of each pixel in the second modal image in the first direction to obtain first filtered data of each pixel in the first direction comprises: Obtaining the horizontal coordinate and the vertical coordinate of each pixel point in the second modality image; In response to the sum of the squares of the horizontal coordinate and the vertical coordinate of the pixel point being greater than a preset threshold, obtaining an arc tangent value of the coordinate ratio between the vertical coordinate and the horizontal coordinate of the pixel point; The ratio between the cosine value of the arctangent value of the coordinate ratio and the square sum is determined as the first filtering data of the pixel point in the first direction.

7. The multimodal object category detection method according to claim 2, characterized in that: The method further comprises: Acquire a first modality image sample set to be trained, wherein the first modality image sample set to be trained includes first modality image samples to be trained and sample labels of each first modality image sample, wherein the sample label represents a sample category of a sample object in the first modality image sample; Inputting the first modality image samples in the first modality image sample set to be trained into the modality conversion module of the category detection model to be trained, and obtaining the second modality image sample data output by the modality conversion module of the category detection model to be trained, wherein the second modality image sample data includes the second modality image samples; Determining a modality conversion loss value according to a sample label of each first modality image sample and corresponding second modality image sample data; Inputting the second modality image sample and the filtered second modality image sample into the category detection module of the category detection model to be trained, and obtaining a predicted probability that the category of the sample object in the first modality image sample output by the category detection module of the category detection model to be trained is the target category; Determining a category detection loss value according to the sample label of each first modality image sample and the predicted probability; The category detection model to be trained is trained according to the modal conversion loss value and / or the category detection loss value until the target category detection model that meets the requirements is obtained.

8. The multimodal object category detection method according to claim 7, characterized in that: The second modality image sample data also includes feature embedding of the first modality image samples, and the step of determining the modality conversion loss value according to the sample label of each first modality image sample and the corresponding second modality image sample data includes: Determining a potential feature contrast loss value according to feature embedding of each first modality image sample and the number of sample labels of the target category representing the category of the sample object of the corresponding image sample in the first modality image sample set; Determining a pixel-level loss value according to the second modality image sample, the filtered second modality image sample, and the number of sample labels; The modality conversion loss value is determined based on the potential feature contrast loss value and / or the pixel level loss value.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the memory stores program instructions, and the processor retrieves the program instructions from the memory to execute the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: include: Program data is stored, and when the program data is executed by a processor, it is used to implement the method according to any one of claims 1 to 8.

Images