Multi-modal image detection method, object detection network training method and related equipment

Abstract

The invention provides a multi-modal image detection method, an object detection network training method and related equipment. In the embodiment of the invention, an original image is processed through a feature extraction model to obtain a first image feature; processing the original image through a first potential encoder to obtain a second image feature; the first feature image is processed through the feature extractor to obtain a third image feature, and the first feature image can be an original image, a depth image or a heat image, that is, the first feature image is a multi-modal image. Therefore, the salient object detection of the multi-modal image can be realized, and the accuracy of the image detection result is improved.

Description

Technical Field

[0001] The present application relates to the field of image detection technology, and in particular to a multimodal image detection method, an object detection network training method, and related equipment. Background Art

[0002] In the field of computer vision, salient object detection (SOD) is an important research direction, which aims to identify and segment the most important or most prominent objects in an image. In the prior art, salient object detection is generally performed on an image in a single-modal manner, that is, object detection is performed based on the original image, or depth image, or thermal image, and then the object in the image is determined based on the detection result.

[0003] However, the above-mentioned image detection method can only predict single-modal images, and the definition of object edges is not clear enough, which reduces the accuracy of image detection results. Summary of the invention

[0004] The main purpose of this application is to provide a multimodal image detection method, an object detection network training method and related equipment, aiming to solve the technical problem of low accuracy of image detection results.

[0005] To achieve the above objectives, the present application provides a multimodal image detection method, comprising: Processing the original image through a feature extraction model to obtain a first image feature; Processing the original image through a first latent encoder to obtain a second image feature; Processing the first feature image by a feature extractor to obtain a third image feature; the first feature image includes at least one of an original image, a first depth image, and a first thermal image, the first depth image is determined based on the depth information of the original image, and the first thermal image is determined based on the thermal information of the original image; The first image feature, the second image feature and the third image feature are input into a trained object detection network to generate a target image; the target image is used to characterize a salient object in the original image.

[0006] To achieve the above objectives, the present application also provides a method for training an object detection network, comprising: Acquire a training set; the training set includes a training image, an annotated image, and a second feature image, the second feature image includes at least one of a training image, a second depth image, and a second thermal image, the second depth image is determined based on depth information of the training image, the second thermal image is determined based on thermal information of the training image, and the annotated image is used to characterize a salient object in the training image; Processing the training image through the feature extraction model and the first latent encoder to obtain a fourth image feature and a fifth image feature respectively; Processing the second feature image by a feature extractor to obtain a sixth image feature; Processing the annotated image through a second latent encoder to obtain a seventh image feature; Inputting the fourth image feature, the fifth image feature, the sixth image feature, and the seventh image feature into a preset object detection network to generate a predicted image; Iteratively training the object detection network based on a loss value between the predicted image and the labeled image; When the loss value between the predicted image and the labeled image is less than or equal to a preset value, a trained object detection network is obtained.

[0007] Optionally, the training image is processed by a feature extraction model to obtain a fourth image feature, including: Extracting image features of the training image by a feature extraction model; The image features of the training image are processed through a multi-scale feature fusion convolution layer to obtain the fourth image features.

[0008] Optionally, the training image is processed by the first latent encoder to obtain a fifth image feature, including: Processing the training image by the first latent encoder to obtain a first encoded feature map; Processing the first encoding feature map through a first convolutional layer to obtain a fifth image feature; The step of processing the labeled image by the second latent encoder to obtain a seventh image feature includes: Processing the annotated image by the second latent encoder to obtain a second encoded feature map; The second encoding feature map is processed by a second convolutional layer and a diffusion model to obtain a seventh image feature.

[0009] Optionally, inputting the fourth image feature, the fifth image feature, the sixth image feature, and the seventh image feature into a preset object detection network to generate a predicted image includes: Performing feature fusion on the fifth image feature and the seventh image feature to obtain a fused image feature, and inputting the fused image feature into the object detection network and the preset first encoding module respectively; Inputting the fourth image feature into a preset intermediate module; inputting the sixth image feature into the object detection network; The output of the object detection network, the output of the first encoding module and the output of the intermediate module are decoded by a decoding module and a decoder to obtain a predicted image.

[0010] Optionally, inputting the sixth image feature into the object detection network comprises: Convolving the sixth image feature through a zero convolutional layer in the object detection network; Inputting the convolution-processed sixth image feature into a feature adaptation network in the object detection network; Inputting the fused image features into the object detection network comprises: The fused image features are input into a feature adaptation network in the object detection network.

[0011] Optionally, before decoding the output of the object detection network, the output of the encoding module, and the output of the intermediate module by a decoding module and a decoder to obtain a predicted image, the method further includes: Inputting the image features output by the feature adaptation network into a second encoding module in the object detection network; Inputting the image features output by the second encoding module into the zero convolution layer; A merging and splicing operation is performed on the image features output by the zero convolution layer and the image features output by the first encoding module to obtain an output of the object detection network.

[0012] In addition, to achieve the above objectives, the present application also provides a multimodal image detection device, comprising: A feature extraction model is used to process the original image to obtain a first image feature; A first latent encoder is used to process the original image to obtain a second image feature; a feature extractor, configured to process a first feature image to obtain a third image feature; the first feature image includes at least one of an original image, a first depth image, and a first thermal image, the first depth image being determined based on depth information of the original image, and the first thermal image being determined based on thermal information of the original image; The object detection network is used to generate a target image according to the first image feature, the second image feature and the third image feature; the target image is used to characterize a salient object in the original image.

[0013] In addition, to achieve the above objectives, the present application also provides a training device for an object detection network, comprising: an acquisition module, configured to acquire a training set; the training set includes a training image, an annotated image, and a second feature image, the second feature image includes at least one of a training image, a second depth image, and a second thermal image, the second depth image is determined based on the depth information of the training image, the second thermal image is determined based on the thermal information of the training image, and the annotated image is used to characterize a salient object in the training image; A feature extraction model, used for processing the training image to obtain a fourth image feature; A first latent encoder is used to process the training image to obtain a fifth image feature; A feature extractor, used for processing the second feature image to obtain a sixth image feature; a second latent encoder, configured to process the annotated image to obtain a seventh image feature; a training module, configured to input the fourth image feature, the fifth image feature, the sixth image feature, and the seventh image feature into a preset object detection network to generate a predicted image; Iteratively training the object detection network based on a loss value between the predicted image and the labeled image; When the loss value between the predicted image and the labeled image is less than or equal to a preset value, a trained object detection network is obtained.

[0014] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution: The computer device includes a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps of the multimodal image detection method described in any one of the embodiments of the present application are implemented, or the steps of the training method of the object detection network described in any one of the embodiments of the present application are implemented.

[0015] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution: The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, it implements the steps of the multimodal image detection method described in any one of the embodiments of the present application, or implements the steps of the object detection network training method described in any one of the embodiments of the present application.

[0016] Compared with the prior art, the embodiments of the present application have the following beneficial effects: The present application provides a multimodal image detection method, an object detection network training method and related equipment, the method comprising: processing the original image through a feature extraction model to obtain a first image feature; processing the original image through a first latent encoder to obtain a second image feature; processing the first feature image through a feature extractor to obtain a third image feature; the first feature image includes at least one of the original image, the first depth image and the first heat image, the first depth image is determined based on the depth information of the original image, and the first heat image is determined based on the heat information of the original image; the first image feature, the second image feature and the third image feature are input into the trained object detection network to generate a target image; the target image is used to characterize the salient object in the original image. In the embodiment of the present application, the original image is processed through a feature extraction model to obtain a first image feature; the original image is processed through a first latent encoder to obtain a second image feature; the first feature image is processed through a feature extractor to obtain a third image feature, wherein the first feature image can be an original image, a depth image or a heat image, that is, the first feature image is a multimodal image. In this way, salient object detection of multimodal images can be achieved, thereby improving the accuracy of image detection results. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 is an exemplary system architecture diagram to which the present application may be applied; Figure 2 is a flow chart of a multimodal image detection method provided in an embodiment of the present application; Figure 3 is a flow chart of a method for training an object detection network provided in an embodiment of the present application; Figure 4 is a structural schematic diagram of an embodiment of a multimodal image detection device provided in an embodiment of the present application; Figure 5It is a structural diagram of an embodiment of a training device for an object detection network provided in an embodiment of the present application; Figure 6 It is a basic structural block diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] The multimodal image detection method provided in the embodiment of the present application is applied to a multimodal image detection device, and the training method of the object detection network provided in the embodiment of the present application is applied to the training device of the object detection network. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those generally understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" in the specification and claims of the present application and the above-mentioned description of the drawings and any variations thereof are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0020] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0021] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0022] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0023] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social online platform software, etc.

[0024] Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV), laptop computers, desktop computers, etc.

[0025] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0026] It should be noted that the multimodal image detection method and the object detection network training method provided in the embodiments of the present application are generally executed by a server / terminal device. Accordingly, the multimodal image detection device and the object detection network training device are generally arranged in the server / terminal device.

[0027] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0028] Please refer to Figure 2 , shows a flow chart of an embodiment of the multimodal image detection method proposed in the present application. The embodiment of the present application can acquire and process relevant data based on artificial intelligence technology. It should be understood that the multimodal image detection method proposed in the present application can be applied to the image detection model.

[0029] The multimodal image detection method proposed in this application includes: S210, processing the original image through a feature extraction model to obtain a first image feature.

[0030] Optionally, the feature extraction model is a ViT (Vision Transformer) model. It should be understood that the original image may be an RGB image.

[0031] It should be understood that the above-mentioned image detection model includes a feature extraction model. In this step, the image detection model obtains the original image and processes the original image through the feature extraction model to obtain the first image feature.

[0032] S220: Process the original image through a first latent encoder to obtain a second image feature.

[0033] It should be understood that the above-mentioned image detection model includes a first latent encoder.

[0034] In this step, the image detection model obtains the original image and processes the original image through the first latent encoder to obtain the second image feature.

[0035] S230, processing the first feature image by a feature extractor to obtain a third image feature.

[0036] It should be understood that the above-mentioned image detection model includes a feature extractor. The above-mentioned first feature image includes at least one of the original image, the first depth image and the first heat image, the first depth image is determined based on the depth information of the original image, and the first heat image is determined based on the heat information of the original image; the depth image can be understood as the depth map corresponding to the original image, and the heat map can be understood as the heat map corresponding to the original image.

[0037] In this step, the feature extractor may process the first feature image to obtain a third image feature.

[0038] S240: Input the first image feature, the second image feature, and the third image feature into a trained object detection network to generate a target image.

[0039] It should be understood that the above-mentioned image detection model includes an object detection network. Optionally, the above-mentioned object detection network is a SOD control network (ControlNet).

[0040] In this step, after obtaining the first image feature, the second image feature and the third image feature, the first image feature, the second image feature and the third image feature are input into the trained object detection network to generate a target image, wherein the target image is used to characterize the salient object in the original image.

[0041] In the embodiment of the present application, the original image is processed by the feature extraction model to obtain the first image feature; the original image is processed by the first latent encoder to obtain the second image feature; the first feature image is processed by the feature extractor to obtain the third image feature, wherein the first feature image can be the original image, the depth image or the thermal image, that is, the first feature image is a multimodal image. In addition, the salient object detection of the multimodal image can be realized, thereby improving the accuracy of the image detection result.

[0042] Please refer to Figure 3 , shows a flow chart of an embodiment of a training method for an object detection network proposed in the present application. The present application embodiment can acquire and process relevant data based on artificial intelligence technology.

[0043] The training method of the object detection network proposed in this application includes: S310, obtaining a training set.

[0044] It should be noted that the above-mentioned training set includes training images, annotated images and second feature images. The second feature images include at least one of the training images, second depth images and second thermal images. The second depth image is determined based on the depth information of the training images, and the second thermal image is determined based on the thermal information of the training images. The annotated images are the original images after manual annotation, which are used to characterize the salient objects in the training images.

[0045] S320, processing the training image through the feature extraction model and the first latent encoder to obtain a fourth image feature and a fifth image feature respectively.

[0046] As described above, the image detection model includes a feature extraction model and a first latent encoder.

[0047] In this step, the training image is processed by the feature extraction model to obtain the fourth image feature; and the training image is processed by the first latent encoder to obtain the fifth image feature.

[0048] S330, processing the second feature image by a feature extractor to obtain a sixth image feature.

[0049] In this step, the second feature image may be processed by a feature extractor to obtain a sixth image feature, that is, the feature extractor processes at least one of the second feature image, the second thermal image, and the second depth image to obtain the sixth image feature.

[0050] S340: Process the labeled image through a second latent encoder to obtain a seventh image feature.

[0051] It should be understood that the image detection model also includes a second latent encoder.

[0052] In this step, the annotated image is processed by the second latent encoder to obtain the seventh image feature.

[0053] S350: Input the fourth image feature, the fifth image feature, the sixth image feature, and the seventh image feature into a preset object detection network to generate a predicted image.

[0054] In this step, after obtaining the fourth image feature, the fifth image feature, the sixth image feature, and the seventh image feature, the fourth image feature, the fifth image feature, the sixth image feature, and the seventh image feature are input into the object detection network to obtain a predicted image.

[0055] S360: Iteratively train the object detection network based on the loss value between the predicted image and the labeled image.

[0056] S370, when the loss value between the predicted image and the labeled image is less than or equal to a preset value, obtaining a trained object detection network.

[0057] Furthermore, a loss value between the predicted image and the annotated image is calculated. The loss value is used to characterize the difference between the predicted image and the annotated image. When the loss value is less than or equal to a preset value, it indicates that the object detection network training is completed, and a trained object detection network is obtained.

[0058] In the embodiment of the present application, in the training of the object detection network, the second feature image is processed by the feature extractor, wherein the second feature image can be a training image, a depth image or a thermal image, that is, the second feature image is a multimodal image. Then, the multimodal image is introduced as a training set in the training of the object detection network to improve the accuracy of the image detection result.

[0059] Optionally, the training image is processed by a feature extraction model to obtain a fourth image feature, including: Extracting image features of the training image by a feature extraction model; The image features of the training image are processed through a multi-scale feature fusion convolution layer to obtain the fourth image features.

[0060] It should be noted that the image detection model also includes a multi-scale feature fusion convolution layer (Mutil-Scale Feature Fusion).

[0061] In this embodiment, the image features of the training image are extracted by the feature extraction model, and then the image features of the training image are processed by the scale feature fusion convolution layer to obtain the fourth image features. In this way, the features of the training image are retained during the training process, and the accuracy of the predicted image is improved.

[0062] Optionally, the training image is processed by the first latent encoder to obtain a fifth image feature, including: Processing the training image by the first latent encoder to obtain a first encoded feature map; Processing the first encoding feature map through a first convolutional layer to obtain a fifth image feature; The step of processing the labeled image by the second latent encoder to obtain a seventh image feature includes: Processing the annotated image by the second latent encoder to obtain a second encoded feature map; The second encoding feature map is processed by a second convolutional layer and a diffusion model to obtain a seventh image feature.

[0063] It should be noted that the image detection model also includes a first convolutional layer, a second convolutional layer and a diffusion model.

[0064] In this embodiment, the training image is processed by the first latent encoder to obtain a first encoding feature map; and then the first encoding feature map is processed by the first convolutional layer to obtain the fifth image feature.

[0065] The annotated image is processed by a second latent encoder to obtain a second encoded feature map; and the second encoded feature map is then processed by a second convolutional layer and a diffusion model to obtain a seventh image feature.

[0066] In this embodiment, the second encoded feature map is processed by a diffusion model to gradually convert the data into noise, and then an inverse process is learned to gradually restore the original data from the noise, ultimately generating high-quality image features.

[0067] Optionally, inputting the fourth image feature, the fifth image feature, the sixth image feature, and the seventh image feature into a preset object detection network to generate a predicted image includes: Performing feature fusion on the fifth image feature and the seventh image feature to obtain a fused image feature, and inputting the fused image feature into the object detection network and the preset first encoding module respectively; Inputting the fourth image feature into a preset intermediate module; inputting the sixth image feature into the object detection network; The output of the object detection network, the output of the first encoding module and the output of the intermediate module are decoded by a decoding module and a decoder to obtain a predicted image.

[0068] It should be noted that the image detection model also includes an intermediate module, a first encoding module, a decoding module and a decoder.

[0069] Optionally, the intermediate module is an 8*8 convolution layer. The first encoding module includes an 8*8 convolution layer, a 16*16 convolution layer, a 32*32 convolution layer, and a 64*64 convolution layer. The decoding module includes an 8*8 convolution layer, a 16*16 convolution layer, a 32*32 convolution layer, and a 64*64 convolution layer.

[0070] In this embodiment, after obtaining the fifth image feature and the seventh image feature, the fifth image feature and the seventh image feature are feature fused to obtain a fused image feature, and the fused image feature is respectively input into the object detection network and the preset first encoding module; the fourth image feature is input into the preset intermediate module; the sixth image feature is input into the object detection network; the output of the object detection network, the output of the first encoding module, and the output of the intermediate module are decoded by the decoding module and the decoder to obtain a predicted image.

[0071] Optionally, inputting the sixth image feature into the object detection network comprises: Convolving the sixth image feature through a zero convolutional layer in the object detection network; Inputting the convolution-processed sixth image feature into a feature adaptation network in the object detection network; Inputting the fused image features into the object detection network comprises: The fused image features are input into a feature adaptation network in the object detection network.

[0072] It should be noted that the object detection network also includes zero convolution layers and feature adaptation networks.

[0073] In this embodiment, the sixth image feature is convolved by the zero convolution layer in the object detection network, and the sixth image feature after the convolution is input into the feature adaptation network in the object detection network. At the same time, the fused image feature is input into the feature adaptation network in the object detection network.

[0074] In this embodiment, a feature adaptation network is set in the object detection network, and convolution processing is performed on the sixth image feature through the feature adaptation network, so that the output of the feature adaptation network meets the requirements of the object detection network, thereby avoiding the loss of image information and improving the accuracy of the predicted image.

[0075] Optionally, before decoding the output of the object detection network, the output of the encoding module, and the output of the intermediate module by a decoding module and a decoder to obtain a predicted image, the method further includes: Inputting the image features output by the feature adaptation network into a second encoding module in the object detection network; Inputting the image features output by the second encoding module into the zero convolution layer; A merging and splicing operation is performed on the image features output by the zero convolution layer and the image features output by the first encoding module to obtain an output of the object detection network.

[0076] It should be noted that the object detection network also includes a second encoding module and a zero convolution layer.

[0077] In this embodiment, the image features output by the feature adaptation network are input to the second encoding module, and then the image features output by the second encoding module are input to the zero convolution layer. The image features output by the zero convolution layer and the image features output by the first encoding module are merged and spliced ​​to obtain the output of the object detection network.

[0078] Furthermore, the output of the object detection network, the output of the first encoding module, and the output of the intermediate module are decoded by the decoding module and the decoder to obtain a predicted image.

[0079] Optionally, the second encoding module includes one 8*8 convolutional layer, one 16*16 convolutional layer, one 32*32 convolutional layer and one 64*64 convolutional layer.

[0080] See also Figure 4 , an embodiment of the present application provides a multimodal image detection device 400, the multimodal image detection device 400 comprising: A feature extraction model 410 is used to process the original image to obtain a first image feature; A first latent encoder 420, used to process the original image to obtain a second image feature; A feature extractor 430 is configured to process a first feature image to obtain a third image feature; the first feature image includes at least one of an original image, a first depth image, and a first thermal image, the first depth image is determined based on depth information of the original image, and the first thermal image is determined based on thermal information of the original image; The object detection network 440 is used to generate a target image according to the first image feature, the second image feature and the third image feature; the target image is used to represent a salient object in the original image.

[0081] See also Figure 5 , an object detection network training device 500 provided in an embodiment of the present application, the object detection network training device 500 comprising: An acquisition module 510 is used to acquire a training set; the training set includes a training image, an annotated image, and a second feature image, the second feature image includes at least one of a training image, a second depth image, and a second thermal image, the second depth image is determined based on the depth information of the training image, the second thermal image is determined based on the thermal information of the training image, and the annotated image is used to characterize a salient object in the training image; A feature extraction model 520, used to process the training image to obtain a fourth image feature; A first latent encoder 530, configured to process the training image to obtain a fifth image feature; A feature extractor 540, configured to process the second feature image to obtain a sixth image feature; A second latent encoder 550, configured to process the annotated image to obtain a seventh image feature; A training module 560, configured to input the fourth image feature, the fifth image feature, the sixth image feature, and the seventh image feature into a preset object detection network to generate a predicted image; Iteratively training the object detection network based on a loss value between the predicted image and the labeled image; When the loss value between the predicted image and the labeled image is less than or equal to a preset value, a trained object detection network is obtained.

[0082] Optionally, the feature extraction model 520 is specifically used for: Extracting image features of the training image; The image features of the training image are processed through a multi-scale feature fusion convolution layer to obtain the fourth image features.

[0083] Optionally, the first potential encoder 530 is specifically configured to: Processing the training image to obtain a first encoding feature map; Processing the first encoding feature map through a first convolutional layer to obtain a fifth image feature; The second potential encoder 550 is specifically configured to: Processing the annotated image to obtain a second encoding feature map; The second encoding feature map is processed by a second convolutional layer and a diffusion model to obtain a seventh image feature.

[0084] Optionally, the training module 560 is specifically used to: Performing feature fusion on the fifth image feature and the seventh image feature to obtain a fused image feature, and inputting the fused image feature into the object detection network and the preset first encoding module respectively; Inputting the fourth image feature into a preset intermediate module; inputting the sixth image feature into the object detection network; The output of the object detection network, the output of the first encoding module and the output of the intermediate module are decoded by a decoding module and a decoder to obtain a predicted image.

[0085] Optionally, the training module 560 is further specifically configured to: Convolving the sixth image feature through a zero convolutional layer in the object detection network; Inputting the convolution-processed sixth image feature into a feature adaptation network in the object detection network; The fused image features are input into a feature adaptation network in the object detection network.

[0086] Optionally, the training module 560 is further specifically configured to: Inputting the image features output by the feature adaptation network into a second encoding module in the object detection network; Inputting the image features output by the second encoding module into the zero convolution layer; A merging and splicing operation is performed on the image features output by the zero convolution layer and the image features output by the first encoding module to obtain an output of the object detection network.

[0087] To solve the above technical problems, the present application also provides a computer device. Figure 6 , Figure 6 This is a basic structural block diagram of the computer device in this embodiment. The computer device 6 includes a memory 61, a processor 62, and a network interface 63 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 6 with components 61-63, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0088] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0089] The memory 61 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 61 can be an internal storage unit of the computer device 6, such as a hard disk or memory of the computer device 6. In other embodiments, the memory 61 can also be an external storage device of the computer device 6, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the computer device 6. Of course, the memory 61 can also include both the internal storage unit of the computer device 6 and its external storage device. In this embodiment, the memory 61 is generally used to store the operating system and various application software installed on the computer device 6, such as the program code of the multimodal image detection method or the training method of the object detection network. In addition, the memory 61 can also be used to temporarily store various types of data that have been output or are to be output.

[0090] The processor 62 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 62 is generally used to control the overall operation of the computer device 6. In this embodiment, the processor 62 is used to run the program code stored in the memory 61 or process data, such as running the program code of the multimodal image detection method or the training method of the object detection network.

[0091] The network interface 63 may include a wireless network interface or a wired network interface. The network interface 63 is generally used to establish a communication connection between the computer device 6 and other electronic devices.

[0092] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores the multimodal image detection program, and the multimodal image detection program can be executed by at least one processor so that the at least one processor performs the steps of the multimodal image detection method as described above; the computer-readable storage medium stores a training program for the object detection network, and the training program for the object detection network can be executed by at least one processor so that the at least one processor performs the steps of the training method for the object detection network as described above.

[0093] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware online platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0094] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0095] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A multimodal image detection method, characterized in that: include: Processing the original image through a feature extraction model to obtain a first image feature; Processing the original image through a first latent encoder to obtain a second image feature; Processing the first feature image by a feature extractor to obtain a third image feature; the first feature image includes at least one of an original image, a first depth image, and a first thermal image, the first depth image is determined based on the depth information of the original image, and the first thermal image is determined based on the thermal information of the original image; The first image feature, the second image feature and the third image feature are input into a trained object detection network to generate a target image; the target image is used to characterize a salient object in the original image.

2. A method for training an object detection network, characterized in that: include: Get the training set; The training set includes a training image, an annotated image, and a second feature image, the second feature image includes at least one of a training image, a second depth image, and a second thermal image, the second depth image is determined based on depth information of the training image, the second thermal image is determined based on thermal information of the training image, and the annotated image is used to characterize a salient object in the training image; Processing the training image through the feature extraction model and the first latent encoder to obtain a fourth image feature and a fifth image feature respectively; Processing the second feature image by a feature extractor to obtain a sixth image feature; Processing the annotated image through a second latent encoder to obtain a seventh image feature; Inputting the fourth image feature, the fifth image feature, the sixth image feature, and the seventh image feature into a preset object detection network to generate a predicted image; Iteratively training the object detection network based on a loss value between the predicted image and the labeled image; When the loss value between the predicted image and the labeled image is less than or equal to a preset value, a trained object detection network is obtained.

3. The method according to claim 2, characterized in that The training image is processed by the feature extraction model to obtain the fourth image feature, including: Extracting image features of the training image by a feature extraction model; The image features of the training image are processed through a multi-scale feature fusion convolution layer to obtain the fourth image features.

4. The method according to claim 2, characterized in that: The training image is processed by the first latent encoder to obtain a fifth image feature, including: Processing the training image by the first latent encoder to obtain a first encoded feature map; Processing the first encoding feature map through a first convolutional layer to obtain a fifth image feature; The step of processing the labeled image by the second latent encoder to obtain a seventh image feature includes: Processing the annotated image by the second latent encoder to obtain a second encoded feature map; The second encoding feature map is processed by a second convolutional layer and a diffusion model to obtain a seventh image feature.

5. The method according to claim 2, characterized in that: The step of inputting the fourth image feature, the fifth image feature, the sixth image feature, and the seventh image feature into a preset object detection network to generate a predicted image includes: Performing feature fusion on the fifth image feature and the seventh image feature to obtain a fused image feature, and inputting the fused image feature into the object detection network and the preset first encoding module respectively; Inputting the fourth image feature into a preset intermediate module; inputting the sixth image feature into the object detection network; The output of the object detection network, the output of the first encoding module and the output of the intermediate module are decoded by a decoding module and a decoder to obtain a predicted image.

6. The method according to claim 5, characterized in that The inputting the sixth image feature into the object detection network comprises: Convolving the sixth image feature with a zero convolutional layer in the object detection network; Inputting the convolution-processed sixth image feature into a feature adaptation network in the object detection network; Inputting the fused image features into the object detection network comprises: The fused image features are input into a feature adaptation network in the object detection network.

7. The method according to claim 6, characterized in that Before decoding the output of the object detection network, the output of the encoding module, and the output of the intermediate module by the decoding module and the decoder to obtain the predicted image, the method further includes: Inputting the image features output by the feature adaptation network into a second encoding module in the object detection network; Inputting the image features output by the second encoding module into the zero convolution layer; A merging and splicing operation is performed on the image features output by the zero convolution layer and the image features output by the first encoding module to obtain an output of the object detection network.

8. A multimodal image detection device, characterized in that: include: A feature extraction model is used to process the original image to obtain a first image feature; A first latent encoder is used to process the original image to obtain a second image feature; a feature extractor, configured to process a first feature image to obtain a third image feature; the first feature image includes at least one of an original image, a first depth image, and a first thermal image, the first depth image being determined based on depth information of the original image, and the first thermal image being determined based on thermal information of the original image; The object detection network is used to generate a target image according to the first image feature, the second image feature and the third image feature; the target image is used to characterize a salient object in the original image.

9. A training device for an object detection network, characterized in that: include: an acquisition module, configured to acquire a training set; the training set includes a training image, an annotated image, and a second feature image, the second feature image includes at least one of a training image, a second depth image, and a second thermal image, the second depth image is determined based on the depth information of the training image, the second thermal image is determined based on the thermal information of the training image, and the annotated image is used to characterize a salient object in the training image; A feature extraction model, used for processing the training image to obtain a fourth image feature; A first latent encoder is used to process the training image to obtain a fifth image feature; A feature extractor, used for processing the second feature image to obtain a sixth image feature; a second latent encoder, configured to process the annotated image to obtain a seventh image feature; a training module, configured to input the fourth image feature, the fifth image feature, the sixth image feature, and the seventh image feature into a preset object detection network to generate a predicted image; Iteratively training the object detection network based on a loss value between the predicted image and the labeled image; When the loss value between the predicted image and the labeled image is less than or equal to a preset value, a trained object detection network is obtained.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the steps of the multimodal image detection method according to claim 1, or the steps of the object detection network training method according to any one of claims 2 to 7.

Images