A deep learning-based multi-source heterogeneous data fusion method

Abstract

The embodiment of the application discloses a multi-source heterogeneous data fusion method based on deep learning, which can be applied to the field of image processing. The method comprises the following steps: acquiring scene information of a text type and an image of a picture type, wherein the scene information at least comprises first-level object information and second-level object information; determining a model cluster according to the first-level object information; determining a target model corresponding to each target sub-object in the model cluster according to the second-level object information; performing coarse segmentation on the image based on each target model to obtain each first image; performing segmentation on each first image based on information aggregation associated with the corresponding target sub-object to obtain a fine segmentation result corresponding to each first image; and rendering a main image corresponding to the image according to the fine segmentation result, wherein the main image represents the fusion result of the scene information and the image. The application can realize multi-source data fusion.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method for fusing multi-source heterogeneous data based on deep learning. Background Technology

[0002] Multi-source data may include information pointing to the same object. In other words, information about the same object can be recorded in different forms in different types of multi-source data. Different data express different aspects of the same object. How to fuse information from multi-source data to express the same object in a multi-dimensional and three-dimensional way, or to fully integrate and extract information about the same object from multi-source data centers and apply the fusion results to other applications, has always been an important research topic in the field of image processing. Summary of the Invention

[0003] To address at least one of the aforementioned technical problems, embodiments of this application provide a method for fusing multi-source heterogeneous data based on deep learning.

[0004] On the one hand, embodiments of this application provide a method for multi-source heterogeneous data fusion based on deep learning, the method comprising:

[0005] Acquire scene information of text type and image of image type. The scene information is used to describe the scene of the image. The scene information includes at least first-level object information and second-level object information. The first-level object information represents the scene, and the second-level object information represents the target subordinate objects associated in the scene.

[0006] A model cluster is determined based on the first-level object information, and the model cluster includes models for segmentation corresponding to each subordinate object in the scene represented by the first-level object information.

[0007] Based on the secondary object information, determine the target model corresponding to each target sub-object in the model cluster;

[0008] Based on each of the target models, the image is coarsely segmented to obtain the first image corresponding to each of the target sub-objects;

[0009] Each of the first images is segmented based on the information set associated with its corresponding target sub-object, to obtain a fine segmentation result for each of the first images;

[0010] The subject image corresponding to the image is rendered based on the fine segmentation result, and the subject image represents the fusion result of the scene information and the image.

[0011] In one embodiment, the fine segmentation result includes a mask matrix, and the segmentation of each first image based on the information set associated with its corresponding target sub-object to obtain the fine segmentation result corresponding to each first image includes:

[0012] For each first image, the first image is input into the corresponding clustering information extractor. The clustering information extractor is trained based on the detector corresponding to the target sub-object of the first image. The clustering information extractor is used to extract clustering information for the target sub-object in the first image to obtain the fusion feature information corresponding to the first image.

[0013] The fused feature information is input into the segmenter corresponding to the target sub-image of the first image to obtain the mask matrix.

[0014] In one embodiment, rendering the subject image corresponding to the image based on the fine segmentation result, wherein the subject image represents the fusion result of the scene information and the image, includes:

[0015] The main image is rendered based on the mask matrix corresponding to each first image and the image itself.

[0016] In one embodiment, the clustering information extractor mainly includes an extraction network and a mask generation network. The mask generation network is used to generate a target mask, which is used to distinguish between lower-level objects and non-lower-level objects in an image input to the clustering information extractor. The extraction network and the mask generation network are trained using the following method:

[0017] A sample image and a detector corresponding to the type of the subordinate object in the sample image are obtained. The sample image carries annotation information, which represents the position information of the subordinate object in the sample image. A sample image includes only a single type of subordinate object.

[0018] The sample image is input into the extraction network, and the extraction network performs the following operations: for each pixel region in the sample image, the sample first feature corresponding to the pixel region is extracted, the associated position feature corresponding to each associated pixel position is extracted, and the associated position features are fused to obtain the sample second feature corresponding to the pixel region, wherein the associated pixel position is other pixel position in the neighborhood of the pixel region that does not belong to the pixel region;

[0019] The first and second features of the sample corresponding to each pixel region are input into the mask generation network to obtain the sample target mask;

[0020] For each pixel region, the first features of other samples excluding the corresponding first features of the sample are input into the detector to obtain the first detection result corresponding to the pixel region, and the second features of other samples excluding the corresponding second features of the sample are input into the detector to obtain the second detection result corresponding to the pixel region.

[0021] Based on each of the first detection results, each of the second detection results, and the annotation information, an indication target mask is obtained;

[0022] Based on the difference between the sample target mask and the indicator target mask, adjust the parameters of the extraction network and the mask generation network until training is complete.

[0023] In one embodiment, obtaining the target indication mask based on each of the first detection results, each of the second detection results, and the annotation information includes:

[0024] For each pixel region, if the difference between the first detection result and the annotation information is greater than a first threshold, and the difference between the second detection result and the annotation information is greater than a second threshold, then the pixel region is determined to belong to the core pixel region.

[0025] If the difference between the first detection result and the annotation information is greater than a first threshold, and the difference between the second detection result and the annotation information is less than or equal to a second threshold, the pixel region is determined to belong to a boundary pixel region.

[0026] The target indication mask is generated based on the core pixel region and the boundary pixel region.

[0027] In one embodiment, the method further includes:

[0028] For each pixel region, the difference between its corresponding first feature and its corresponding second feature is calculated to obtain the difference information.

[0029] If the difference between the first detection result and the annotation information is greater than a first threshold, and the difference between the second detection result and the annotation information is less than or equal to a second threshold, determining that the pixel region belongs to a boundary pixel region includes:

[0030] If the difference between the first detection result and the annotation information is greater than a first threshold, and the difference between the second detection result and the annotation information is less than or equal to a second threshold, and the difference information is greater than a third threshold, then the pixel region is determined to belong to a boundary pixel region.

[0031] In one embodiment, the clustering information extractor further includes a clustering network, and after the first image is input into the corresponding clustering information extractor, the clustering information extractor performs the following operations:

[0032] The first image is input into the extraction network, which triggers the mask generation network to generate a target mask corresponding to the first image.

[0033] Based on the clustering network, the depth features of the part of the first image covered by the target mask are extracted to obtain the first target features;

[0034] Based on the clustering network, multi-scale feature extraction is performed on the portion of the first image not covered by the target mask to obtain the second target features;

[0035] The first target feature and the second target feature are fused to obtain the fused feature information.

[0036] On the other hand, embodiments of this application provide a multi-source heterogeneous data fusion device based on deep learning, the device comprising:

[0037] A multi-source heterogeneous data acquisition module is used to acquire scene information of text type and image type. The scene information is used to describe the scene of the image. The scene information includes at least first-level object information and second-level object information. The first-level object information represents the scene, and the second-level object information represents the target subordinate objects associated in the scene.

[0038] A cluster determination module is used to determine a model cluster based on the first-level object information, wherein the model cluster includes models for segmentation corresponding to each subordinate object in the scene represented by the first-level object information.

[0039] The model determination module is used to determine the target model corresponding to each target sub-object in the model cluster based on the secondary object information.

[0040] The coarse segmentation module is used to coarsely segment the image based on each of the target models to obtain the first image corresponding to each of the target sub-objects.

[0041] The fine segmentation module is used to segment each of the first images based on the information set associated with its corresponding target sub-object, and to obtain the fine segmentation result corresponding to each of the first images respectively.

[0042] The main image rendering module is used to render the main image corresponding to the image based on the fine segmentation result. The main image represents the fusion result of the scene information and the image.

[0043] On the other hand, embodiments of this application provide a computer-readable storage medium storing at least one instruction or at least one program, wherein the at least one instruction or at least one program is loaded and executed by a processor to implement the above-described deep learning-based multi-source heterogeneous data fusion method.

[0044] On the other hand, embodiments of this application provide an electronic device, including at least one processor and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the at least one processor implements the above-described deep learning-based multi-source heterogeneous data fusion method by executing the instructions stored in the memory.

[0045] On the other hand, embodiments of this application provide a computer program product, including a computer program or instructions, which, when executed by a processor, implement the aforementioned deep learning-based multi-source heterogeneous data fusion method.

[0046] This application provides a method for multi-source heterogeneous data fusion based on deep learning. This method first performs coarse segmentation of the image based on scene information. However, the accuracy of coarse segmentation is limited, and it only achieves this based on a preliminary fusion of scene information and the image. To achieve deeper multi-source heterogeneous data fusion, a clustering information extractor corresponding to the target sub-object can be selected by combining scene information. This clustering information extractor performs information aggregation on the image. This information aggregation operation can be considered as being guided by prior knowledge related to the target sub-object. Thus, the information aggregation process completes the deep fusion of prior knowledge related to the scene information and the image information. Based on the result of this aggregation process, fine segmentation can be performed, resulting in accurate fine segmentation results. The main image rendered based on this fine segmentation result reflects the fusion result of the scene information and the image, and even utilizes relevant prior knowledge, representing a deep fusion of multi-source heterogeneous data. Attached Figure Description

[0047] To more clearly illustrate the technical solutions and advantages in the embodiments of this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a schematic diagram of a feasible implementation framework for a deep learning-based multi-source heterogeneous data fusion method provided in the embodiments of this specification;

[0049] Figure 2 This is a flowchart illustrating the information aggregation method provided in the embodiments of this application;

[0050] Figure 3 This is a block diagram of a deep learning-based multi-source heterogeneous data fusion device provided in the embodiments of this application. Detailed Implementation

[0051] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the embodiments of this application.

[0052] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of the embodiments of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the present application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices.

[0053] To make the objectives, technical solutions, and advantages disclosed in the embodiments of this application clearer, the embodiments of this application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of the embodiments of this application and are not intended to limit the embodiments of this application.

[0054] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this embodiment, unless otherwise stated, "multiple" means two or more. To facilitate understanding of the above-described technical solutions and their resulting technical effects in the embodiments of this application, the embodiments of this application first explain the relevant technical terms:

[0055] The following describes a deep learning-based method for multi-source heterogeneous data fusion according to an embodiment of this application. Figure 1This illustration shows a flowchart of a deep learning-based multi-source heterogeneous data fusion method according to an embodiment of this application. The embodiments of this application provide the method operation steps described in the embodiments or flowcharts, but based on conventional or non-inventive labor, more or fewer operation steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many steps and does not represent the only execution order. In actual systems, terminal devices, or server products, the method can be executed sequentially according to the embodiments or drawings, or in parallel (e.g., in a parallel processor or multi-threaded processing environment). The above method may include:

[0056] S101. Obtain scene information of text type and image of image type. The scene information is used to describe the scene of the image. The scene information includes at least first-level object information and second-level object information. The first-level object information represents the scene, and the second-level object information represents the target subordinate object associated in the scene.

[0057] Scene information is a type of textual information that describes primary and secondary object information. Primary object information refers to the scene itself, such as an office scene, a sports scene, or a zoo scene. Secondary object information represents the target sub-objects within that scene and embodied in the image. Taking a zoo scene as an example, this scene could have four types of sub-objects: felines, birds, reptiles, and fish. If the image only shows two cats and one dog, then there are three target sub-objects: two cats and one dog. The method for constructing scene information can use existing technologies and is not the focus of this application; therefore, it will not be elaborated upon here.

[0058] S102. Determine a model cluster based on the first-level object information, wherein the model cluster includes models for segmentation corresponding to each subordinate object in the scene represented by the first-level object information.

[0059] This application also requires the construction of model clusters for various scenarios. The models in the model clusters can come from existing technologies or be trained by relevant developers. The models in the model clusters are used for coarse segmentation based on sub-objects, with low segmentation accuracy requirements. The difficulty in obtaining and training the models is also not high. Therefore, this is not the focus of this application and will not be elaborated here. Taking the zoo scenario as an example, the model cluster corresponding to the zoo scenario can include segmentation models for segmenting sub-objects such as cats, dogs, fish, and birds.

[0060] S103. Based on the secondary object information, determine the target model corresponding to each target subordinate object in the model cluster.

[0061] S104. Based on each of the target models, the image is coarsely segmented to obtain the first image corresponding to each of the target sub-objects.

[0062] Taking an image containing two cats and one dog as an example, two target models may be identified: target model 1 for segmenting the cats and target model 2 for segmenting the dog. Based on target model 1, two first images can be segmented, and based on target model 2, one first image can be segmented.

[0063] S105. Perform segmentation on each of the first images based on the information set associated with its corresponding target sub-object, to obtain the fine segmentation results corresponding to each of the first images.

[0064] In this embodiment, a clustering information extractor is provided for each type of subordinate object. This extractor is trained based on a detector corresponding to that type of subordinate object. The detector can be obtained using a detection model or segmentation model corresponding to that subordinate object, or it can be obtained using existing technologies, as long as it can detect that type of subordinate object. The method of obtaining the detector is not limited. This clustering information extractor can extract clustering information from the corresponding first image to obtain its corresponding fused feature information.

[0065] Taking the previous example, target model 1 can segment a first image containing a cat from the image. This first image definitely contains a cat, but the specific outline of the cat cannot be clearly determined. In this case, the first image can be input into the corresponding clustering information extractor to obtain feature fusion information about the cat. This clustering information extractor is trained based on a detector capable of detecting cats, making it particularly suitable for fusing features of objects like cats to obtain feature fusion information about cats.

[0066] Specifically, for each first image, the first image can be input into the corresponding clustering information extractor. The clustering information extractor is trained based on the detector corresponding to the target sub-object of the first image. The clustering information extractor is used to extract clustering information for the target sub-object in the first image to obtain the fusion feature information corresponding to the first image.

[0067] Then, the fused feature information can be input into the segmenter corresponding to the target sub-image of the first image to obtain the mask matrix. Continuing with the previous example, because the information extraction through clustering yields very rich fused feature information about the cat—meaning the fused feature information itself has very high information clustering and quality—inputting this information into a segmenter capable of segmenting objects like cats yields a mask matrix representing the fine-grained segmentation result. This application does not limit the method for obtaining this segmenter; it can use existing segmenters or be trained independently, as the fine-grained segmentation effect mainly depends on the quality of the fused feature information, and the requirements for the segmenter are not particularly high.

[0068] S106. Render the subject image corresponding to the image based on the fine segmentation result, wherein the subject image represents the fusion result of the scene information and the image.

[0069] Fine segmentation, compared to coarse segmentation, can very accurately segment the outlines of specific objects such as cats and dogs. Therefore, based on the fine segmentation results, the main image corresponding to the object can be rendered. That is, the outlines and actual content of the main objects recorded in the scene information of the image are rendered to obtain the main image. Essentially, it is based on the scene information to complete the fine outline segmentation of the image. Therefore, the main image representing the fusion result of the scene information and the image is obtained.

[0070] Specifically, the main image can be rendered based on the mask matrix corresponding to each first image and the image itself.

[0071] The main concept of this application embodiment is to perform coarse segmentation, information aggregation, and fine segmentation based on the target sub-object in the scene information. That is to say, the neural networks used in these three stages are all designed for this type of target sub-object. Thus, at each logical node, coarse segmentation, information aggregation, and fine segmentation adapted to the target sub-object are performed respectively. These nodes all use neural networks. Since the training process of the neural network necessarily uses knowledge of this type of object to which the target sub-object belongs, it can be considered that in each processing stage, this application embodiment uses prior knowledge of this type of object throughout the entire process. This knowledge can ultimately be applied to the mask matrix output by the fine segmentation, making the mask matrix clear and accurate.

[0072] The methods for obtaining or training the model or segmenter used in the coarse and fine segmentation stages are not difficult and can be referenced from existing technologies. Therefore, this application does not limit the specific methods used in this application. Instead, this application limits the overall inventive concept of processing the target sub-object through the entire process of coarse segmentation, information aggregation, and fine segmentation, thereby ensuring that the knowledge of objects of the type described in the target sub-object can be applied to the mask matrix output by the fine segmentation, thus obtaining a clear and accurate mask matrix.

[0073] Of course, the concept of information aggregation is also proposed in the embodiments of this application. Therefore, the original parts of the steps related to information aggregation are described in detail in the embodiments of this application. Specifically, the aggregated information extractor mainly includes an extraction network and a mask generation network. The mask generation network is used to generate a target mask. The target mask is used to distinguish between lower-level objects and non-lower-level objects in the image input to the aggregated information extractor. For this application, it can be used to distinguish the target lower-level object in the first image from other objects.

[0074] like Figure 2 As shown, the extraction network and mask generation network are trained using the following method:

[0075] S201. Obtain a sample image and a detector corresponding to the type of the subordinate object corresponding to the sample image. The sample image carries annotation information, which represents the position information of the subordinate object in the sample image. A sample image includes only a single type of subordinate object.

[0076] This application uses the training of a network for extracting information from a clustered information extractor corresponding to a cat as an example to illustrate the application. The sample image contains only cats and is labeled with the cat's location information. The detector can also be used to detect objects such as cats.

[0077] S202. Input the sample image into the extraction network, and the extraction network performs the following operations: for each pixel region in the sample image, extract the sample first feature corresponding to the pixel region, extract the associated position features corresponding to each associated pixel position, and fuse the associated position features to obtain the sample second feature corresponding to the pixel region, wherein the associated pixel position is other pixel position in the neighborhood of the pixel region that does not belong to the pixel region.

[0078] This application does not limit the division of pixel regions in its embodiments; it can be set according to actual conditions. The result of coarse segmentation is a detection box. The first image is obtained by extracting the image from the detection box. For relevant knowledge on obtaining the detection box through segmentation, please refer to existing technologies, which will not be elaborated here. Therefore, the first image is a rectangular image. The first image can be divided into a 3x3 or 16x3 grid, with each grid corresponding to a pixel region. Of course, the finer the pixel region division, the better the segmentation effect.

[0079] The embodiments of this application do not limit the scope and determination method of the neighborhood. For example, it is sufficient to include the pixel region and other pixels outside the pixel region that do not belong to the pixel region.

[0080] In this embodiment, the first feature of the sample represents the feature of the pixel region itself, while the second feature of the sample actually represents the scene feature where the pixel region is located. The extraction methods of these two features will not be elaborated here. They can be implemented by one or more combinations of convolution, multi-layer convolution, self-attention-based convolution, multi-channel fusion, pooling, etc. In this embodiment, they will not be elaborated here.

[0081] S203. Input the first feature and second feature of the sample corresponding to each pixel region into the mask generation network to obtain the sample target mask.

[0082] The mask generation network predicts the target mask of a sample based on the first and second features of the sample. The part of the sample image covered by the target mask can be considered as the region where more important pixels are formed, thus filtering out the region with the highest effective information content in the sample image.

[0083] S204. For each pixel region, input the first features of other samples excluding the corresponding first features of the sample into the detector to obtain a first detection result corresponding to the pixel region, and input the second features of other samples excluding the corresponding second features of the sample into the detector to obtain a second detection result corresponding to the pixel region.

[0084] For each pixel region, the other sample first features excluding the corresponding sample first features form a sample first feature information set that does not include the sample first features in that pixel region. The sample first feature information set is input into the detector to obtain a first detection result. If the first detection result is very consistent with the annotation information, it means that the presence or absence of the pixel region has little impact. Therefore, the pixel region must be a region formed by unimportant pixels, and the effective information content therein must be low. The pixel region is likely unrelated to the lower objects in the sample image.

[0085] For each pixel region, the other sample second features excluding its corresponding sample second features form a sample second feature information set that does not include the sample second features in that pixel region. This sample second feature information set is input into the detector to obtain a second detection result. If the second detection result closely matches the annotation information, it means that the presence or absence of scene information in that pixel region has little impact. In this case, the other locations in the neighborhood of that pixel region must be the locations of unimportant pixels. The effective information content of the other locations in the neighborhood of that pixel region must be low. That pixel region is likely unrelated to the lower objects in the sample image or located on the edge of the lower objects in the sample image.

[0086] S205. Based on each of the first detection results, each of the second detection results, and the annotation information, an indication target mask is obtained.

[0087] Specifically, for each pixel region corresponding to the first and second detection results, if the difference between the first detection result and the labeled information is greater than a first threshold, and the difference between the second detection result and the labeled information is greater than a second threshold, the pixel region is determined to belong to a core pixel region. This application does not limit the difference degree; it simply represents the degree of difference between the detection result and the labeled information. Any calculation method is acceptable. This application does not limit the difference degree calculation method; many methods for measuring difference degree can be used in the neural network field, as long as they can reflect the difference between the two pieces of information. If both the first and second detection results differ significantly from the labeled information, it indicates that the pixel region is very important, and therefore it is likely to belong to a core pixel region. A core pixel region represents a region carrying partial information about a lower-level object. This application does not limit the first and second thresholds; they can be set during actual neural network training. The first threshold can be greater than the second threshold.

[0088] If the difference between the first detection result and the annotation information is greater than a first threshold, and the difference between the second detection result and the annotation information is less than or equal to a second threshold, the pixel region is determined to belong to a boundary pixel region.

[0089] Specifically, if the first detection result differs significantly from the annotation information, it indicates that the pixel region is very important and cannot be omitted. However, if the second detection result differs very little from the annotation information, it indicates that the area surrounding the pixel region is not so important and can be omitted. In this case, the pixel region is likely located at the boundary of the lower object in the sample image.

[0090] The indicator target mask can be generated based on the core pixel region and the boundary pixel region. The indicator target mask is determined based on the annotation information and can be considered as the ground truth information corresponding to the sample target mask predicted by the mask generation network.

[0091] A highly accurate target indication mask can be generated by combining the first and second detection information with the annotation information. However, in another embodiment of this application, the accuracy of the target indication mask can be further improved by utilizing the difference information between the first feature and the second feature of the sample. Specifically:

[0092] S301. For each pixel region, calculate the difference between its corresponding sample first feature and its corresponding sample second feature to obtain difference information.

[0093] This difference information can be obtained by calculating the feature distance between the first feature and the second feature of the sample. Feature distance is a type of information distance and can be measured using information distance measurement methods. This application does not limit the scope of this measurement. This difference information characterizes the information distance between a pixel region and its surrounding adjacent regions. If the pixels are very close and do not exhibit significant jumps, the pixel region is likely located inside or outside a lower-level object in the sample image; in short, there will be no intersection. Conversely, the pixel region is likely located at the edge of a lower-level object.

[0094] If the difference between the first detection result and the annotation information is greater than a first threshold, and the difference between the second detection result and the annotation information is less than or equal to a second threshold, determining that the pixel region belongs to a boundary pixel region includes:

[0095] If the difference between the first detection result and the annotation information is greater than a first threshold, and the difference between the second detection result and the annotation information is less than or equal to a second threshold, and the difference information is greater than a third threshold, then the pixel region is determined to belong to a boundary pixel region.

[0096] S206. Based on the difference between the sample target mask and the indicator target mask, adjust the parameters of the extraction network and the mask generation network until training is complete.

[0097] The methods for expressing the differences between the sample target mask and the indicator target mask, the methods for feedback parameter adjustment, and the conditions for training completion can all refer to existing technologies in the field of neural networks, and will not be elaborated here. Furthermore, the network structures of the extraction network and the mask generation network can be designed according to actual conditions, for example, based on a deep convolutional neural network. This application does not limit the specific implementation of these structures.

[0098] After training, this embodiment of the application can extract information from the first image based on the clustering information extractor to obtain fused feature information. Specifically, the clustering information extractor further includes a clustering network. After the first image is input into the corresponding clustering information extractor, the clustering information extractor performs the following operations:

[0099] The first image is input into the extraction network, triggering the mask generation network to generate a target mask corresponding to the first image. This process, as described above, is performed by the extraction network and the mask generation network. Based on the clustering network, depth features are extracted from the portion of the first image covered by the target mask to obtain the first target features. In this embodiment, the portion covered by the target mask is considered to carry important and effective information, representing a region where effective information is concentrated. This region highly overlaps with the region containing the target's lower-level object in the first image. Therefore, important first target features need to be extracted from this region. The first target features can be extracted using a clustering network with a pyramid multi-scale structure, which can extract rich multi-scale information. By fusing this multi-scale information, the first target features can be obtained.

[0100] This application embodiment believes that although effective information is concentrated in the area covered by the target mask, there may also be sparse effective information in the areas not covered by the target mask. If this sparse effective information is extracted simultaneously with the effective information in the area covered by the target mask, it will be submerged by the concentrated effective information in the area covered by the target mask and will be difficult to extract. In order to extract the sparse effective information, further improve the concentration of fused feature information, and make full use of the target mask, this application embodiment, after obtaining the first target feature, can also perform multi-scale feature extraction on the part of the first image not covered by the target mask based on the concentration network to obtain the second target feature. Specifically, the pixels of the part of the first image covered by the target mask can be set to zero, thereby erasing the concentrated effective information in the area covered by the target mask. Multi-scale feature extraction is then performed on the erasure result to obtain the second target feature. The erasure operation reduces the submersion effect, thereby extracting the aforementioned sparse effective information.

[0101] Finally, the first target feature and the second target feature are fused to obtain the fused feature information. In this application embodiment, there are many methods for fusion, such as weighted addition, multiplication, and convolution, which can all be used for fusion. The appropriate method can be selected as needed, and no limitation is imposed.

[0102] This application provides a method for multi-source heterogeneous data fusion based on deep learning. This method first performs coarse segmentation of the image based on scene information. However, the accuracy of coarse segmentation is limited, and it only achieves this based on a preliminary fusion of scene information and the image. To achieve deeper multi-source heterogeneous data fusion, a clustering information extractor corresponding to the target sub-object can be selected by combining scene information. This clustering information extractor performs information aggregation on the image. This information aggregation operation can be considered as being guided by prior knowledge related to the target sub-object. Thus, the information aggregation process completes the deep fusion of prior knowledge related to the scene information and the image information. Based on the result of this aggregation process, fine segmentation can be performed, resulting in accurate fine segmentation results. The main image rendered based on this fine segmentation result reflects the fusion result of the scene information and the image, and even utilizes relevant prior knowledge, representing a deep fusion of multi-source heterogeneous data.

[0103] Please refer to Figure 3 This diagram illustrates a block diagram of a deep learning-based multi-source heterogeneous data fusion device in this embodiment. The device includes:

[0104] The multi-source heterogeneous data acquisition module 101 is used to acquire scene information of text type and image type. The scene information is used to describe the scene of the image. The scene information includes at least first-level object information and second-level object information. The first-level object information represents the scene, and the second-level object information represents the target subordinate object associated in the scene.

[0105] Cluster determination module 102 is used to determine a model cluster based on the first-level object information, wherein the model cluster includes models for segmentation corresponding to each subordinate object in the scene represented by the first-level object information.

[0106] The model determination module 103 is used to determine the target model corresponding to each target sub-object in the model cluster based on the secondary object information.

[0107] The coarse segmentation module 104 is used to coarsely segment the image based on each of the target models to obtain the first image corresponding to each of the target sub-objects.

[0108] The fine segmentation module 105 is used to segment each of the first images based on the information set associated with its corresponding target sub-object, and obtain the fine segmentation result corresponding to each of the first images respectively.

[0109] The main image rendering module 106 is used to render the main image corresponding to the image based on the fine segmentation result, wherein the main image represents the fusion result of the scene information and the image.

[0110] In one embodiment, the fine segmentation module is used for:

[0111] For each first image, the first image is input into the corresponding clustering information extractor. The clustering information extractor is trained based on the detector corresponding to the target sub-object of the first image. The clustering information extractor is used to extract clustering information for the target sub-object in the first image to obtain the fusion feature information corresponding to the first image.

[0112] The fused feature information is input into the segmenter corresponding to the target sub-image of the first image to obtain the mask matrix.

[0113] In one embodiment, the main image rendering module is used for:

[0114] The main image is rendered based on the mask matrix corresponding to each first image and the image itself.

[0115] The apparatus portion of the embodiments in this application is based on the same inventive concept as the method embodiment, and will not be described in detail here.

[0116] On the other hand, embodiments of this application provide an electronic device, including at least one processor and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the at least one processor implements the above-described deep learning-based multi-source heterogeneous data fusion method by executing the instructions stored in the memory.

[0117] On the other hand, embodiments of this application provide a computer program product, including a computer program or instructions, which, when executed by a processor, implement the aforementioned deep learning-based multi-source heterogeneous data fusion method.

[0118] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present application should be included within the protection scope of the present application.

Claims

1. A method for fusing multi-source heterogeneous data based on deep learning, characterized in that, include: Acquire scene information in text type and image type, wherein the scene information includes at least first-level object information and second-level object information, the first-level object information representing the scene, and the second-level object information representing the target subordinate objects associated in the scene; A model cluster is determined based on the first-level object information, and the model cluster includes segmentation models corresponding to each subordinate object in the scene represented by the first-level object information; a target model corresponding to each target subordinate object is determined in the model cluster based on the second-level object information; the image is coarsely segmented based on each target model to obtain a first image corresponding to each target subordinate object; By inputting each first image into the corresponding clustering information extractor, a corresponding fine segmentation result is obtained; based on the fine segmentation result, a subject image corresponding to the image is rendered, the subject image representing the fusion result of the scene information and the image; the clustering information extractor includes an extraction network and a mask generation network, and is trained using the following method: A sample image and a detector corresponding to the corresponding lower-level object are obtained, wherein the annotation information of the sample image represents the position information of the lower-level object; the sample image is input into an extraction network to perform the following for each pixel region in the sample image: extracting the corresponding sample first feature and sample second feature; the sample second feature is obtained by extracting the associated position features corresponding to each associated pixel position and fusing the associated position features, wherein the associated pixel position is other pixel position in the neighborhood of the pixel region that does not belong to the pixel region; The first feature and the second feature of each sample are input into the mask generation network to obtain the sample target mask; For each pixel region, a first detection result and a second detection result are obtained. The first detection result is obtained by inputting the first feature of other samples that do not correspond to the pixel region into the detector, and the second detection result is obtained by inputting the second feature of other samples that do not correspond to the pixel region into the detector. Based on each of the first detection results, each of the second detection results, and the annotation information, an indication target mask is obtained; Based on the difference between the sample target mask and the indicator target mask, adjust the parameters of the extraction network and the mask generation network until training is complete.

2. The method according to claim 1, characterized in that, The refined segmentation result includes a mask matrix. The process of obtaining the corresponding refined segmentation result by inputting each first image into a corresponding clustering information extractor includes: For each first image, the first image is input into the corresponding clustering information extractor. The clustering information extractor is trained based on the detector corresponding to the target sub-object of the first image. The clustering information extractor is used to extract clustering information for the target sub-object in the first image to obtain the fusion feature information corresponding to the first image. The fused feature information is input into the segmenter corresponding to the target sub-image of the first image to obtain the mask matrix.

3. The method according to claim 2, characterized in that, The step of rendering the subject image corresponding to the image based on the fine segmentation result, wherein the subject image represents the fusion result of the scene information and the image, includes: The main image is rendered based on the mask matrix corresponding to each first image and the image itself.

4. The method according to claim 3, characterized in that, A sample image contains only a single type of sub-object.

5. The method according to claim 4, characterized in that, The step of obtaining the target indication mask based on each of the first detection results, each of the second detection results, and the annotation information includes: For each pixel region, if the difference between the first detection result and the annotation information is greater than a first threshold, and the difference between the second detection result and the annotation information is greater than a second threshold, then the pixel region is determined to belong to the core pixel region. If the difference between the first detection result and the annotation information is greater than a first threshold, and the difference between the second detection result and the annotation information is less than or equal to a second threshold, the pixel region is determined to belong to a boundary pixel region. The target mask is generated based on the core pixel region and the boundary pixel region.

6. A multi-source heterogeneous data fusion device based on deep learning, characterized in that, The device includes: A multi-source heterogeneous data acquisition module is used to acquire scene information of text type and image type. The scene information includes at least first-level object information and second-level object information. The first-level object information represents the scene, and the second-level object information represents the target subordinate objects associated in the scene. A cluster determination module is used to determine a model cluster based on the first-level object information, wherein the model cluster includes segmentation models corresponding to each lower-level object in the scene represented by the first-level object information. The model determination module is used to determine the target model corresponding to each target sub-object in the model cluster based on the secondary object information. The coarse segmentation module is used to coarsely segment the image based on each of the target models to obtain the first image corresponding to each of the target sub-objects. The fine segmentation module is used to obtain the corresponding fine segmentation result by inputting each first image into the corresponding clustering information extractor; The main image rendering module is used to render the main image corresponding to the image based on the fine segmentation result. The main image represents the fusion result of the scene information and the image. The clustering information extractor includes an extraction network and a mask generation network, and is trained by the following method: A sample image and a detector corresponding to the corresponding lower-level object are obtained, wherein the annotation information of the sample image represents the position information of the lower-level object; the sample image is input into an extraction network to perform the following for each pixel region in the sample image: extracting the corresponding sample first feature and sample second feature; the sample second feature is obtained by extracting the associated position features corresponding to each associated pixel position and fusing the associated position features, wherein the associated pixel position is other pixel position in the neighborhood of the pixel region that does not belong to the pixel region; The first feature and the second feature of each sample are input into the mask generation network to obtain the sample target mask; For each pixel region, a first detection result and a second detection result are obtained. The first detection result is obtained by inputting the first feature of other samples that do not correspond to the pixel region into the detector, and the second detection result is obtained by inputting the second feature of other samples that do not correspond to the pixel region into the detector. Based on each of the first detection results, each of the second detection results, and the annotation information, an indicator target mask is obtained; based on the difference between the sample target mask and the indicator target mask, the parameters of the extraction network and the mask generation network are adjusted until training is complete.

7. The apparatus according to claim 6, characterized in that, The refined segmentation result includes a mask matrix, and the refined segmentation module is used for: For each first image, the first image is input into the corresponding clustering information extractor. The clustering information extractor is trained based on the detector corresponding to the target sub-object of the first image. The clustering information extractor is used to extract clustering information for the target sub-object in the first image to obtain the fusion feature information corresponding to the first image. The fused feature information is input into the segmenter corresponding to the target sub-image of the first image to obtain the mask matrix.

8. The apparatus according to claim 7, characterized in that, The main image rendering module is used for: The main image is rendered based on the mask matrix corresponding to each first image and the image itself.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one instruction or at least one program, which is loaded and executed by a processor to implement a deep learning-based multi-source heterogeneous data fusion method as described in any one of claims 1 to 5.

10. An electronic device, characterized in that, The method includes at least one processor and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the at least one processor implements a deep learning-based multi-source heterogeneous data fusion method as described in any one of claims 1 to 5 by executing the instructions stored in the memory.

Images