Method, device, computer-readable storage medium, and electronic device for multi-source heterogeneous data fusion using deep learning

Abstract

To provide a method for fusing multi-source heterogeneous data on the basis of deep learning applicable to a field of image processing.SOLUTION: The method includes: acquiring scene information of a text type including primary object information and secondary object information and an image of a picture type; determining a model cluster from the first object information; determining a target model for each target subordinate object in the model cluster on the basis of the second level object information; roughly dividing an image on the basis of each target model; obtaining each first image and dividing each first image on the basis of information accumulation related to the corresponding target subordinate object; obtaining respective minute division results for the first images and rendering a body image of an image which characterizes the result of fusion of the scene information and the image on the basis of the result of refining.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to the field of image processing technology, in particular, deep learning-based Multi-source This invention relates to a heterogeneous data fusion method. [Background technology]

[0002] Multi The data from a source may contain information for the same object, i.e. information for the same object may be stored in different types of Multi-source Data may be recorded on different carriers. Different data may be stored on the same Object It represents different aspects of information and how Multi-source Perform information fusion on the data, so that the same Object or Multi-source Fully integrated from the data center Object Extracting the information from the images and carrying out other applications based on the fusion results has always been an important research issue in the field of image processing. Summary of the Invention [Problem to be solved by the invention]

[0003] The present invention aims to solve at least one of the technical problems existing in the prior art. [Means for solving the problem]

[0004] First, object information characterizing the scene and associated objects in the scene Target Lower and second object information characterizing the object. Scene description using images Text type scene information, Picture type images and get. Based on the first object information, a model cluster is determined that includes a segmentation model corresponding to each lower-level object in the scene characterized by the first object information, and the model cluster includes a segmentation model corresponding to each lower-level object in the scene characterized by the first object information. Second Object Based on the information, each Target Lower Corresponding to Objects target The model is determined. each target Based on the model, the image is roughly segmented and each Target Subobject Each of them obtains a corresponding first image. For each of the first images, target A segmentation based on information aggregation associated with the lower-level objects is performed to obtain subdivision results corresponding to each of the first images. Characterizing the results of scene information and image fusion 、 The main image corresponding to the image is Split Rendered based on the results. In one embodiment, the subdivision results include a mask matrix, and each of the first images includes a corresponding Target Lower Information associated with an object Aggregation of the first image. each and each of the first images corresponds to a subdivision result of the first image. each can be obtained by subdivision First image For each first image of, handle Consolidation Input to the information extractor, Consolidation The information extractor extracts the first image but handle Target Subobject Detector for Based on training and obtaining fusion feature information corresponding to the first image by Target Subobject Against Consolidation Perform information extraction Consolidation It is an information extractor. The fusion feature information is targetThe sub-images are input to a divider corresponding to the sub-image to obtain a mask matrix.

[0005] A body image is rendered based on a mask matrix and an image corresponding to each of the first images.

[0006] Aggregated Information Extraction vessel The method mainly consists of an extraction network and a mask generation network. The mask generation network is used to generate a target mask for distinguishing between subordinate objects and non-subordinate objects in the image input to the aggregate information extractor. The extraction network and the mask generation network are trained in the following manner: A sample image and a detector corresponding to a type of subordinate object corresponding to the sample image are acquired, the sample image has labeling information characterizing position information of the subordinate objects within the sample image, and the sample image includes only a single type of subordinate object. A sample image is input to the extraction network, and the extraction network performs the following operations: for each pixel region in the sample image, extract a sample first feature corresponding to the pixel region, extract an associated position feature corresponding to each associated pixel location, and fuse the associated position features to obtain a sample second feature corresponding to the pixel region, where the associated pixel locations are other pixel locations that do not belong to the neighboring pixel regions in which the pixel region is located. The sample first feature and the sample second feature corresponding to each pixel region are input to a mask generation network to obtain a sample target mask. For each pixel region, the other sample first features excluding the corresponding sample first feature are input to the detector to obtain a first detection result corresponding to the pixel region, and the other sample second features excluding the corresponding sample second feature are input to the detector to obtain a second detection result corresponding to the pixel region. An instruction target mask is determined based on each of the first detection results, each of the second detection results, and the labeling information. Depending on the difference between the sample target mask and the instruction target mask, the parameters of the extraction network and the mask generation network are adjusted until training is complete.

[0007] For the first detection result and the second detection result corresponding to each pixel region, if the degree of difference between the first detection result and the labeling information is greater than a first threshold value, and the degree of difference between the second detection result and the labeling information is greater than a second threshold value, the pixel region is determined to belong to the core pixel region. If the degree of difference between the first detection result and the labeling information is greater than a first threshold and the degree of difference between the second detection result and the labeling information is equal to or smaller than a second threshold, the pixel region is determined to belong to a boundary pixel region. Based on the core pixel area and the boundary pixel area, Target Mask is generated. For each pixel region, the difference between the corresponding sample first feature and the corresponding sample second feature is calculated to obtain dissimilarity information. When a degree of difference between the first detection result and the labeling information is greater than a first threshold and a degree of difference between the second detection result and the labeling information is equal to or smaller than a second threshold, the pixel region is determined to belong to a boundary pixel region: If the degree of difference between the first detection result and the labeling information is greater than a first threshold, the degree of difference between the second detection result and the labeling information is equal to or less than a second threshold, and the degree of difference information is greater than a third threshold, the pixel region is determined to belong to a boundary pixel region.

[0008] A first image is input to the extraction network, and a mask generation network is triggered to generate a target mask corresponding to the first image. For the portion of the first image covered by the target mask, Consolidation Depth feature extraction is performed based on the network to obtain the first target feature. For a portion of the first image that is not masked by the target mask, ConsolidationMulti-scale feature extraction is performed based on the network to obtain the second target feature. The first target feature and the second target feature are fused to obtain fused feature information. On the other hand, the embodiment of the present invention is based on deep learning Multi-source Providing a heterogeneous data fusion device: Multi-source A heterogeneous data acquisition module acquires information about a first object characterizing a scene and information about associated objects in the scene. Target Lower and second object information characterizing the object. Scene description using images Scene information of text type and images of picture type are obtained. The cluster determination module determines, based on the first object information, a model cluster including a segmentation model corresponding to each subordinate object in the scene characterized by the first object information. The model determination module Second Object Based on the information, each Target Lower Corresponding to Objects target Determine the model. The coarse segmentation module coarsely segments the image based on each target model, and each target sub-object is divided into a corresponding first sub-object. of Obtain an image. each 1st of For the image, Target Lower The object is associated Information aggregation A subdivision module for performing subdivision based on each of the first of Obtain subdivision results corresponding to each image. Characterize scene information and image fusion results, subdivision A body image rendering module is provided for rendering a body image corresponding to the image based on the result. On the other hand, the embodiment of the present invention is based on deep learning Multi-source A computer readable storage medium is provided having stored thereon at least one instruction or at least one program that is loaded and executed by a processor to implement one of the heterogeneous data fusion methods. In another aspect, an embodiment of the present invention provides an electronic device including at least one processor and a memory communicatively coupled to the at least one processor, where the memory stores instructions executable by the at least one processor, and where the at least one processor executes the instructions stored in the memory to perform the deep learning-based computation described above. Multi-source This realizes one of the heterogeneous data fusion methods. On the other hand, the embodiments of the present invention, when executed by a processor, Multi A computer program product is provided that includes a computer program or instructions for implementing one of the source heterogeneous data fusion methods. Effect of the Invention

[0009] An embodiment of the present invention is based on deep learning Multi-source A heterogeneous data fusion method is provided. This method first completes the coarse segmentation of an image based on scene information through coarse segmentation. However, the accuracy of the rough segmentation is limited, and the scene information and the image are only merged. In order to integrate the diverse data more deeply, the scene information is further combined with the target sub-objects in the scene. Consolidation It is also possible to select an information extractor. Consolidation Image information extracted by information extractor Aggregation This information is Aggregation The manipulation of the target object is considered to be based on prior knowledge about the target object. Aggregation During this process, the fusion of prior knowledge about the scene information and the image information becomes deeper, Aggregation Based on the result of the process, a fine segmentation can be performed, so that an accurate subdivision result can be obtained. The main image rendered based on the subdivision result embodies the fusion result of the scene information and the image, and further uses related prior knowledge to perform a kind of Multi-source This is the embodiment of deep integration of heterogeneous data. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a schematic diagram of a possible implementation framework of the deep learning-based multi-source heterogeneous data fusion method provided by an embodiment. [Diagram 2] FIG. 2 is a diagram showing a flow of the information aggregation method according to the embodiment of the present invention. [Diagram 3] FIG. 3 is a block diagram of a multi-source heterogeneous data fusion device based on deep learning provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Based on deep learning of the embodiment Multi-source FIG. 1 illustrates a heterogeneous data fusion method based on deep learning provided by an embodiment of the present application. Multi-source 1 shows the flow of a heterogeneous data fusion method. The embodiments of the present application provide the operation steps of the method described above in the embodiments or flowcharts, but may include more or less operation steps based on conventional or non-inventive efforts. The step order listed in the embodiments is only one of many step execution orders, and does not represent a unique execution order. When an actual system, terminal device, or server product is executed, it can be executed sequentially or in parallel according to the embodiment or the method shown in the attached figures (e.g., a parallel processor or a multi-threaded processing environment), and the above method can include the following. EXAMPLES

[0012] Step S101: First object information characterizing a scene and associated objects in the scene are obtained. target and second object information characterizing the subordinate object. Scene description using images Text type scene information and , Picture type image and get. The scene information is text information that describes first object information and second object information, and the first object information is the scene itself, such as an office scene, a sports scene, or a zoo scene. Second Object Information characterizes the target sub-objects that are in the scene and embodied in the image. Take a zoo scene for example, there can be four types of sub-objects: felines, birds, reptiles, and fish. However, if there are only two cats and one dog in the image, target There are three sub-objects, two cats and one dog. The construction method of the scenario information can use the conventional technology, and is not the focus of the invention of this application, so it will not be mentioned here.

[0013] Step S102. First Object Based on the information, First Object A model cluster is determined that includes a segmentation model corresponding to each subordinate object in the scene that the information characterizes. The present application also provides models in various related scenarios. cluster You need to build Model Cluster The models in the model may be obtained from prior art or may be trained by relevant developers themselves. Model Cluster The model in the middle is lower Object It is used to perform rough segmentation based on, and the requirements for segmentation accuracy are not high, and the difficulty of obtaining and training the model is not high, so it is not the focus of the invention of this application, so it will not be mentioned here. Taking a zoo scene as an example, the model cluster corresponding to the zoo scene can include segmentation models for segmenting sub-objects such as cats, dogs, fish, and birds.

[0014] Step S103. Second Object Based on the information, each Target Lower Corresponding to Objects target Determine the model.

[0015] Step S104: Coarsely segment the image based on each of the target models, and generate first sub-objects corresponding to each of the target sub-objects. of Obtain an image. If there are two cats and one dog in an image, two target models can be determined: target model 1 for segmenting the cats and target model 2 for segmenting the dog; two first images can be segmented based on target model 1, and one first image can be segmented based on target model 2.

[0016] Step S105: For each of the first images, perform segmentation based on information aggregation associated with the corresponding target subordinate objects, to obtain detailed segmentation results respectively corresponding to each of the first images. In the embodiment of the present invention, one aggregated information extractor is provided corresponding to each type of lower-level object, and the aggregated information extractor is obtained based on a detector training corresponding to the lower-level object of this type. The detector may be obtained from a detection model or segmentation model corresponding to the lower-level object, or may be obtained using a conventional technique. However, as long as it can detect such a type of lower-level object, the detector acquisition method is not limited. vessel may perform aggregate information extraction on the corresponding first image to obtain corresponding fusion feature information.

[0017] In the above example, the target model 1 can extract a first image including a cat in which there is no doubt that a cat exists in the image, but the specific outline of the cat cannot be clearly identified. Consolidation By inputting the information into the information extractor, we can obtain feature fusion information about cats. Consolidation The information extractor is trained on a detector that can detect cats, so it is Object The method is particularly suitable for fusing features of a cat to obtain feature fusion information about the cat.

[0018] in particular, First image For each first image of ,handle Consolidation Input to information extractor death , ConsolidationThe information extractor extracts the first image but handle Target Subobject Detector for Based on training and obtaining fusion feature information corresponding to the first image by Target Subobject Against Consolidation Perform information extraction Consolidation It is an information extractor. Next, the first image but handle Target Lower Following the previous example, we can input the fused feature information into the segmenter corresponding to the image to obtain the mask matrix. Consolidation By the information extraction, we can obtain a very rich set of fusion feature information about cats, i.e., the information of the fusion feature information itself. Aggregation The accuracy and quality of the information is very high, and this information is sent to cat-like types. Object By inputting the mask matrix to a divider that can divide, a mask matrix can be obtained to characterize the fine division result. The present application does not limit the method of obtaining this divider, but it can use the divider in the prior art or can be obtained by self-training. Because the effect of fine division mainly depends on the quality of fusion feature information, and the requirements for the divider are not particularly high.

[0019] Step S106: Characterize the fusion result between the scene information and the image. 、 An object image corresponding to the image is rendered based on the subdivision results. subdivision Compared to the coarse segmentation, Object The contours of the object can be segmented very accurately, and therefore, based on the subdivision result, a subject image corresponding to said object can be rendered, i.e., the contours and substance of the main subject recorded in the scene information of the image are rendered to obtain a main image, essentially a detailed contour segmentation of the image based on the scene information to obtain a main image that characterizes the fusion result of the scene information and the image. EXAMPLES

[0020] Step S201. Obtain a sample image carrying labeling information characterizing the position information of subordinate objects in the sample image and a detector corresponding to the type of subordinate object corresponding to the sample image, where the sample image only contains a single type of subordinate object. In the embodiment of the present invention, Object Corresponding to Consolidation We have taken the training of the extraction network and mask generation network in the information extractor as an example. The sample image contains only cats, and the cat's position information is added. The detector also detects cats. Object It is a detector that can be used to detect.

[0021] Step S202. Input the sample image to the extraction network. The extraction network performs the following operations: for each pixel region in the sample image, Pixels Samples corresponding to the region First Feature Extract a related position feature corresponding to each related pixel position, and fuse the related position features to obtain a sample second feature corresponding to the pixel region, where the related pixel positions are other pixel positions that do not belong to the neighboring pixel region in which the pixel region is located.

[0022] In this embodiment, the division of the pixel region is not limited, and can be set from the row according to the actual situation. The result of the rough division is the detection frame, and the image within the detection frame can be cut out to obtain the first image. Since the knowledge on dividing the detection frame and obtaining it can be found in the prior art, it will be omitted, and the first image is a rectangular image. of An image can be divided into 9 or 16 grids, with each grid corresponding to one pixel region. Naturally, the finer the pixel region division, the greater the division effect. The embodiment of the present invention does not limit the range and the method of determining the neighboring region, and may simply include, for example, the pixel region and other pixels that do not belong to the pixel region surrounding the pixel region. Samples in the embodiments of the present application First Feature characterizes the pixel region's own characteristics, while the sample Second feature actually characterize the scene feature where the pixel region is located, and there is no mention of how to extract these two features, which can be implemented by one or more combinations of methods such as convolution, multi-layer convolution, self-attention based convolution, multi-channel fusion, pooling, etc., which are not mentioned in the embodiments of this application.

[0023] Step S203: input the sample first feature and the sample second feature corresponding to each pixel region into a mask generation network to obtain a sample target mask. The mask generation network predicts a sample target mask based on the sample first feature and the sample second feature, and the portion of the sample image covered by this sample target mask can be considered to be an area formed by more important pixels, and the area of ​​the sample image with the highest content of useful information is screened.

[0024] Step S204. For each pixel region, the corresponding sample First Feature Other samples except First Feature is input to the detector, a first detection result corresponding to a pixel area is obtained, and a corresponding sample Second feature Other samples except Second feature is input to the detector, and a second detection result corresponding to the pixel area is obtained. each Pixels For each region, the corresponding sample First Feature Except for other samples First Feature is the sample in the pixel domain First Feature Sample 1 feature does not include Information Set Forming a sample First feature information set is input into the detector, a first detection result is obtained, and if this first detection result is highly consistent with the labeling information, the impact of the presence or absence of this pixel area is not significant, this pixel area is inevitably an area formed by unimportant pixels, and the content of effective information therein is inevitably low, and this pixel area is likely to be unrelated to the subordinate object in the sample image.

[0025] each Pixels For each region, the corresponding sample Second feature Other samples except Second feature is the sample in the pixel domain Second feature Sample No. 2 does not include Feature information set Forming a sample Second feature information set is input into the detector, the second inspection result is obtained. If this second detection result is highly consistent with the markup information, that is, whether there is scene information of that pixel area does not have much impact, therefore, other positions other than the pixel area in the vicinity of the pixel area will be positions where there are unimportant pixels, which are likely to be unrelated to the lower object in the sample image or located on the edge of the lower object in the sample image, the content of effective information of other positions other than the pixel area in the vicinity of the pixel area is inevitably low.

[0026] Step S205: The first detection result, the second detection result, and the Labeling Based on information and instructions Target Mask Request. Specifically, for the first detection result and the second detection result corresponding to each pixel region, if the difference between the first detection result and the labeling information is greater than a first threshold value, and the difference between the second detection result and the labeling information is greater than a second threshold value, the pixel region is determined to belong to the core pixel region. The embodiment of the present application does not limit the difference, characterizing the difference between the measurement result and the notation information, and may be any calculation method, but the embodiment of the present application does not limit the difference calculation method, and many difference measurement methods may be used in the field of neural networks, as long as it can show the difference between the two pieces of information. Whether it is the first detection result or the second detection result, it is very different from the tagged information, but when we explain that the pixel region is very important, it is probably because the lower Object It is highly likely that the pixel belongs to a core pixel region that characterizes a region having partial information. The first and second thresholds are not limited in this specification, and may be used in the actual neural network. training The first threshold value may be set to be greater than the second threshold value.

[0027] If the degree of difference between the first detection result and the labeling information is greater than a first threshold and the degree of difference between the second detection result and the labeling information is equal to or smaller than a second threshold, the pixel region is determined to belong to a boundary pixel region. Specifically, the first detection result and Labeling If the difference between the first and second detection results is large, it indicates that the pixel area is very important and that this area is necessary. Labeling If the difference between the information is small, indicating that the area surrounding the pixel area is not very important and may or may not be present, then the pixel area is likely located on the boundary of an underlying object in the sample image. EXAMPLES

[0028] Step S301: For each pixel region, the difference between the corresponding sample first feature and the corresponding sample second feature is calculated to obtain difference information. This dissimilarity information is First Feature and sample Second feature The feature distance can be obtained by calculating the feature distance between the pixel region and the neighboring region around it. The feature distance is a kind of information distance and can be measured using the information distance measurement method. On the other hand, although the embodiment of this application is not limited to this, this difference degree information characterizes the information distance between the pixel region and its surrounding adjacent region. If the pixel does not jump significantly when the distance is close, the pixel region is likely to be located inside or outside the subordinate object in the sample image, and in any case, no intersection will occur. Conversely, this Pixels The region is more likely to be located on the edge of an underlying object.

[0029] When a degree of difference between the first detection result and the labeling information is greater than a first threshold and a degree of difference between the second detection result and the labeling information is equal to or smaller than a second threshold, the pixel region is determined to belong to a boundary pixel region: If the degree of difference between the first detection result and the labeling information is greater than a first threshold, the degree of difference between the second detection result and the labeling information is equal to or less than a second threshold, and the degree of difference information is greater than a third threshold, the pixel region is determined to belong to a boundary pixel region.

[0030] Step S206. Sample Target Mask and instructed Target Mask Depending on the difference with training The parameters of the extraction network and the mask generation network are adjusted until A method for expressing the difference between the sample target mask and the instruction target mask, a method for feedback adjustment parameters, and training The completion condition can refer to the prior art in the field of neural networks, and is not mentioned here. Note that the network configuration of the extraction network and the mask generation network can be designed based on, for example, a deep convolutional neural network, but this is not limited to the embodiment of the present application.

[0031] After training, in this embodiment, based on the aggregated information extractor, information in the first image can be extracted to obtain fusion feature information. Specifically, the aggregated information extractor further includes an aggregated network that performs the following operations after the first image is input into the corresponding aggregated information extractor: The process of inputting the first image into the extraction network and triggering the mask generation network to generate a target mask corresponding to the first image can be referred to above, and is performed by the extraction network and the mask generation network. For the part covered by the target mask in the first image, Consolidation Based on the network, depth feature extraction is performed to obtain the first target feature. In the embodiment of the present invention, the part considered to be covered by the target mask is the part carrying important and effective information, that is, the effective information Aggregation This sub-region is very similar to the region where the target sub-object is located in the first image, so we define the first sub-region as the important region. target Extraction of features is performed, but first target The feature is a pyramidal multi-scale structure. Consolidation The network can extract rich multi-scale information hierarchically, and the multi-scale information can be fused to obtain the first target features can be obtained.

Claims

1. obtaining scene information of text type for describing the image, the scene information including at least first object information characterizing the scene and second object information characterizing the associated target subordinate object in the scene, and an image of picture type; determining, based on the first object information, a model cluster including a segmentation model corresponding to each subordinate object in a scene characterized by the first object information; determining a target model corresponding to each target subordinate object in the model cluster based on the second object information; coarsely segmenting the image based on each of the target models to obtain first images corresponding to each of the target sub-objects; performing segmentation based on information aggregation associated with a corresponding target subordinate object for each of the first images to obtain a sub-segmentation result corresponding to each of the first images; - rendering a body image corresponding to said image, characterizing a fusion result between said scene information and said image based on said subdivision result; A deep learning-based multi-source heterogeneous data fusion method comprising:

2. The subdivision results include a mask matrix, and the step of performing a segmentation based on information aggregation associated with a corresponding target subordinate object for each of the first images to obtain a subdivision result corresponding to each of the first images includes: For each first image, input the first image to a corresponding aggregate information extractor, which is obtained by training based on a detector corresponding to a target sub-object to which the first image corresponds, and the aggregate information extractor performs aggregate information extraction on the target sub-object in the first image to obtain fusion feature information corresponding to the first image; inputting the fusion feature information into a divider corresponding to the target sub-image to which the first image corresponds, to obtain the mask matrix; 2. The method of claim 1, comprising:

3. Rendering a body image corresponding to the image, characterizing a fusion result of the scene information and the image based on the subdivision result, comprising: Rendering the body image based on the mask matrix and the image corresponding to each of the first images.

3. The method of claim 2 .

4. The aggregated information extractor is mainly composed of an extraction network and a mask generation network, and the mask generation network is used to generate a target mask for distinguishing between subordinate objects and non-subordinate objects in an image input to the aggregated information extractor, and the extraction network and the mask generation network are trained by the following method: acquiring a sample image and a detector corresponding to a type of subordinate object corresponding to the sample image, the sample image having labeling information characterizing positional information of the subordinate object within the sample image, the sample image including only a single type of subordinate object; The sample image is input to the extraction network, and the extraction network extracts, for each pixel region in the sample image, a sample first feature corresponding to the pixel region, extracts a related position feature corresponding to each relevant pixel position, and fuses the related position features to obtain a sample second feature corresponding to the pixel region, the related pixel positions being other pixel positions not belonging to the pixel region in the vicinity of where the pixel region is located; Input the sample first feature and the sample second feature corresponding to each pixel region into a mask generation network to obtain a sample target mask; For each pixel region, inputting other sample first features excluding the corresponding sample first feature into the detector to obtain a first detection result corresponding to the pixel region, inputting other sample second features excluding the corresponding sample second feature into the detector to obtain a second detection result corresponding to the pixel region, determining an instruction target mask based on each of the first detection results, each of the second detection results, and the labeling information; Adjusting parameters of the extraction network and the mask generation network according to differences between the sample target mask and the instruction target mask until training is complete.

4. The method according to claim 3 .

5. The step of determining an instruction target mask based on each of the first detection results, each of the second detection results, and the labeling information includes: determining that a pixel region belongs to a core pixel region when a degree of difference between the first detection result and the labeling information is greater than a first threshold value and a degree of difference between the second detection result and the labeling information is greater than a second threshold value for a first detection result and a second detection result corresponding to each pixel region; determining that the pixel region belongs to a boundary pixel region when a degree of difference between the first detection result and the labeling information is greater than a first threshold and a degree of difference between the second detection result and the labeling information is equal to or smaller than a second threshold; generating the indication target mask based on the core pixel region and the boundary pixel region; 5. The method of claim 4, comprising:

6. A multi-source heterogeneous data acquisition module for acquiring text type scene information for scene description of an image, the text type scene information including at least first object information characterizing a scene and second object information characterizing a target subordinate object associated within the scene, and an image of a picture type; a cluster determination module that determines, based on the first object information, a model cluster including a segmentation model corresponding to each subordinate object in a scene characterized by the first object information; a model determination module for determining a target model corresponding to each target subordinate object in the model cluster based on the second object information; a coarse segmentation module for coarsely segmenting the image based on each of the target models to obtain first images corresponding to each of the target sub-objects; a subdivision module for performing a segmentation based on information aggregation associated with a corresponding target sub-object for each of the first images to obtain a subdivision result corresponding to each of the first images; a body image rendering module for rendering a body image corresponding to the image, the body image characterizing a fusion result of the scene information and the image based on the subdivision result; A deep learning-based multi-source heterogeneous data fusion device comprising:

7. The subdivision result includes a mask matrix, The subdivision module comprises: For each first image, input the first image into a corresponding aggregate information extractor, which is obtained by training based on a detector corresponding to a target sub-object to which the first image corresponds, and the aggregate information extractor performs aggregate information extraction on the target sub-object in the first image to obtain fusion feature information corresponding to the first image; The fusion feature information is input to a divider corresponding to the target sub-image to which the first image corresponds, and the mask matrix is ​​obtained.

7. The apparatus according to claim 6 .

8. The body image rendering module includes: Rendering the body image based on the mask matrix and the images corresponding to each of the first images.

8. The device according to claim 7 .

9. At least one instruction or at least one program is stored which is loaded and executed by a processor to implement the deep learning based multi-source heterogeneous data fusion method according to any one of claims 1 to 5. A computer-readable storage medium comprising:

10. 1. An electronic device comprising at least one processor and a memory communicatively coupled to the at least one processor, The memory stores instructions executable by the at least one processor, and the at least one processor executes the instructions stored in the memory to realize one of the deep learning-based multi-source heterogeneous data fusion methods according to any one of claims 1 to 5.

1. An electronic device comprising:

Images