A cross-modal causal reasoning method, device, storage medium and electronic device

Abstract

The embodiments of the present application provide a cross-modal causal reasoning method, device, storage medium and electronic device, which relate to the field of causal reasoning technology. The method includes: performing intervention operations on the original sample to obtain an intervention sample; using a pre-trained model to extract the features of the original sample, the intervention sample and the target sample respectively; weighting the confusion factor and the feature of the target sample to obtain the weighted feature of the confusion factor; performing feature splicing and feature fusion processing based on the features of the original sample, the features of the intervention sample and the weighted feature of the confusion factor to calculate the average causal effect value of the target sample; obtaining the average causal effect value of multiple target samples, and confirming the target sample that has a causal relationship with the original sample based on the average causal effect value of the multiple target samples. The technical solution of the present application improves the accuracy of interactive representation and the reliability of causal evaluation.

Description

Technical Field

[0001] The present application relates to the technical field of causal reasoning, and in particular, to a cross-modal causal reasoning method, a cross-modal causal reasoning device, a computer-readable storage medium, and an electronic device. Background Art

[0002] In recent years, with the increasing number of model network layers in deep learning, the computer's ability to understand images and texts has made great progress. For example, it has been able to achieve relatively high accuracy in various cross-modal reasoning tasks of 2D vision-language. However, in the process of reasoning, the computer still generates answers based on the results of statistical analysis, often ignoring the causal mechanism contained therein. Although there are some tasks and methods to explore causal mechanisms in single-modal fields, such as computer vision and natural language processing, work on cross-modal causal reasoning of vision-language is still in its infancy. The cross-modal causal reasoning task requires the computer to fully understand the semantic information of images and texts and then determine whether there is causality between the two, that is, the occurrence of the former leads to the occurrence of the latter. Cross-modal causal reasoning involves both the processing of multimodal information and the identification and determination of causality, which is very challenging. First, information from different modalities, namely image information and text information, needs to be aligned before feature fusion and interaction can be performed. Second, the information contained in images and texts is relatively complex, and causality often exists in the deep layer, which is not easy to identify and determine.

[0003] In order to solve the problems related to cross-modal causal reasoning, some existing methods focus on enhancing feature extraction and interaction. When extracting features, an encoder with a deeper network layer is used, and then the cross-attention mechanism is used to fuse the features and directly perform reasoning. However, this method completely ignores the mining of causal information, which is somewhat contrary to the starting point of the task setting itself, that is, exploring causality. At the same time, the model is also poor in interpretability. Therefore, some subsequent work began to introduce causal mechanisms into the model. The biggest impact on causal identification is the existence of confusion factors, which will interfere with the final results. Therefore, in order to eliminate the interference of confusion factors, this type of model incorporates all predefined confusion factors into the calculation and discrimination process. However, in this type of method, the pre-definition of confusion factors is very limited, and the feature selection of each confusion factor is also a difficult problem. At the same time, they do not intervene in the generation of instances to produce the final causal score. Therefore, cross-modal causal reasoning must consider not only how to extract more fine-grained features and the interaction between features, but also the mining of causal mechanisms, which also makes cross-modal causal reasoning very challenging. Summary of the invention

[0004] Embodiments of the present application provide a cross-modal causal reasoning method, a cross-modal causal reasoning device, a computer-readable storage medium, and an electronic device to solve technical problems existing in the prior art.

[0005] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0006] According to a first aspect of an embodiment of the present application, a cross-modal causal reasoning method is provided, comprising:

[0007] Performing an intervention operation on the original sample to obtain an intervention sample, wherein the original sample includes an image sample and a text sample;

[0008] Using the pre-trained model to extract features of the original sample, the intervention sample, and the target sample respectively;

[0009] The confusion factor between the original sample and the target sample is weighted with the feature of the target sample to obtain a confusion factor weighted feature;

[0010] Performing feature splicing processing based on the features of the original sample, the features of the intervention sample, and the weighted features of the confounding factor, and performing feature fusion processing based on the feature splicing processing result to calculate the average causal effect value of the target sample;

[0011] The average causal effect value of the multiple target samples is obtained, and the target samples having a causal relationship with the original samples are identified based on the average causal effect value of the multiple target samples.

[0012] In some embodiments of the present application, based on the above scheme, performing an intervention operation on the original sample to obtain an intervention sample includes:

[0013] Using a target detector pre-trained on a data set, the instances in the image sample are replaced with other instances of the same category to obtain an intervention image sample;

[0014] The original text content in the text sample is modified using a controllable universal text counterfactual generator to obtain an intervention text sample.

[0015] In some embodiments of the present application, based on the above scheme, the weighting of the confusion factor between the original sample and the target sample and the feature of the target sample includes:

[0016] First, the feature of each target sample is inner-producted with the confusion factor, and then the confusion factor is weighted by a softmax function.

[0017] In some embodiments of the present application, based on the aforementioned solution, the feature splicing process based on the feature of the original sample, the feature of the intervention sample and the weighted feature of the confusion factor includes:

[0018] Concatenate the confusion factor weighted feature with the feature of the original sample to obtain a first concatenated feature;

[0019] The confounding factor weighted feature is concatenated with the feature of the intervention sample to obtain a second concatenated feature.

[0020] In some embodiments of the present application, based on the above-mentioned solution, the feature fusion processing is performed based on the feature splicing processing result to calculate the average causal effect value of the target sample, including:

[0021] Use a cross attention module to perform feature interaction between the first concatenated feature and the feature of the target sample, and then calculate a first causal score through a fully connected layer;

[0022] Use a cross attention module to perform feature interaction between the second concatenated feature and the feature of the target sample, and then calculate a second causal score through a fully connected layer;

[0023] An average causal effect value is calculated based on the first causal score and the second causal score.

[0024] In some embodiments of the present application, based on the aforementioned solution, the step of confirming a target sample having a causal relationship with the original sample based on the average causal effect value of multiple target samples includes:

[0025] The target sample corresponding to the maximum value among the average causal effect values ​​of the multiple target samples is confirmed as the target sample having a causal relationship with the original sample.

[0026] In some embodiments of the present application, based on the above solution, after confirming the target sample having a causal relationship with the original sample based on the average causal effect value of multiple target samples, the method further includes:

[0027] The loss of the average causal effect value of the target sample that has a causal relationship with the original sample is calculated using formula (1);

[0028] L margin =max(0, my score corrct +y·score incorrect ); (1)

[0029] Among them, m is a hyperparameter that controls the margin, indicating the tolerance of the classification boundary; y is the true label of the target sample; score correctThe score represents the average causal effect value of the target sample that has a causal relationship with the original sample. incorrect Represents the average causal effect value of the target sample that has no causal relationship with the original sample.

[0030] According to a second aspect of an embodiment of the present application, a cross-modal causal reasoning device is provided, comprising:

[0031] An intervention unit, used for performing an intervention operation on an original sample to obtain an intervention sample, wherein the original sample includes an image sample and a text sample;

[0032] A feature extraction unit, used to respectively extract features of the original sample, the intervention sample, and the target sample using a pre-trained model;

[0033] A weighted processing unit, used for weighting the confusion factor between the original sample and the target sample and the feature of the target sample to obtain a confusion factor weighted feature;

[0034] A feature splicing unit, used for performing feature splicing processing based on the features of the original sample, the features of the intervention sample and the weighted features of the confounding factor;

[0035] A feature fusion unit, used for performing feature fusion processing based on the feature splicing processing result, and calculating the average causal effect value of the target sample;

[0036] The confirmation unit is used to obtain an average causal effect value of multiple target samples, and confirm the target sample that has a causal relationship with the original sample based on the average causal effect value of the multiple target samples.

[0037] According to a third aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which computer instructions are stored. When the computer instructions are executed on a computer, the computer executes the method according to the first aspect above.

[0038] According to a fourth aspect of the embodiments of the present application, there is provided an electronic device, including: a memory and a processor;

[0039] The memory is used to store computer instructions;

[0040] The processor is used to call the computer instructions stored in the memory so that the electronic device executes the method as described in the first aspect above.

[0041] The technical solution of the present application improves the accuracy of interactive representation and eliminates the interference of confounding factors on causal evaluation through intervention operations, confounding factor weighting processing, feature splicing and feature fusion processing.

[0042] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0044] Figure 1 A schematic diagram of a process of a cross-modal causal reasoning method according to an embodiment of the present application is shown;

[0045] Figure 2 A schematic diagram showing a cross-modal causal reasoning method implemented by using a specific module according to an embodiment of the present application is shown;

[0046] Figure 3 A first structural cause-effect diagram according to an embodiment of the present application is shown;

[0047] Figure 4 A second structural cause-effect diagram according to an embodiment of the present application is shown;

[0048] Figure 5 A block diagram of a cross-modal causal reasoning device according to an embodiment of the present application is shown;

[0049] Figure 6 A structural block diagram of an electronic device according to an embodiment of the present application is shown;

[0050] Figure 7 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0051] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be more comprehensive and complete and fully convey the concept of the example embodiments to those skilled in the art.

[0052] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0053] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0054] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0055] It should be noted that the “plurality” mentioned in this article refers to two or more.

[0056] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the objects used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those shown or described.

[0057] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all the embodiments. Based on the embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] Some embodiments of the present application will be described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments may be combined with each other.

[0059] In order to solve the technical problems existing in the prior art, the embodiment of the present application proposes a cross-modal causal reasoning method. Specifically, in order to solve the information representation and interaction problems caused by different modal sources, the technical solution of the present application uses a pre-trained feature encoder to extract features, and then uses an attention mechanism to perform feature fusion to improve the accuracy of the features.

[0060] In order to solve the problem of interference of confusion factors on the final causal discrimination, the technical solution of the present application adopts an intervention generation method to replace the sample, and then introduces a learnable confusion factor dictionary, and finally calculates the transformation of the sample before and after the intervention, thereby improving the reliability and accuracy of causal discrimination.

[0061] See also Figure 1 , showing a flow chart of a cross-modal causal reasoning method according to an embodiment of the present application.

[0062] like Figure 1 As shown, a cross-modal causal reasoning method is presented, including steps S100 to S600.

[0063] Step S100, performing an intervention operation on the original sample to obtain an intervention sample, wherein the original sample includes an image sample and a text sample.

[0064] It can be understood that, when an intervention operation is performed on the original sample, the obtained intervention sample is a counterfactual representation of the corresponding original sample.

[0065] In some feasible embodiments, based on the above scheme, performing an intervention operation on the original sample to obtain an intervention sample includes:

[0066] Using a target detector pre-trained on a data set, the instances in the image sample are replaced with other instances of the same category to obtain an intervention image sample;

[0067] The original text content in the text sample is modified using a controllable universal text counterfactual generator to obtain an intervention text sample.

[0068] It should be noted that the process of modifying the original text content mainly includes semantic negation, word deletion and semantic replacement of the original text content.

[0069] Continue to refer Figure 1 , step S200, using the pre-trained model to extract the features of the original sample, the intervention sample and the target sample respectively.

[0070] It can be understood that in the embodiments of the present application, by using a pre-trained model to extract features from images or texts, a feature sequence with the same dimension can be obtained, and the problem of alignment between image features and text features can be solved.

[0071] It should be noted that in the embodiments of the present application, the original samples, intervention samples and target samples should all include image samples and text samples, so that the dimensions of their respective feature sequences extracted using the pre-trained model are the same.

[0072] Continue to refer Figure 1 , step S300, weighting the confusion factor between the original sample and the target sample and the feature of the target sample to obtain a confusion factor weighted feature.

[0073] It can be understood that the addition of confounding factors in this step can eliminate the interference of confounding factors on causal assessment.

[0074] In some feasible embodiments, based on the above solution, the weighting of the confusion factor between the original sample and the target sample and the feature of the target sample includes:

[0075] First, the feature of each target sample is inner-producted with the confusion factor, and then the confusion factor is weighted by a softmax function.

[0076] It should be noted that since confusion factors are often unobservable variables, especially in the cross-modal setting, their definition is more complicated. Therefore, in some embodiments, all confusion factors are defined as a learnable confusion factor dictionary. A total of 60 confusion factors are set, and the feature dimension of each confusion factor is 512 dimensions.

[0077] Continue to refer Figure 1 , step S400, performing feature splicing processing based on the features of the original sample, the features of the intervention sample and the weighted features of the confounding factor.

[0078] In some feasible embodiments, based on the above scheme, the feature splicing process based on the feature of the original sample, the feature of the intervention sample and the weighted feature of the confusion factor includes:

[0079] Concatenate the confusion factor weighted feature with the feature of the original sample to obtain a first concatenated feature;

[0080] The confounding factor weighted feature is concatenated with the feature of the intervention sample to obtain a second concatenated feature.

[0081] Continue to refer Figure 1 , step S500, performing feature fusion processing based on the feature splicing processing result, and calculating the average causal effect value of the target sample.

[0082] In some feasible embodiments, based on the above scheme, the feature fusion processing is performed based on the feature splicing processing result to calculate the average causal effect value of the target sample, including:

[0083] Use a cross attention module to perform feature interaction between the first concatenated feature and the feature of the target sample, and then calculate a first causal score through a fully connected layer;

[0084] Use a cross attention module to perform feature interaction between the second concatenated feature and the feature of the target sample, and then calculate a second causal score through a fully connected layer;

[0085] An average causal effect value is calculated based on the first causal score and the second causal score.

[0086] Continue to refer Figure 1 , step S600, obtaining the average causal effect value of multiple target samples, and confirming the target sample that has a causal relationship with the original sample based on the average causal effect value of the multiple target samples.

[0087] It should be noted that in the embodiment of the present application, steps S100 to 500 can obtain the average causal effect value of a single target sample. Therefore, by repeating steps S100 to 500 multiple times and inputting different target samples, the average causal effect values ​​of multiple target samples can be obtained.

[0088] In some feasible embodiments, based on the above scheme, the step of confirming the target sample having a causal relationship with the original sample based on the average causal effect value of the plurality of target samples includes:

[0089] The target sample corresponding to the maximum value among the average causal effect values ​​of the multiple target samples is confirmed as the target sample having a causal relationship with the original sample.

[0090] It can be understood that, among all target samples, the target sample with the largest average causal effect value is the sample that is finally inferred to have a causal relationship with the original sample.

[0091] In some feasible embodiments, based on the above solution, after confirming the target sample having a causal relationship with the original sample based on the average causal effect value of multiple target samples, the method further includes:

[0092] The loss of the average causal effect value of the target sample that has a causal relationship with the original sample is calculated using formula (1);

[0093] L margin =max(0, my score correct +y·scoreincorrect); (1)

[0094] Among them, m is a hyperparameter that controls the margin, indicating the tolerance of the classification boundary; y is the true label of the target sample; score correct The score represents the average causal effect value of the target sample that has a causal relationship with the original sample. incorrect Represents the average causal effect value of the target sample that has no causal relationship with the original sample.

[0095] In summary, the technical solution of the present application proposes a feature fusion scheme based on attention interaction, which improves the accuracy of interaction representation; proposes a causal discrimination scheme with causal mechanism as the core, intervenes in entities, and improves the reliability of causal evaluation by eliminating confounding factors.

[0096] Specifically, an example of implementing the method of the present application using a specific module is provided below.

[0097] See also Figure 2 , showing a schematic diagram of a cross-modal causal reasoning method implemented using specific modules according to an embodiment of the present application.

[0098] like Figure 2 As shown in the figure, we first perform intervention operations on the original premise X to generate the corresponding counterfactual representation. The original premise X can be either an image or a text. For the intervention operation on images, we use the object detector YOLO-v7 pre-trained on the COCO dataset to replace the instances detected in the image with other instances of the same category. For the intervention operation on text, we use the controllable general text counterfactual generator Polyjuice to modify the original text content, mainly including semantic negation, word deletion and semantic replacement.

[0099] In order to solve the problem of alignment between image features and text features, the pre-trained CLIP encoder is used to extract features from images or texts to obtain feature sequences with a dimension of 512.

[0100] The CLIP model has obtained better image encoders and text encoders through contrastive learning on a large number of image-text pairs, so that their features can coexist in a shared embedding space. Therefore, during the training process, the CLIP model has frozen parameters and will not participate in training.

[0101] like Figure 2 As shown, the original premise X, the post-intervention premise X′ obtained after the intervention operation, and the candidate item Y (i.e., the target sample, referred to as the candidate item in this example) are respectively extracted by the pre-trained CLIP encoder to obtain the original features, post-intervention features, and candidate item features.

[0102] Since the process of determining the causal relationship between the original premise X and the candidate Y is often interfered by the confusion factor Z, because the confusion factor Z affects both X and Y, there is a backdoor path: X←Z→Y, which makes the causal relationship from X to Y directly evaluated inaccurate, such as Figure 3 shown.

[0103] Therefore, the backdoor adjustment method should be used to eliminate the interference of the confusion factor Z as much as possible, such as Figure 4 So the average causal effect of X to Y estimated using backdoor adjustment is expressed as follows:

[0104] ATE=P(Y|do(X=x))-P(Y|do(X=x′))=∑ z P(Y||X=x, Z=z)P(Z=z)-∑ z P(Y|X=x′, Z=z)P(Z=z);

[0105] Since the confounding factor is often an unobservable variable, especially in the cross-modal setting, its definition is more complicated. Therefore, we define all the confounding factors as a learnable confounding factor dictionary. A total of 60 confounding factors are set, and the feature dimension of each confounding factor is 512. Different confounding factors have different effects on the final result, especially related to Y. Therefore, the inner product operation of Z and Y is performed, and then different weights are given to each confounding factor Z through the softmax operation. Finally, the weighted confounding factor feature sequence is spliced ​​to the original feature and the post-intervention feature, such as Figure 2 As shown. Therefore, the original evaluation formula of the average causal effect can be transformed into:

[0106] ATE=P(Y|concat(x, softmax(Z☉Y)Z))-P(Y|concat(x′, softmax(Z☉Y)Z)).

[0107] like Figure 2 As shown in the figure, the original features concatenated with the confounding factor feature sequence and the candidate item features are fused. Specifically, the cross attention module is used for feature interaction, and then a fully connected layer is used to calculate the causal score. The causal score of the post-intervention feature concatenated with the confounding factor feature sequence is calculated according to the same process. Finally, the two causal scores are subtracted, and then a tanh activation function is used to constrain its value in the range of -1 to 1 to obtain the final average causal effect value.

[0108] In this example, two target samples Y1 and Y2 are input, and their average causal effect values ​​are calculated according to the above process to obtain ATE1 and ATE2. The option with the highest average causal effect value between the two is selected as the output answer, which is output as the correct target sample.

[0109] The following describes an apparatus embodiment of the present application, which can be used to execute a cross-modal causal reasoning method in the above-mentioned embodiment of the present application. For details not disclosed in the apparatus embodiment of the present application, please refer to the above-mentioned method embodiment of the present application.

[0110] Reference Figure 5 As shown, a cross-modal causal reasoning device 500 according to an embodiment of the present application includes: an intervention unit 501, a feature extraction unit 502, a weighted processing unit 503, a feature splicing unit 504 and a feature fusion unit 505.

[0111] Among them, the intervention unit 501 is used to perform intervention operations on the original sample to obtain the intervention sample, and the original sample includes an image sample and a text sample; the feature extraction unit 502 is used to use the pre-trained model to respectively extract the features of the original sample, the intervention sample and the target sample; the weighted processing unit 503 is used to weight the confusion factor between the original sample and the target sample and the feature of the target sample to obtain the confusion factor weighted feature; the feature splicing unit 504 is used to perform feature splicing processing based on the feature of the original sample, the feature of the intervention sample and the confusion factor weighted feature; the feature fusion unit 505 is used to perform feature fusion processing based on the feature splicing processing result, and calculate the average causal effect value of the target sample; the confirmation unit 506 is used to obtain the average causal effect value of multiple target samples, and confirm the target sample that has a causal relationship with the original sample based on the average causal effect value of the multiple target samples.

[0112] like Figure 6 As shown, an embodiment of the present application also provides an electronic device 600, including a memory 610, a processor 620, and a computer program 611 stored in the memory 610 and executable on the processor, and when the processor 620 executes the computer program 611, the steps of any one of the above-mentioned cross-modal causal reasoning methods are implemented.

[0113] Since the electronic device introduced in this embodiment is a device used to implement a cross-modal causal reasoning device in the embodiment of the present application, based on the method introduced in the embodiment of the present application, technical personnel in this field can understand the specific implementation of the electronic device of this embodiment and its various variations. Therefore, how the electronic device implements the method in the embodiment of the present application is not introduced in detail here. As long as the equipment used by technical personnel in this field to implement the method in the embodiment of the present application is within the scope of protection of this application.

[0114] During the specific implementation process, when the computer program 611 is executed by a processor, any implementation method in the embodiments corresponding to the first aspect can be implemented.

[0115] Figure 7 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown.

[0116] It should be noted that Figure 7 The computer system 700 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0117] like Figure 7 As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 702 or the program loaded from the storage part 708 to the random access memory (RAM) 703, such as executing the method described in the above embodiment. In the RAM 703, various programs and data required for system operation are also stored. The CPU 701, ROM 702 and RAM 703 are connected to each other through a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0118] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed so that a computer program read therefrom is installed into the storage section 708 as needed.

[0119] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 709, and / or installed from a removable medium 711. When the computer program is executed by a central processing unit (CPU) 701, various functions defined in the system of the present application are executed.

[0120] It should be noted that the computer-readable medium shown in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, - but not limited to - an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by an instruction execution system, device or device or used in combination with it. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0121] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0122] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0123] As another aspect, the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes a cross-modal causal reasoning method described in the above embodiment.

[0124] As another aspect, the present application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements a cross-modal causal reasoning method described in the above embodiment.

[0125] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0126] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the implementation methods of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation methods of the present application.

[0127] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary techniques in the art that are not disclosed in the present application. It should be understood that the present application is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A cross-modal causal reasoning method, characterized in that: include: Performing an intervention operation on the original sample to obtain an intervention sample, wherein the original sample includes an image sample and a text sample; Using the pre-trained model, respectively extracting features of the original sample, the intervention sample, and the target sample; The confusion factor between the original sample and the target sample is weighted with the feature of the target sample to obtain a confusion factor weighted feature; Performing feature splicing processing based on the features of the original sample, the features of the intervention sample, and the weighted features of the confounding factor, and performing feature fusion processing based on the feature splicing processing result to calculate the average causal effect value of the target sample; Obtaining an average causal effect value of multiple target samples, and confirming a target sample that has a causal relationship with the original sample based on the average causal effect value of the multiple target samples; The intervening operation is performed on the original sample to obtain the intervening sample, including: Using a target detector pre-trained on a data set, the instances in the image sample are replaced with other instances of the same category to obtain an intervention image sample; Using a controllable universal text counterfactual generator to modify the original text content in the text sample to obtain an intervention text sample; The weighting of the confusion factor between the original sample and the target sample and the feature of the target sample includes: First, the feature of each target sample is inner-producted with the confusion factor, and then the confusion factor is weighted by a softmax function; The feature splicing process based on the feature of the original sample, the feature of the intervention sample and the weighted feature of the confounding factor includes: Concatenate the confusion factor weighted feature with the feature of the original sample to obtain a first concatenated feature; Concatenating the confounding factor weighted feature with the feature of the intervention sample to obtain a second concatenated feature; The feature fusion processing is performed based on the feature splicing processing result to calculate the average causal effect value of the target sample, including: Use a cross attention module to perform feature interaction between the first concatenated feature and the feature of the target sample, and then calculate a first causal score through a fully connected layer; Use a cross attention module to perform feature interaction between the second concatenated feature and the feature of the target sample, and then calculate a second causal score through a fully connected layer; An average causal effect value is calculated based on the first causal score and the second causal score.

2. The method according to claim 1, characterized in that The step of confirming a target sample having a causal relationship with the original sample based on the average causal effect value of the plurality of target samples includes: The target sample corresponding to the maximum value among the average causal effect values ​​of the multiple target samples is confirmed as the target sample having a causal relationship with the original sample.

3. The method according to claim 2, characterized in that After confirming the target sample having a causal relationship with the original sample based on the average causal effect value of the plurality of target samples, the method further includes: The loss of the average causal effect value of the target sample that has a causal relationship with the original sample is calculated using formula (1); L margin =max(0,m-y·score correct +y·score incorrect ); (1) Among them, m is a hyperparameter that controls the margin, indicating the tolerance of the classification boundary; y is the true label of the target sample; score correct The score represents the average causal effect value of the target sample that has a causal relationship with the original sample. incorrect Represents the average causal effect value of the target sample that has no causal relationship with the original sample.

4. A cross-modal causal reasoning device, applied to the method according to any one of claims 1 to 3, characterized in that: include: An intervention unit, used for performing an intervention operation on an original sample to obtain an intervention sample, wherein the original sample includes an image sample and a text sample; A feature extraction unit, used to respectively extract features of the original sample, the intervention sample, and the target sample using a pre-trained model; A weighted processing unit, used for weighting the confusion factor between the original sample and the target sample and the feature of the target sample to obtain a confusion factor weighted feature; A feature splicing unit, used for performing feature splicing processing based on the features of the original sample, the features of the intervention sample and the weighted features of the confounding factor; A feature fusion unit, used for performing feature fusion processing based on the feature splicing processing result, and calculating the average causal effect value of the target sample; The confirmation unit is used to obtain an average causal effect value of multiple target samples, and confirm the target sample that has a causal relationship with the original sample based on the average causal effect value of the multiple target samples.

5. A computer-readable storage medium, characterized in that: The storage medium stores computer instructions, and when the computer instructions are executed on a computer, the computer is enabled to execute the method according to any one of claims 1 to 3.

6. An electronic device, characterized in that: Including: memory and processor; The memory is used to store computer instructions; The processor is used to call the computer instructions stored in the memory so that the electronic device executes the method according to any one of claims 1 to 3.

Images