Cross-modal association reasoning method, device and system based on small sample learning

Abstract

The invention relates to a cross-modal association reasoning method, device and system based on small sample learning, and relates to the technical field. The method comprises the following steps: acquiring multi-source heterogeneous data, pre-processing the multi-source heterogeneous data, calculating respective characteristic values of the pre-processed multi-source heterogeneous data by utilizing a characteristic vector representation model, and inputting the respective characteristic values of the multi-source heterogeneous data into a trained association reasoning model, the association reasoning model is obtained by performing association reasoning based on respective characteristic values of the multi-source heterogeneous data, the association reasoning model is obtained by training a meta-learning model by adopting a meta-learning method, and an association reasoning result of the multi-source heterogeneous data is determined by utilizing the association probability vector. According to the scheme, cross-modal reasoning of heterogeneous data can be realized in a small sample data environment.

Description

Technical Field

[0001] The present disclosure relates to the technical field of multi-source heterogeneous data processing, and in particular to a cross-modal association reasoning method, device and system based on small sample learning. Background Art

[0002] In the Industrial Internet environment, widely distributed industrial equipment, sensors, and other data sources generate a large number of heterogeneous data sources. These data sources typically include different types of modal data, such as time series, image data, and device status signals. Complex relationships exist between these data sources, and effective associative reasoning on this data can be used to optimize equipment monitoring, fault prediction, quality inspection, and process optimization in industrial production. However, due to the differences in heterogeneous data sources and the limitations of small sample sizes, achieving associative reasoning between cross-modal data sources in small sample sizes has become a difficult problem in current Industrial Internet data analysis. Traditional associative reasoning methods require a large number of training samples to ensure accuracy, but in practical applications of the Industrial Internet, the number of labeled samples is limited. Therefore, efficient cross-modal reasoning in small sample sizes remains a technical challenge. Summary of the Invention

[0003] To overcome the problems existing in the related art, the present disclosure provides a cross-modal association reasoning method, device and system based on small sample learning.

[0004] According to a first aspect of an embodiment of the present disclosure, a cross-modal association reasoning method based on small sample learning is provided, comprising:

[0005] Acquiring multi-source heterogeneous data; the multi-source heterogeneous data includes any one or more of spatial information, temporal information, semantic information, and perceptual information;

[0006] Preprocessing the multi-source heterogeneous data;

[0007] The eigenvalues of each preprocessed multi-source heterogeneous data are calculated using the eigenvector representation model;

[0008] Inputting the characteristic values of each of the multi-source heterogeneous data into a trained association reasoning model to obtain an association probability vector obtained by the association reasoning model based on the characteristic values of each of the multi-source heterogeneous data; the association reasoning model is obtained by training the meta-learning model using a meta-learning method;

[0009] The association probability vector is used to determine the association reasoning result of the multi-source heterogeneous data.

[0010] In some embodiments of the present disclosure, the preprocessing of the multi-source heterogeneous data includes:

[0011] Performing standardization on the multi-source heterogeneous data to obtain standardized multi-source heterogeneous data;

[0012] Performing normalization on the standardized multi-source heterogeneous data to obtain normalized multi-source heterogeneous data;

[0013] Performing temporal and spatial alignment processing on the normalized multi-source heterogeneous data to obtain aligned multi-source heterogeneous data;

[0014] Outlier removal and data enhancement are performed on the aligned multi-source heterogeneous data to obtain the pre-processed multi-source heterogeneous data.

[0015] In some embodiments of the present disclosure, the association reasoning model is trained using the following steps:

[0016] Acquire multi-source heterogeneous sample data, and preprocess the multi-source heterogeneous sample data;

[0017] The eigenvector representation model is used to calculate the sample eigenvalues of the preprocessed multi-source heterogeneous sample data;

[0018] Generate nodes corresponding to each multi-source heterogeneous sample data;

[0019] Based on the sample feature values corresponding to each node, the association relationship between nodes is constructed to obtain an association relationship set;

[0020] The association relationship set is utilized to train the meta-learning model using a meta-learning method to obtain the association reasoning model.

[0021] In some embodiments of the present disclosure, after constructing the association relationship between nodes based on the sample feature value corresponding to each node to obtain the association relationship set, the following steps are further included:

[0022] For each node, a graph convolutional neural network is used to aggregate the node and the nodes adjacent to the node to obtain the neighbor information of the node, and the sample feature value of the node is updated based on the neighbor information.

[0023] In some embodiments of the present disclosure, after constructing the association relationship between nodes based on the sample feature value corresponding to each node to obtain the association relationship set, the following steps are further included:

[0024] For each node, a graph attention network is used to calculate the weight value between the node and the nodes adjacent to the node;

[0025] The method of using the association relationship set and adopting a meta-learning method to train the meta-learning model to obtain the association reasoning model includes:

[0026] Initializing model parameters of the meta-learning model using the weight values and the association relationship set;

[0027] The association relationship set is utilized to train the meta-learning model using a meta-learning method to obtain the association reasoning model.

[0028] In some embodiments of the present disclosure, the step of calculating the weight value between the node and the nodes adjacent to the node using a graph attention network includes:

[0029] The weight value is calculated using the following formula:

[0030]

[0031] in, is the node representation of node i in the l+1 layer, σ is the activation function, α ij is the attention weight between node i and its neighbor node j, W is the weight matrix, Ν(i) is the set of neighbor nodes of node i, h j is the representation of neighbor node j.

[0032] According to a second aspect of an embodiment of the present disclosure, a cross-modal association reasoning device based on small sample learning is provided, comprising:

[0033] An acquisition unit, configured to acquire multi-source heterogeneous data; the multi-source heterogeneous data includes any one or more of spatial information, temporal information, semantic information, and perceptual information;

[0034] A preprocessing unit, configured to preprocess the multi-source heterogeneous data;

[0035] A calculation unit, used to calculate the eigenvalues of each of the preprocessed multi-source heterogeneous data using a eigenvector representation model;

[0036] an association reasoning unit, configured to input the characteristic values of each of the multi-source heterogeneous data into a trained association reasoning model, and obtain an association probability vector obtained by the association reasoning model through association reasoning based on the characteristic values of each of the multi-source heterogeneous data; the association reasoning model is obtained by training a meta-learning model using a meta-learning method;

[0037] A determination unit is used to determine the association reasoning result of the multi-source heterogeneous data using the association probability vector.

[0038] According to a third aspect of an embodiment of the present disclosure, an electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method described in any one of the first aspects is implemented.

[0039] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method according to any one of the first aspects is implemented.

[0040] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements the method as described in any one of the first aspects when executed by a processor.

[0041] The technical solution provided by the embodiments of the present disclosure may include the following beneficial effects: by acquiring multi-source heterogeneous data, pre-processing the multi-source heterogeneous data, using a feature vector representation model to calculate the eigenvalues of each of the pre-processed multi-source heterogeneous data, inputting the eigenvalues of each of the multi-source heterogeneous data into a trained association reasoning model, and obtaining an association probability vector obtained by the association reasoning model based on the eigenvalues of each of the multi-source heterogeneous data. The association reasoning model is obtained by training the meta-learning model using a meta-learning method, and the association probability vector is used to determine the association reasoning result of the multi-source heterogeneous data, thereby realizing rapid mapping adaptation between cross-modal data, and being able to realize cross-modal reasoning of heterogeneous data in a small sample data environment.

[0042] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0044] Figure 1 The present invention is a flowchart of a cross-modal association reasoning method based on small sample learning according to an exemplary embodiment.

[0045] Figure 2 FIG. 4 is a schematic diagram of a feature vector representation model according to an embodiment of the present disclosure.

[0046] Figure 3 The present invention is a flowchart of cross-modal association reasoning based on small sample learning according to an exemplary embodiment.

[0047] Figure 4 The present invention is a block diagram of a cross-modal association reasoning device based on small sample learning according to an exemplary embodiment.

[0048] Figure 5 The present invention is a block diagram of an apparatus for a cross-modal association inference method based on small sample learning according to an exemplary embodiment. DETAILED DESCRIPTION

[0049] Exemplary embodiments will be described in detail herein, examples of which are illustrated in the accompanying drawings. In the following description, when referring to the drawings, like numbers in different figures represent like or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present invention. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present invention, as detailed in the appended claims.

[0050] The terms used in the present disclosure are only for the purpose of describing specific embodiments and are not intended to limit the present disclosure. The singular forms "a", "an" and "the" used in the present disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0051] It should be understood that although the terms first, second, third, etc. may be used to describe various information in the embodiments of the present disclosure, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of the embodiments of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0052] Furthermore, the various forms of processes shown in the embodiments of this disclosure may be used to reorder, add, or delete steps. For example, the steps described in this application may be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0053] In the Industrial Internet environment, widely distributed industrial equipment, sensors, and other data sources generate a large number of heterogeneous data sources. These data sources typically include different types of modal data, such as time series, image data, and device status signals. Complex relationships exist between these data sources, and effective associative reasoning on this data can be used to optimize equipment monitoring, fault prediction, quality inspection, and process optimization in industrial production. However, due to the differences in heterogeneous data sources and the limitations of small sample sizes, achieving associative reasoning between cross-modal data sources in small sample sizes has become a difficult problem in current Industrial Internet data analysis. Traditional associative reasoning methods require a large number of training samples to ensure accuracy, but in practical applications of the Industrial Internet, the number of labeled samples is limited. Therefore, efficient cross-modal reasoning in small sample sizes remains a technical challenge.

[0054] In order to solve the above problems, the present disclosure provides a cross-modal association reasoning method, device and system based on small sample learning, which obtains multi-source heterogeneous data, pre-processes the multi-source heterogeneous data, uses a feature vector representation model to calculate the eigenvalues of each of the pre-processed multi-source heterogeneous data, and inputs the eigenvalues of each of the multi-source heterogeneous data into the trained association reasoning model to obtain an association probability vector obtained by the association reasoning model based on the eigenvalues of each of the multi-source heterogeneous data. The association reasoning model is obtained by training the meta-learning model using a meta-learning method, and the association probability vector is used to determine the association reasoning result of the multi-source heterogeneous data, thereby realizing rapid mapping adaptation between cross-modal data and realizing cross-modal reasoning of heterogeneous data in a small sample data environment.

[0055] Figure 1 is a flowchart of a cross-modal association reasoning method based on small sample learning according to an exemplary embodiment. Figure 1 and Figure 2 As shown, it should be noted that the cross-modal association reasoning method based on small sample learning in the embodiment of the present disclosure is applied to the cross-modal association reasoning device based on small sample learning. Figure 1 As shown, the method may include the following steps:

[0056] Step 101: Acquire multi-source heterogeneous data.

[0057] Among them, multi-source heterogeneous data includes any one or more of spatial information, temporal information, semantic information and perceptual information.

[0058] In one embodiment, multi-source heterogeneous data may be different types of data sources from the Industrial Internet.

[0059] In one embodiment, spatial information may include city, company, public place, administrative district, and venue names; temporal information may include date, holiday, timestamp, time interval, and period; semantic information may include climate conditions, policy elements, label classification, attributes, and constraint rules; and perceptual information may include environment, motion position, video image point cloud, sound, voice, and social behavior.

[0060] Step 102: pre-process the multi-source heterogeneous data.

[0061] In one embodiment, multi-source heterogeneous data is first standardized. Then, specific data is normalized and pre-processed to facilitate placing the multi-source heterogeneous data in a unified feature space. Then, the multi-source heterogeneous data is temporally and spatially aligned to ensure that the temporal and spatial dimensions of the data from each data source match. Finally, outliers in the multi-source heterogeneous data are processed and data augmentation is performed.

[0062] In some embodiments of the present application, step 102 may specifically include the following steps:

[0063] Performing standardization on multi-source heterogeneous data to obtain standardized multi-source heterogeneous data;

[0064] Normalizing the standardized multi-source heterogeneous data to obtain normalized multi-source heterogeneous data;

[0065] Performing time and space alignment processing on the normalized multi-source heterogeneous data to obtain aligned multi-source heterogeneous data;

[0066] The aligned multi-source heterogeneous data are subjected to outlier removal and data enhancement processing to obtain preprocessed multi-source heterogeneous data.

[0067] Step 103: Calculate the eigenvalues of each of the pre-processed multi-source heterogeneous data using the eigenvector representation model.

[0068] In one embodiment, the feature vector representation model may be pre-constructed based on multi-source heterogeneous data.

[0069] It should be noted that the aforementioned eigenvalues are numerical indicators generated by quantifying different elements in multimodal data (such as spatial, temporal, semantic, and perceptual information). Eigenvalues can be used to discover hidden dependencies in the data and provide a basis for subsequent rule construction.

[0070] In step 104 , the characteristic values of each of the multi-source heterogeneous data are input into the trained association reasoning model to obtain an association probability vector obtained by the association reasoning model based on the characteristic values of each of the multi-source heterogeneous data.

[0071] Among them, the association reasoning model is obtained by training the meta-learning model using the meta-learning method.

[0072] In one embodiment, the characteristic values of each of the multi-source heterogeneous data are input into a trained association reasoning model, and the association reasoning model performs association reasoning based on the characteristic values of each of the multi-source heterogeneous data, and outputs the above association probability vector.

[0073] Step 105: Determine the association reasoning result of the multi-source heterogeneous data using the association probability vector.

[0074] In one embodiment, for the missing information in the association probability vector, the missing information in the heterogeneous data source can be supplemented based on the existing information association vector and the known cross-modal data reasoning relationship, so as to obtain a comprehensive association relationship set (i.e., association vector data set), further improving the data integrity and reasoning accuracy.

[0075] In some embodiments of this application, relationships between heterogeneous data sources can be inferred through constructed data vector association rules, association reasoning results, association relationship sets (i.e., association vector datasets), and supplemented feature factors, enabling accurate decision-making and reasoning based on data association. This reasoning process can be applied to multiple scenarios, such as equipment status prediction, quality inspection, and industrial monitoring, to improve industrial production efficiency and safety.

[0076] In some embodiments of the present application, the associative reasoning model can be trained using the following steps:

[0077] Step a1: Acquire multi-source heterogeneous sample data and preprocess the multi-source heterogeneous sample data.

[0078] In one embodiment, the multi-source heterogeneous sample data is first standardized. Then, specific multi-source heterogeneous sample data is subjected to normalization preprocessing to facilitate placing the multi-source heterogeneous sample data in a unified feature space. Then, the multi-source heterogeneous sample data is temporally and spatially aligned to ensure that the temporal and spatial dimensions of each sample data source match. Finally, outlier processing and data enhancement are performed on the multi-source heterogeneous sample data.

[0079] Step a2: Calculate the sample eigenvalues of each of the preprocessed multi-source heterogeneous sample data using the eigenvector representation model.

[0080] In some embodiments of the present application, a unified feature vector representation model can be constructed in advance using multi-source heterogeneous data. In view of the characteristics of multi-source heterogeneous data contained in the industrial Internet, a feature vector representation model for a specific scenario is constructed. The model consists of four parts: space, time, semantics, and perception information. These four parts each contain many information elements, and each information element can be numerically constructed based on all the features of the information element. As an example, the specific structure of the feature vector representation model (that is, the space-time-semantic-perception data resource pool) is as follows: Figure 2 shown.

[0081] Step a3: Generate a node corresponding to each multi-source heterogeneous sample data.

[0082] In one embodiment, different modal information can correspond to different types of nodes in the graph, with each spatial location (such as a geographic name or coordinates, equipment location) as a node; the time point, time period, or event timestamp as a time node; semantic information such as text, keywords, tags, policy points, etc. as a semantic node; and each sensor or its perceived features (such as temperature, pressure, image, etc.) can be a perception node.

[0083] Step a4: Based on the sample feature values corresponding to each node, the association relationship between the nodes is constructed to obtain an association relationship set.

[0084] In one embodiment, an association relationship can be established between nodes corresponding to isomorphic data, for example: Spatial association: If the geographical locations of two nodes are close, an association can be established between them; or the association can be defined based on the structure of the physical network (such as the connection or communication between equipment). Temporal association: Based on the similarity of time series or the chronological order of events, the temporal association between a time node and other nodes is defined. Semantic association: If two nodes have similar semantic information (such as the same keywords in a text), a semantic association is established. Perceptual association: When the numerical change trends of different sensors are similar, or when their monitoring objects are associated, it is defined as a perceptual association.

[0085] In one embodiment, it is also possible to build association relationships between nodes corresponding to heterogeneous data, and to build association relationships between space, time, semantics, and perception information through complete chains such as events. For example, an event occurs at a certain time in a certain place due to certain semantic information, and then perception is performed on the scene of the event.

[0086] In some embodiments of the present application, since the perception information contains multiple data sources and a complex data structure, heterogeneous perception data fusion algorithms can be designed for different application scenarios to establish correlations between perception information. For example, the perception of a specific scene may include data such as timestamp, coordinate position, scene temperature, scene climate environment, scene omnidirectional image, scene thermal image, and 3D spatial point cloud, and corresponding correlations need to be established between these data.

[0087] For example, the above-mentioned perceptual heterogeneous data fusion algorithm can adopt common association analysis methods (such as mutual information, correlation coefficient), multimodal fusion framework (such as neural network, Transformer), etc.

[0088] In step a5, the association relationship set is used to train the meta-learning model using a meta-learning method to obtain an association reasoning model.

[0089] In one embodiment, a gradient-based meta-learning approach is used to rapidly construct a small-sample adaptive meta-learning model for small-sample data. This model first uses the aforementioned association relationships as samples, further enabling rapid mapping and adaptation between data of different modalities, reducing reliance on large-scale training data. This model then forms information-related numerical vectors, thereby constructing mapping relationships between the aforementioned discrete spatial, temporal, semantic, and perceptual information.

[0090] In one embodiment, a unified vector encoding nodes (such as space, time, semantics, and perception) can be constructed, and the parameters of the meta-learning model can be initialized through the association relationship and its weight value. Then, the meta-learning task is designed: the above association relationship (such as "a certain sales point in a certain city in a certain province + a timestamp at 4 pm → a product sales event") is used as a support set sample, and a gradient-based meta-learning method is used. In the "inner loop", a small number of samples are used to quickly fine-tune the model parameters (for example, by calculating the association loss between nodes and back-propagating to update the mapping layer weights), and in the "outer loop", the cross-modal generalization ability of the model is optimized, so that the model can finally accept heterogeneous data from new scenarios as input. , output cross-modal association numerical vectors (such as a certain province or region → timestamp → introduction of a certain policy or occurrence of a certain event → perception of social impact information → degree of impact on future product sales → inference of production capacity adjustment; for example, a certain factory area → timestamp → equipment fault detection, abnormal information from a certain sensor → confidence level of equipment damage → degree of impact on the entire production line and repair time → inference of the impact on production plans and sales chains). The model's preliminary association relationships are constructed through association rules, so that the model does not need to relearn the underlying associations during reasoning, and cross-modal mapping can be achieved directly based on existing weights and node features (such as inferring "future plans or events" from "space + time + semantics").

[0091] In one embodiment, the input data of the association reasoning model can be multimodal nodes (original features such as space and time) and association relationship pairs (some with labels), and the output data can be association probability vectors between modes or features (such as [modality A→modality B: 0.7, feature C→feature D: 0.3]), which is suitable for small sample scenarios such as equipment failure prediction and product production planning.

[0092] In some embodiments of the present application, after step a4, the method may further include: for each node, using a graph convolutional neural network to perform aggregation processing on the node and the nodes adjacent to the node to obtain the neighbor information of the node, and updating the sample feature value of the node based on the neighbor information.

[0093] In one embodiment, a graph convolutional network (GCN) can be used to aggregate information about each node and its neighboring nodes. GCN can effectively propagate multimodal information (such as spatial, temporal, semantic, and perceptual features) of nodes through the graph structure and learn complex relationships between nodes. For example, each GCN layer aggregates information about a node's neighbors and updates the node's feature representation. By using multiple layers of GCN, the perceptual domain can be gradually expanded, allowing the representation of each node to include more contextual information.

[0094] In some embodiments of the present application, after step a4, the method may further include: for each node, using a graph attention network to calculate a weight value between the node and the nodes adjacent to the node;

[0095] Step a5 may specifically include: initializing model parameters of the meta-learning model using weight values and the association relationship set; and training the meta-learning model using the association relationship set and a meta-learning method to obtain an association reasoning model.

[0096] In one embodiment, among spatial, temporal, semantic, and perceptual nodes, some nodes may be more important than others. Therefore, using a Graph Attention Network (GAT), different weights can be assigned between each node and its neighboring nodes, dynamically adjusting the relevance.

[0097] In some embodiments of the present application, the weight value may be calculated using the following formula:

[0098]

[0099] in, is the node representation of node i in the l+1 layer, σ is the activation function, α ij is the attention weight between node i and its neighbor node j, W is the weight matrix, Ν(i) is the set of neighbor nodes of node i, h j is the representation of neighbor node j.

[0100] According to the cross-modal association reasoning method based on small sample learning proposed in the embodiment of the present disclosure, multi-source heterogeneous data is obtained, the multi-source heterogeneous data is preprocessed, and the eigenvalues of each of the preprocessed multi-source heterogeneous data are calculated using a feature vector representation model. The eigenvalues of each of the multi-source heterogeneous data are input into the trained association reasoning model to obtain an association probability vector obtained by the association reasoning model based on the eigenvalues of each of the multi-source heterogeneous data. The association reasoning model is obtained by training the meta-learning model using a meta-learning method, and the association reasoning results of the multi-source heterogeneous data are determined using the association probability vector, thereby realizing rapid mapping and adaptation between cross-modal data, and realizing cross-modal reasoning of heterogeneous data in a small sample data environment, thereby effectively improving the processing and information mining capabilities of multi-source heterogeneous data of the industrial Internet, thereby improving the data utilization efficiency of the industrial Internet.

[0101] Figure 4 This is a block diagram of a cross-modal association reasoning device based on small sample learning according to an exemplary embodiment. Figure 4 The device includes an acquisition unit 401, a preprocessing unit 402, a calculation unit 403, an association reasoning unit 404 and a determination unit 405.

[0102] The acquisition unit 401 is configured to acquire multi-source heterogeneous data; the multi-source heterogeneous data includes any one or more of spatial information, temporal information, semantic information, and perceptual information;

[0103] A preprocessing unit 402 is used to preprocess multi-source heterogeneous data;

[0104] A calculation unit 403 is used to calculate the eigenvalues of the pre-processed multi-source heterogeneous data using the eigenvector representation model;

[0105] The association reasoning unit 404 is configured to input the characteristic values of each of the multi-source heterogeneous data into the trained association reasoning model to obtain an association probability vector obtained by the association reasoning model through association reasoning based on the characteristic values of each of the multi-source heterogeneous data; the association reasoning model is obtained by training the meta-learning model using a meta-learning method;

[0106] The determining unit 405 is configured to determine the association inference result of the multi-source heterogeneous data using the association probability vector.

[0107] In some embodiments of the present application, the pre-processing unit 402 may be specifically configured to:

[0108] Performing standardization on multi-source heterogeneous data to obtain standardized multi-source heterogeneous data;

[0109] Normalizing the standardized multi-source heterogeneous data to obtain normalized multi-source heterogeneous data;

[0110] Performing time and space alignment processing on the normalized multi-source heterogeneous data to obtain aligned multi-source heterogeneous data;

[0111] The aligned multi-source heterogeneous data are subjected to outlier removal and data enhancement processing to obtain preprocessed multi-source heterogeneous data.

[0112] In some embodiments of the present application, the apparatus may further include a training unit, which may be specifically configured to:

[0113] Acquire multi-source heterogeneous sample data and preprocess the multi-source heterogeneous sample data;

[0114] The eigenvector representation model is used to calculate the sample eigenvalues of the preprocessed multi-source heterogeneous sample data;

[0115] Generate nodes corresponding to each multi-source heterogeneous sample data;

[0116] Based on the sample feature values corresponding to each node, the association relationship between nodes is constructed to obtain an association relationship set;

[0117] By using the set of association relations, the meta-learning method is adopted to train the meta-learning model and obtain the association reasoning model.

[0118] In some embodiments of the present application, the training unit may also be used to:

[0119] For each node, a graph convolutional neural network is used to aggregate the node and the nodes adjacent to the node to obtain the node's neighbor information, and the node's sample feature value is updated based on the neighbor information.

[0120] In some embodiments of the present application, the training unit may also be used to:

[0121] For each node, a graph attention network is used to calculate the weight value between the node and its adjacent nodes;

[0122] Initialize the model parameters of the meta-learning model using the weight values and the set of associations;

[0123] By using the set of association relations, the meta-learning method is adopted to train the meta-learning model and obtain the association reasoning model.

[0124] In some embodiments of the present application, the training unit may also be used to:

[0125] The weight value is calculated using the following formula:

[0126]

[0127] in, is the node representation of node i in the l+1 layer, σ is the activation function, α ij is the attention weight between node i and its neighbor node j, W is the weight matrix, Ν(i) is the set of neighbor nodes of node i, h j is the representation of neighbor node j.

[0128] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0129] According to the cross-modal association reasoning device based on small sample learning proposed in the embodiment of the present disclosure, multi-source heterogeneous data is obtained, the multi-source heterogeneous data is preprocessed, and the eigenvalues of each of the preprocessed multi-source heterogeneous data are calculated using a feature vector representation model. The eigenvalues of each of the multi-source heterogeneous data are input into the trained association reasoning model to obtain an association probability vector obtained by the association reasoning model based on the eigenvalues of each of the multi-source heterogeneous data. The association reasoning model is obtained by training the meta-learning model using a meta-learning method, and the association reasoning results of the multi-source heterogeneous data are determined using the association probability vector, thereby realizing rapid mapping adaptation between cross-modal data and being able to realize cross-modal reasoning of heterogeneous data in a small sample data environment.

[0130] Figure 5This is a block diagram of an apparatus for a method for cross-modal relevance reasoning based on small sample learning, according to an exemplary embodiment. For example, apparatus 500 may be an electronic device, such as a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0131] Reference Figure 5 , apparatus 500 may include one or more of the following components: a processing component 502 , a memory 504 , a power component 506 , a multimedia component 508 , an audio component 510 , an input / output (I / O) interface 512 , a sensor component 514 , and a communication component 516 .

[0132] The processing component 502 generally controls the overall operation of the device 500, such as operations associated with display, phone calls, data communications, camera operation, and recording operations. The processing component 502 may include one or more processors 520 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 502 may include one or more modules to facilitate interaction between the processing component 502 and other components. For example, the processing component 502 may include a multimedia module to facilitate interaction between the multimedia component 508 and the processing component 502.

[0133] The memory 504 is configured to store various types of data to support operations on the device 500. Examples of such data include instructions for any application or method operating on the device 500, contact data, phone book data, messages, pictures, videos, etc. The memory 504 can be implemented by any type of volatile or non-volatile storage device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0134] Power component 506 provides power to the various components of device 500. Power component 506 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to device 500.

[0135] The multimedia component 508 includes a screen that provides an output interface between the device 500 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, slides, and gestures on the touch panel. The touch sensor can not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 508 includes a front camera and / or a rear camera. When the device 500 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have a focal length and optical zoom capability.

[0136] The audio component 510 is configured to output and / or input audio signals. For example, the audio component 510 includes a microphone (MIC), which is configured to receive external audio signals when the device 500 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 504 or transmitted via the communication component 516. In some embodiments, the audio component 510 also includes a speaker for outputting audio signals.

[0137] I / O interface 512 provides an interface between processing component 502 and peripheral interface modules, such as a keyboard, click wheel, buttons, etc. These buttons may include but are not limited to: a home button, volume buttons, a start button, and a lock button.

[0138] The sensor assembly 514 includes one or more sensors for providing various aspects of the status assessment of the device 500. For example, the sensor assembly 514 can detect the open / closed state of the device 500, the relative positioning of components, such as the display and keypad of the device 500. The sensor assembly 514 can also detect changes in the position of the device 500 or a component of the device 500, the presence or absence of user contact with the device 500, the orientation or acceleration / deceleration of the device 500, and temperature changes of the device 500. The sensor assembly 514 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 514 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 514 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0139] The communication component 516 is configured to facilitate wired or wireless communication between the device 500 and other devices. The device 500 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 516 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 516 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0140] In an exemplary embodiment, the apparatus 500 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above-described method.

[0141] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 504 including instructions, which can be executed by the processor 520 of the apparatus 500 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0142] In an exemplary embodiment, a computer program product is also provided, comprising a computer program, which implements the above method when executed by the processor 520 of the apparatus 500 .

[0143] Other embodiments of the present invention will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the invention that follow from the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the invention being indicated by the following claims.

[0144] It should be understood that the present invention is not limited to the exact construction described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A cross-modal association reasoning method based on small sample learning, characterized by: include: Acquiring multi-source heterogeneous data; the multi-source heterogeneous data includes any one or more of spatial information, temporal information, semantic information, and perceptual information; Preprocessing the multi-source heterogeneous data; The eigenvalues of each preprocessed multi-source heterogeneous data are calculated using the eigenvector representation model; Inputting the characteristic values of each of the multi-source heterogeneous data into the trained association reasoning model to obtain an association probability vector obtained by the association reasoning model through association reasoning based on the characteristic values of each of the multi-source heterogeneous data; The association reasoning model is obtained by training the meta-learning model using a meta-learning method; The association probability vector is used to determine the association reasoning result of the multi-source heterogeneous data.

2. The cross-modal association reasoning method based on small sample learning according to claim 1 is characterized in that: The preprocessing of the multi-source heterogeneous data includes: Performing standardization on the multi-source heterogeneous data to obtain standardized multi-source heterogeneous data; Performing normalization on the standardized multi-source heterogeneous data to obtain normalized multi-source heterogeneous data; Performing temporal and spatial alignment processing on the normalized multi-source heterogeneous data to obtain aligned multi-source heterogeneous data; The outlier removal processing and data enhancement processing are performed on the multi-source heterogeneous data after the alignment processing to obtain the pre-processed multi-source heterogeneous data.

3. The cross-modal association reasoning method based on small sample learning according to claim 1 is characterized in that: The association reasoning model is trained by the following steps: Acquire multi-source heterogeneous sample data, and preprocess the multi-source heterogeneous sample data; The eigenvector representation model is used to calculate the sample eigenvalues of the preprocessed multi-source heterogeneous sample data; Generate nodes corresponding to each multi-source heterogeneous sample data; Based on the sample feature values corresponding to each node, the association relationship between nodes is constructed to obtain an association relationship set; The association relationship set is utilized to train the meta-learning model using a meta-learning method to obtain the association reasoning model.

4. The cross-modal association reasoning method based on small sample learning according to claim 3 is characterized in that: After constructing the association relationship between nodes based on the sample feature value corresponding to each node to obtain the association relationship set, the method further includes: For each node, a graph convolutional neural network is used to aggregate the node and the nodes adjacent to the node to obtain the neighbor information of the node, and the sample feature value of the node is updated based on the neighbor information.

5. The cross-modal association reasoning method based on small sample learning according to claim 3 is characterized in that: After constructing the association relationship between nodes based on the sample feature value corresponding to each node to obtain the association relationship set, the method further includes: For each node, a graph attention network is used to calculate the weight value between the node and the nodes adjacent to the node; The method of using the association relationship set and adopting a meta-learning method to train the meta-learning model to obtain the association reasoning model includes: Initializing model parameters of the meta-learning model using the weight values and the set of association relationships; The association relationship set is utilized to train the meta-learning model using a meta-learning method to obtain the association reasoning model.

6. The cross-modal association reasoning method based on small sample learning according to claim 5 is characterized in that: The using of a graph attention network to calculate the weight value between the node and the nodes adjacent to the node includes: The weight value is calculated using the following formula: in, is the node representation of node i in the l+1 layer, σ is the activation function, α ij is the attention weight between node i and its neighbor node j, W is the weight matrix, Ν(i) is the neighbor node set of node i, h j is the representation of neighbor node j.

7. A cross-modal association reasoning device based on small sample learning, characterized in that: include: An acquisition unit, configured to acquire multi-source heterogeneous data; the multi-source heterogeneous data includes any one or more of spatial information, temporal information, semantic information, and perceptual information; A preprocessing unit, configured to preprocess the multi-source heterogeneous data; A calculation unit, used to calculate the eigenvalues of each of the preprocessed multi-source heterogeneous data using a eigenvector representation model; An association reasoning unit, configured to input the characteristic values of each of the multi-source heterogeneous data into a trained association reasoning model, and obtain an association probability vector obtained by the association reasoning model through association reasoning based on the characteristic values of each of the multi-source heterogeneous data; The association reasoning model is obtained by training the meta-learning model using a meta-learning method; A determination unit is used to determine the association reasoning result of the multi-source heterogeneous data using the association probability vector.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program, characterized in that The computer program implements the method according to any one of claims 1 to 6 when executed by a processor.

Images