Service fusion method and system based on multi-jump multi-modal knowledge graph

Abstract

The invention discloses a service fusion method and system based on a multi-jump multi-modal knowledge graph. Encoding the service request and the knowledge graph by using a pre-training model to obtain a service request feature, an entity feature, a relationship feature, a text description feature, an image semantic feature and a video semantic feature; fusing the entity feature, the text description feature, the image semantic feature and the video semantic feature to obtain a service entity feature, inputting the service entity feature into a Transform model, and filtering the knowledge graph by using the service request feature to obtain a knowledge graph subset; constructing and utilizing a reinforcement learning model and knowledge graph training to form an offline strategy network; and updating the off-line strategy network on line, and establishing a relation between different service entities under the guidance of a reward mechanism to realize service fusion. According to the method, the problems of low reasoning efficiency, difficulty in cross-modal association and the like are solved; the defects of a traditional model in the aspects of cross-domain relation and context capture are overcome, and the knowledge graph reasoning precision and efficiency are improved.

Description

Technical Field

[0001] The present invention relates to the fields of knowledge graphs and reinforcement learning, and in particular to a service fusion method and system based on multi-jump multimodal knowledge graphs. Background Art

[0002] With the acceleration of urbanization and people's constant pursuit of quality of life, the construction of smart communities has become one of the important directions of urban development. Smart communities involve multiple fields, including property management, community services, public safety, environmental monitoring, etc. Therefore, it is necessary to cross multiple fields and integrate various information and service resources to achieve intelligent and efficient community operation and management. In this context, building a smart community knowledge graph has become an important measure, which can integrate information and knowledge in various fields and realize cross-border integration and intelligent management of multi-field information. The application of knowledge graphs helps to realize information association and interaction between different fields, and promote the coordinated operation of various services within the community and intelligent decision-making support. In addition, the smart community knowledge graph can also be used as a knowledge source for the intelligent question-answering system to provide residents with intelligent question-answering and interactive services, thereby improving the quality of life of community residents and the intelligent level of urban management.

[0003] The study of knowledge graphs involves multiple aspects such as knowledge representation and modeling, extraction and acquisition, fusion and reasoning. As a form of structured human knowledge, it can help integrate this information and build a cross-domain smart community knowledge graph, including property information, resident services, security data, environmental monitoring, etc. Through such a knowledge graph, a comprehensive understanding and management of community resources can be achieved, supporting the coordinated operation of various services in the smart community and intelligent decision-making support. For example, knowledge graphs can be used to provide intelligent question-answering and interactive services to provide residents with personalized community service information; knowledge graphs can also be used for intelligent recommendations to provide decision-making support for community managers, such as early warning of community safety hazards or optimizing community resource allocation. Therefore, knowledge graphs play an important role in the construction of smart communities and are one of the important technical means to achieve smart community operation and management.

[0004] Although knowledge graphs have broad potential in multiple application fields, there are still some challenges and problems at the current stage. First, cross-domain smart community knowledge graphs have different characteristics and representations. How to effectively fuse and model these data to achieve accurate knowledge graph description is still an important research issue. Second, the algorithms at the current stage mainly focus on models that complete single-hop knowledge graph fusion. However, the reasoning performance of these models is relatively low and they cannot capture complex cross-domain relationships and contexts well. Therefore, it is difficult for them to integrate services of different types or sources to provide a more comprehensive and integrated solution. This limitation limits the application effect and development potential of knowledge graphs in the field of service fusion, and further research and improvement are needed to overcome these challenges. Therefore, how to improve the reasoning performance of knowledge graph fusion models to better capture complex cross-domain relationships and contexts and integrate services of different types or sources has become an urgent problem to be solved in the field of service fusion at the current stage.

[0005] One of the current existing technologies is the fusion method based on traditional knowledge graphs in the paper "Metap: Meta pattern learning for one-shot knowledge graph completion". This type of technology regards text knowledge graphs as an important form of human knowledge structure. These graphs use text as the main object and construct triples to describe the relationship and attributes between entities, thereby establishing a rich semantic network. In this network, each entity is given a semantic label, so that the association and semantic connection between different entities can be clearly presented, thus providing strong support for the storage, retrieval and reasoning of knowledge. This fusion method enables text knowledge to be represented and utilized in a structured way, realizing the expansion of application potential in various fields. The disadvantages of this technology are that, first, these methods do not consider multimodal knowledge, resulting in a lack of comprehensive consideration of diverse information in the process of knowledge representation and reasoning. Secondly, these methods only focus on completing the model of single-hop reasoning, and cannot handle complex multi-hop reasoning tasks well, resulting in low reasoning performance and difficulty in meeting the current complex needs in fields such as smart community construction.

[0006] The second existing technology is the fusion method based on multimodal knowledge graphs in the paper "Multimodal data enhanced representation learning for knowledge graphs". This type of technology integrates different types of data (such as text, images, videos, etc.) into the knowledge graph to enrich and improve the representation of knowledge. This method not only takes into account the construction method of traditional text knowledge graphs, but also combines multimodal data with it to form a cross-modal knowledge representation. By associating and fusing data of different modalities, knowledge can be expressed more comprehensively and richer semantic information can be provided. Such a fusion method helps to better understand and reason about complex cross-modal knowledge, thereby providing stronger support and solutions for fields such as smart community construction. The disadvantage of this technology is that the current fusion method based on multimodal knowledge graphs may perform poorly on cross-modal associations, because it tends to integrate data of different modalities into the graph independently, and lacks in-depth mining and modeling of cross-modal associations. Summary of the invention

[0007] The purpose of the present invention is to overcome the shortcomings of the existing methods and propose a service fusion method and system based on multi-jump multimodal knowledge graph. The main problem solved by the present invention is how to effectively use multimodal auxiliary features to complete multi-jump knowledge graph reasoning, and how to overcome the problems of low reasoning efficiency of service fusion model and difficulty in cross-modal association.

[0008] In order to solve the above problems, the present invention proposes a service fusion method based on multi-jump multimodal knowledge graph, the method comprising:

[0009] Input the multimodal cross-border property service data knowledge graph G and service request Q, use the pre-trained model to encode the service request Q and the multimodal data in the knowledge graph G, and extract the service request feature q, entity feature e, relationship feature r, text description feature td, image semantic feature p, and video semantic feature v;

[0010] The entity feature e, text description feature td, image semantic feature p, and video semantic feature v are matrix-multiplied to map them to the same semantic space to obtain the service entity feature The service entity feature set of all service entities is recorded as

[0011] The service entity feature set Input into the Transformer model, and use the service request feature q to filter the knowledge graph G, narrow the service search scope, and obtain the knowledge graph subset g of the knowledge graph G;

[0012] Constructing a service fusion reinforcement learning model within the scope of the knowledge graph subset g, and defining the state, action, state transfer and reward mechanism of reinforcement learning in the service fusion process for the intelligent agent that executes the service fusion strategy;

[0013] The agent is trained offline using the service fusion reinforcement learning model and the knowledge graph G to form a policy network after offline training, the input of which is a knowledge graph subset feature expression Z consisting of the knowledge graph subset g and the service request feature q;

[0014] During the service query process, the offline trained policy network is updated online. During the updating process, the intelligent agent, under the guidance of the reward mechanism, continuously establishes connections between different service entities, thereby completing the relationship between different service entities in the multimodal cross-border property service data knowledge graph G, and then realizing service integration.

[0015] Preferably, the input multimodal cross-border property service data knowledge graph G and service request Q, use the pre-trained model to encode the service request Q and the multimodal data in the knowledge graph G, and extract the service request feature q, entity feature e, relationship feature r, text description feature td, image semantic feature p, and video semantic feature v, specifically:

[0016] The multimodal cross-border property service data knowledge graph G is represented as where ε is the set of entities, is a set of semantic relations. is a set of triplets present in G;

[0017] Use the pre-trained BERT model to encode the text data in the knowledge graph G to obtain entity features e, relationship features r, and text description features td, and encode the service request Q to obtain the service request feature q;

[0018] Use the pre-trained convolutional neural network ResNet to encode the image data in the knowledge graph G to obtain the image semantic feature p;

[0019] Use the pre-trained convolutional neural network C3D to encode the video data in the knowledge graph G to obtain the video semantic features v.

[0020] Preferably, the entity feature e, text description feature td, image semantic feature p, and video semantic feature v are matrix-multiplied to map them to the same semantic space to obtain the service entity feature The service entity feature set of all service entities is recorded as Specifically:

[0021] Service Entity Characteristics Calculated by the following formula:

[0022]

[0023] in is the learnable weight matrix, represents a real number field with 4d rows and d columns, where d is the feature dimension; by performing matrix multiplication on the entity feature e, text description feature td, image semantic feature p, and video semantic feature v, they are mapped to the same semantic space, and finally a service entity feature set is obtained Where m represents the total number of service entities.

[0024] Preferably, the service entity feature set Input into the Transformer model, and use the service request feature q to filter the knowledge graph G, narrow the service search scope, and obtain the knowledge graph subset g of the knowledge graph G, specifically:

[0025] The knowledge graph subset g is calculated by the following formula:

[0026] g=FFN(Concat i∈1,....,h head i )

[0027]

[0028]

[0029]

[0030]

[0031] in After The variants of the original information obtained after linear transformation of three learnable matrices, h = 8 represents the number of heads of multi-head attention, FFN and Concat represent deep network and feature connection respectively; are the queries, keys, and values ​​in the attention mechanism, especially in the Transformer model, is the characteristic dimension of K, mainly to prevent With K i The dot product value of pos is too large; q and pos e It is the position code of the service request feature q and the entity feature e.

[0032] Preferably, a service fusion reinforcement learning model is constructed within the scope of the knowledge graph subset g, and the state, action, state transfer and reward mechanism of reinforcement learning in the service fusion process are defined for the agent executing the service fusion strategy, specifically:

[0033] Use S = {s1,s2,…,s t} to represent the state space in the service request process, where s t = Describes the status of the service fusion process, t represents the number of multi-level jumps between the relationships between entities in the knowledge graph, and e t is the entity visited at step t, e s and r q They are service entities and service relationships. and ε t Indicates e t Adjacent entities and relationships;

[0034] Use A={A1,A2,…,A t , STOP} to represent the action space in the service request process, where STOP means the action is terminated, A t Indicates that t The edge set of the remaining entities connected, and extend the service path to the next service entity e t+1 ; The complete process can be expressed as follows:

[0035] A t ={(e t+1 , r t+1 )|(e t , t+1 , e t+1 )∈g}

[0036] It means in e t With e t+1 There is a relationship t+1 When it is used as an action A t ;

[0037] In order to avoid infinite jumps in service fusion and query process, the stop action STOP is executed when the query step number t increases to the maximum step number T;

[0038] Use P = {P1, P2, ..., P r} to represent the state transition set in the service fusion and query process, where P r Indicates that from the current state s t Reach the next state s through the service relationship r t+1 ; The complete process can be expressed as follows:

[0039]

[0040] In order to better encourage agents to achieve service integration of different types and sources, accuracy rewards, distance rewards, and diversity rewards are designed respectively:

[0041] Accuracy reward R d The purpose is to encourage intelligent acquisition of accurate related service entities, thereby achieving service integration of different types and sources; the specific process is shown in the following formula:

[0042]

[0043] When the correct service entity is found, the reward is 1; when the service fusion is wrong and a service entity that does not match the query is found, the reward is two service entities e t ,e d Semantic cosine similarity to avoid the problem of too sparse rewards;

[0044] Distance Reward R step The purpose is to encourage the agent to complete service fusion under a limited number of jump steps T to avoid excessive service fusion; the specific process is shown in the following formula:

[0045]

[0046] Where k represents the number of jump steps used in the current service fusion process. If it is less than the specified compensation range, a reward is given, and if it exceeds the range, a penalty is given to help the model converge better;

[0047] Diversity Rewards div The purpose is to encourage the agent to explore more action spaces and strike a balance between exploration and utilization, so as to ensure that the knowledge graph can be fully explored during the learning process, and the existing knowledge can be used to maximize the cumulative reward and avoid the local optimal service fusion of the system; the specific process is shown in the following formula:

[0048]

[0049] Where L represents the number of known reasoning paths, and represents the state transition probability distribution in the state transition set P in the service fusion and query process, μ 2 and Represents the variance of the state transition probability distribution to balance the dimensions of the two;

[0050] The final reward function is as follows:

[0051] R=R d +R step +R div .

[0052] Preferably, the agent is trained offline using the service fusion reinforcement learning model and the knowledge graph G to form a policy network after offline training, the input of which is the knowledge graph subset feature expression Z consisting of the knowledge graph subset g and the service request feature q, specifically:

[0053] Using the knowledge graph Construct missing triples (e s ,r,? ),(e s ,? ,e d ), (?,r,e d ) as a service request, where “?” represents the service entity or service relationship that needs to be fused; the training goal is to infer the missing service entity or relationship element through a relationship path that is shorter or equal to k hops, where k is an integer not less than 1;

[0054] The knowledge graph subset g and the service request feature q are used to construct a knowledge graph subset feature expression Z, the formula is: Z = Concat [g: q], where Concat represents feature connection;

[0055] Use a feedforward neural network to implement the policy network π θ , the policy network inputs the feature expression Z of the knowledge graph subset and outputs the next action with the highest executable probability to realize the interaction between the knowledge graph and the agent; as shown in the following formula:

[0056] π θ (a t |s t )=softmax(A t (W2Relu(Z)))

[0057] Where Relu represents the activation function, Denotes the set A at step t t Any one of;

[0058] For the policy network π θ To train:

[0059] During the training process of reinforcement learning, the agent interacts with the multimodal cross-border property service data knowledge graph. The agent observes the current state of the knowledge graph and selects actions to perform based on the current state. The behavior of the agent is determined by the policy network, which defines the rules for selecting actions in a given state. After the agent performs an action, the environment obtains the reward of the reward mechanism based on the agent's action and current state.

[0060] The interaction between the agent and the knowledge graph is ongoing. Each step of the interaction includes the following process: the agent observes the current state, selects an action based on the current strategy, and after executing the action, the environment moves to a new state and gives a reward. The agent updates its strategy and action set based on the reward and the new state.

[0061] In the process of interaction between the agent and the knowledge graph, the agent will collect a lot of interaction experience, including observed states, executed actions, and rewards obtained; the learning algorithm uses this experience to continuously update the agent's strategy and value function so that the agent can achieve better performance;

[0062] Finally, the following formula is used as the training goal to maximize the cumulative reward of the model and obtain the optimal strategy network:

[0063]

[0064] Where E represents expectation, e s ,r,e d is a triple in the knowledge graph, representing the starting entity, relationship, and target entity in the graph respectively; π θ is a policy network based on parameters θ, which determines the probability of choosing an action at each time step; a1,…,a t is based on the policy network π θ The action sequence selected at each time step; [R(S t |e s ,r)] represents the agent in a given initial service entity e s , in the case of service relationship r, according to the policy network π θ Select an action sequence a1,…,a t The final state S obtained after t The reward J(θ) represents the agent's strategy network π under all possible starting entity, relationship and target entity triples in the graph. θ The expected cumulative reward of the action is selected at each time step; finally, by maximizing J(θ), the cumulative reward of the model is maximized to obtain the optimal policy network.

[0065] Accordingly, the present invention also proposes a service fusion system based on a multi-jump multimodal knowledge graph, comprising:

[0066] The feature encoding unit is used to input the multimodal cross-border property service data knowledge graph G and the service request Q, use the pre-trained model to encode the service request Q and the multimodal data in the knowledge graph G, and extract the service request feature q, entity feature e, relationship feature r, text description feature td, image semantic feature p, and video semantic feature v;

[0067] The feature fusion unit is used to perform matrix multiplication on the entity feature e, text description feature td, image semantic feature p, and video semantic feature v, so as to map them to the same semantic space and obtain the service entity feature The service entity feature set of all service entities is recorded as

[0068] A feature filtering unit is used to filter the service entity feature set Input into the Transformer model, and use the service request feature q to filter the knowledge graph G, narrow the service search scope, and obtain the knowledge graph subset g of the knowledge graph G;

[0069] A reinforcement learning model construction unit, used to construct a service fusion reinforcement learning model within the scope of the knowledge graph subset g, and define the state, action, state transfer and reward mechanism of reinforcement learning in the service fusion process for the intelligent agent that executes the service fusion strategy;

[0070] An offline training unit, used to perform offline training on the agent using the service fusion reinforcement learning model and the knowledge graph G to form a policy network after offline training, wherein the input of the policy network is a knowledge graph subset feature expression Z consisting of the knowledge graph subset g and the service request feature q;

[0071] The online training and service fusion unit is used to update the offline trained policy network online during the service query process. During the updating process, the intelligent agent, under the guidance of the reward mechanism, continuously establishes connections between different service entities, thereby completing the relationship between different service entities in the multimodal cross-border property service data knowledge graph G, and then realizing service fusion.

[0072] The implementation of the present invention has the following beneficial effects:

[0073] The present invention proposes an effective service fusion model, which overcomes the problems of low reasoning efficiency and difficulty in cross-modal association by extracting features and fine-grained fusion of structured data and multimodal auxiliary data, and adopting an attention-based representation method to generate multimodal complementary representations. This integration method effectively overcomes the shortcomings of traditional models in capturing cross-domain relationships and contexts, improves the accuracy and efficiency of knowledge graph reasoning, and thus provides a foundation for applications in the field of service fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 It is an overall flow chart of a service fusion method based on a multi-jump multi-modal knowledge graph according to an embodiment of the present invention;

[0075] Figure 2It is a structural diagram of a service fusion system based on a multi-jump multimodal knowledge graph according to an embodiment of the present invention. DETAILED DESCRIPTION

[0076] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0077] Figure 1 is an overall flow chart of a service fusion method based on a multi-jump multi-modal knowledge graph according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0078] S1, input multimodal cross-border property service data knowledge graph G and service request Q, use the pre-trained model to encode the service request Q and the multimodal data in the knowledge graph G, and extract the service request feature q, entity feature e, relationship feature r, text description feature td, image semantic feature p, and video semantic feature v;

[0079] S2 performs matrix multiplication on the entity feature e, text description feature td, image semantic feature p, and video semantic feature v extracted by S1, thereby mapping them to the same semantic space to obtain the service entity feature The service entity feature set of all service entities is recorded as

[0080] S3, the service entity feature set of S2 Input into the Transformer model, and use the service request feature q of S1 to filter the knowledge graph G, narrow the service search scope, and obtain the knowledge graph subset g of the knowledge graph G;

[0081] S4, constructs a service fusion reinforcement learning model within the scope of the knowledge graph subset g of S3, and defines the state, action, state transfer and reward mechanism of reinforcement learning in the service fusion process for the intelligent agent that executes the service fusion strategy;

[0082] S5, uses the service fusion reinforcement learning model and knowledge graph G of S4 to train the agent offline to form a policy network after offline training. The input of the policy network is the knowledge graph subset feature expression Z composed of the knowledge graph subset g of S3 and the service request feature q of S1;

[0083] S6, during the service query process, updates the offline trained policy network of S5 online. During the updating process, under the guidance of the reward mechanism, the intelligent agent continuously establishes connections between different service entities, thereby completing the relationship between different service entities in the multimodal cross-border property service data knowledge graph G, and then realizing service integration.

[0084] Step S1 is as follows:

[0085] The multimodal cross-border property service data knowledge graph G is represented as where ε is the set of entities, is a set of semantic relations. is a set of triplets present in G;

[0086] S1-1, use the pre-trained BERT model to encode the text data in the knowledge graph G to obtain entity features e, relationship features r, text description features td, and encode the service request Q to obtain service request features q;

[0087] S1-2, use the pre-trained convolutional neural network ResNet to encode the image data in the knowledge graph G to obtain the image semantic feature p;

[0088] S1-3, use the pre-trained convolutional neural network C3D to encode the video data in the knowledge graph G to obtain the video semantic feature v.

[0089] Step S2 is as follows:

[0090] Service Entity Characteristics Calculated by the following formula:

[0091]

[0092] in is the learnable weight matrix, Represents a real number field with 4d rows and d columns, where d is the feature dimension; by performing matrix multiplication on entity feature e, text description feature td, image semantic feature p, and video semantic feature v, they are mapped to the same semantic space and finally obtain the service entity feature set Where m represents the total number of service entities.

[0093] Step S3 is as follows:

[0094] The knowledge graph subset g is calculated by the following formula:

[0095] g=FFN(Concat i∈1,....,h head i )

[0096]

[0097]

[0098]

[0099]

[0100] in After The variants of the original information obtained after linear transformation of three learnable matrices, h = 8 represents the number of heads of multi-head attention, FFN and Concat represent deep network and feature connection respectively; are the queries, keys, and values ​​in the attention mechanism, especially in the Transformer model, is the characteristic dimension of K, mainly to prevent With K i The dot product value of pos is too large; q and pos e It is the position encoding of the service request feature q and the entity feature e.

[0101] Step S4 is as follows:

[0102] S4-1, using S={s1,s2,…,s t} to represent the state space in the service request process, where Describes the status of the service fusion process, t represents the number of multi-level jumps between the relationships between entities in the knowledge graph, and e t is the entity visited at step t, e s and r q They are service entities and service relationships. and ε t Indicates e t Adjacent entities and relationships;

[0103] S4-2, using A={A1,A2,…,A t , STOP} to represent the action space in the service request process, where STOP means the action is terminated, A t Indicates that t The edge set of the remaining entities connected, and extend the service path to the next service entity e t+1 ; The complete process can be expressed as follows:

[0104] A t ={(e t+1 , r t+1 )|(e t , r t+1 , e t+1 )∈g}

[0105] It means in e t With e t+1 There is a relationship t+1 When it is used as an action A t ;

[0106] In order to avoid infinite jumps in service fusion and query process, the stop action STOP is executed when the query step number t increases to the maximum step number T;

[0107] S4-3, using P = {P1, P2, ..., P r} to represent the state transition set in the service fusion and query process, where P r Indicates that from the current state s t Reach the next state s through the service relationship r t+1 ; The complete process can be expressed as follows:

[0108]

[0109] S4-4, in order to better encourage agents to achieve service integration of different types and sources, accuracy rewards, distance rewards and diversity rewards are designed respectively:

[0110] S4-4-1, Accuracy Reward R d The purpose is to encourage intelligent acquisition of accurate related service entities, thereby achieving service integration of different types and sources; the specific process is shown in the following formula:

[0111]

[0112] When the correct service entity is found, the reward is 1; when the service fusion is wrong and a service entity that does not match the query is found, the reward is two service entities e t ,e d Semantic cosine similarity to avoid the problem of too sparse rewards;

[0113] S4-4-2, Distance Bonus R step The purpose is to encourage the agent to complete service fusion under a limited number of jump steps T to avoid excessive service fusion; the specific process is shown in the following formula:

[0114]

[0115] Where k represents the number of jump steps used in the current service fusion process. If it is less than the specified compensation range, a reward is given, and if it exceeds the range, a penalty is given to help the model converge better;

[0116] S4-4-3, Diversity Reward R divThe purpose is to encourage the agent to explore more action spaces and strike a balance between exploration and utilization, so as to ensure that the knowledge graph can be fully explored during the learning process, and the existing knowledge can be used to maximize the cumulative reward and avoid the local optimal service fusion of the system; the specific process is shown in the following formula:

[0117]

[0118] Where L represents the number of known reasoning paths, and represents the state transition probability distribution in the state transition set P in the service fusion and query process, μ 2 and Represents the variance of the state transition probability distribution to balance the dimensions of the two;

[0119] S4-4-4, the final reward function is as follows:

[0120] R=R d +R step +R div .

[0121] Step S5 is as follows:

[0122] S5-1, using knowledge graph Construct missing triples (e s ,r,? ),(e s ,? ,e d ), (?,r,e d ) as a service request, where “?” represents the service entity or service relationship that needs to be fused; the training goal is to infer the missing service entity or relationship element through a relationship path that is shorter or equal to k hops, where k is an integer not less than 1;

[0123] S5-2, construct the knowledge graph subset feature expression Z using the knowledge graph subset g and the service request feature q, the formula is: Z = Concat[g:q], where Concat represents feature connection;

[0124] S5-3, Use feedforward neural network to implement policy network π θ , the policy network inputs the feature expression Z of the knowledge graph subset and outputs the next action with the highest executable probability to realize the interaction between the knowledge graph and the agent; as shown in the following formula:

[0125] π θ (a t |s t )=softmax(A t (W2Relu(Z)))

[0126] Where Relu represents the activation function, Denotes the set A at step t t Any one of;

[0127] S5-4, for the strategy network π θ To train:

[0128] During the training process of reinforcement learning, the agent interacts with the multimodal cross-border property service data knowledge graph. The agent observes the current state of the knowledge graph and selects actions to perform based on the current state. The behavior of the agent is determined by the policy network, which defines the rules for selecting actions in a given state. After the agent performs an action, the environment obtains the reward of the reward mechanism based on the agent's action and current state.

[0129] The interaction between the agent and the knowledge graph is ongoing. Each step of the interaction includes the following process: the agent observes the current state, selects an action based on the current strategy, and after executing the action, the environment moves to a new state and gives a reward. The agent updates its strategy and action set based on the reward and the new state.

[0130] In the process of interaction between the agent and the knowledge graph, the agent will collect a lot of interaction experience, including observed states, executed actions, and rewards obtained; the learning algorithm uses this experience to continuously update the agent's strategy and value function so that the agent can achieve better performance;

[0131] Finally, the following formula is used as the training goal to maximize the cumulative reward of the model and obtain the optimal strategy network:

[0132]

[0133] Where E represents expectation, e s ,r,e d is a triple in the knowledge graph, representing the starting entity, relationship, and target entity in the graph respectively; π θ is a policy network based on parameters θ, which determines the probability of choosing an action at each time step; a1,…,a t is based on the policy network π θ The action sequence selected at each time step; [R(S t |e s ,r)] represents the agent in a given initial service entity e s , in the case of service relationship r, according to the policy network π θ Select an action sequence a1,…,a t The final state S obtained after t The reward J(θ) represents the agent's strategy network π under all possible starting entity, relationship and target entity triples in the graph.θ The expected cumulative reward of the action is selected at each time step; finally, by maximizing J(θ), the cumulative reward of the model is maximized to obtain the optimal policy network.

[0134] Step S6 is as follows:

[0135] S6-1, in this embodiment, the agent operates in the following order during the training process:

[0136] Service request: The agent first observes the current environment, that is, obtains relevant information from the smart community knowledge graph to understand the current status of the community and the needs of residents. This may involve data and knowledge in multiple fields such as property management, community services, and public safety.

[0137] Select actions to perform: Based on the observed environmental information, the agent selects the actions to perform according to the current state. For example, if an abnormal environment in a community area is detected, the agent may choose to arrange patrols or increase security police force.

[0138] Execute action: The agent executes the selected action, that is, takes corresponding actions in the smart community according to the instructions formulated by the strategy. This may include dispatching personnel, sending alarms, optimizing services, etc.

[0139] Get rewards: After performing an action, the agent interacts with the environment and may receive rewards or feedback. Rewards can be positive, such as solving a community problem or improving service quality, or negative, such as failing to effectively respond to an emergency. The agent evaluates the effectiveness of the action based on the reward information and uses it to update the strategy.

[0140] S6-2, this process is continuous. The agent selects and executes actions based on the continuously updated status information and continuously optimizes its behavior strategy based on the feedback from the environment to provide better service integration to support the operation and management of the smart community.

[0141] Accordingly, the present invention also provides a service fusion system based on multi-jump multi-modal knowledge graph, such as Figure 2 As shown, including:

[0142] Feature encoding unit 1, used to input multimodal cross-border property service data knowledge graph G and service request Q, use the pre-trained model to encode the service request Q and the multimodal data in the knowledge graph G, and extract the service request feature q, entity feature e, relationship feature r, text description feature td, image semantic feature p, and video semantic feature v;

[0143] The feature fusion unit 2 is used to perform matrix multiplication on the entity feature e, text description feature td, image semantic feature p, and video semantic feature v, so as to map them to the same semantic space and obtain the service entity feature The service entity feature set of all service entities is recorded as

[0144] Feature filtering unit 3, used to filter the service entity feature set Input into the Transformer model, and use the service request feature q to filter the knowledge graph G, narrow the service search scope, and obtain the knowledge graph subset g of the knowledge graph G;

[0145] A reinforcement learning model construction unit 4 is used to construct a service fusion reinforcement learning model within the scope of the knowledge graph subset g, and define the state, action, state transfer and reward mechanism of reinforcement learning in the service fusion process for the intelligent agent that executes the service fusion strategy;

[0146] An offline training unit 5 is used to perform offline training on the agent using the service fusion reinforcement learning model and the knowledge graph G to form a policy network after offline training, wherein the input of the policy network is a knowledge graph subset feature expression Z composed of the knowledge graph subset g and the service request feature q;

[0147] The online training and service fusion unit 6 is used to update the offline trained policy network online during the service query process. During the updating process, the intelligent agent, under the guidance of the reward mechanism, continuously establishes connections between different service entities, thereby completing the relationship between different service entities in the multimodal cross-border property service data knowledge graph G, and then realizing service fusion.

[0148] Therefore, the present invention proposes an effective service fusion model, which overcomes the problems of low reasoning efficiency and difficulty in cross-modal association by performing feature extraction and fine-grained fusion of structured data and multimodal auxiliary data, and adopting an attention-based representation method to generate multimodal complementary representations. This integration method effectively overcomes the shortcomings of traditional models in capturing cross-domain relationships and contexts, improves the accuracy and efficiency of knowledge graph reasoning, and thus provides a basis for applications in the field of service fusion.

[0149] The above is a detailed introduction to a service fusion method and system based on a multi-jump multimodal knowledge graph provided by an embodiment of the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A service fusion method based on multi-jump multimodal knowledge graph, characterized in that: The method comprises: Input the multimodal cross-border property service data knowledge graph G and service request Q, use the pre-trained model to encode the service request Q and the multimodal data in the knowledge graph G, and extract the service request feature q, entity feature e, relationship feature r, text description feature td, image semantic feature p, and video semantic feature v; The entity feature e, text description feature td, image semantic feature p, and video semantic feature v are matrix-multiplied to map them to the same semantic space to obtain the service entity feature The service entity feature set of all service entities is recorded as The service entity feature set Input into the Transformer model, and use the service request feature q to filter the knowledge graph G, narrow the service search scope, and obtain the knowledge graph subset g of the knowledge graph G; Constructing a service fusion reinforcement learning model within the scope of the knowledge graph subset g, and defining the state, action, state transfer and reward mechanism of reinforcement learning in the service fusion process for the intelligent agent that executes the service fusion strategy; The agent is trained offline using the service fusion reinforcement learning model and the knowledge graph G to form a policy network after offline training, the input of which is a knowledge graph subset feature expression Z consisting of the knowledge graph subset g and the service request feature q; During the service query process, the offline trained policy network is updated online. During the updating process, the intelligent agent, under the guidance of the reward mechanism, continuously establishes connections between different service entities, thereby completing the relationship between different service entities in the multimodal cross-border property service data knowledge graph G, and then realizing service integration.

2. A service fusion method based on multi-jump multi-modal knowledge graph as claimed in claim 1, characterized in that: The input multimodal cross-border property service data knowledge graph G and service request Q are encoded using the pre-trained model to extract the service request feature q, entity feature e, relationship feature r, text description feature td, image semantic feature p, and video semantic feature v, specifically: The multimodal cross-border property service data knowledge graph G is represented as where ε is the set of entities, is a set of semantic relations. is a set of triplets present in G; Use the pre-trained BERT model to encode the text data in the knowledge graph G to obtain entity features e, relationship features r, and text description features td, and encode the service request Q to obtain the service request feature q; Use the pre-trained convolutional neural network ResNet to encode the image data in the knowledge graph G to obtain the image semantic feature p; Use the pre-trained convolutional neural network C3D to encode the video data in the knowledge graph G to obtain the video semantic features v.

3. The service fusion method based on multi-jump multi-modal knowledge graph according to claim 1, characterized in that: The entity feature e, text description feature td, image semantic feature p, and video semantic feature v are matrix-multiplied to map them to the same semantic space to obtain the service entity feature The service entity feature set of all service entities is recorded as Specifically: Service Entity Characteristics Calculated by the following formula: in is the learnable weight matrix, represents a real number field with 4d rows and d columns, where d is the feature dimension; by performing matrix multiplication on the entity feature e, text description feature td, image semantic feature p, and video semantic feature v, they are mapped to the same semantic space, and finally a service entity feature set is obtained Where m represents the total number of service entities.

4. The service fusion method based on multi-jump multi-modal knowledge graph according to claim 1, characterized in that: The service entity feature set Input into the Transformer model, and use the service request feature q to filter the knowledge graph G, narrow the service search scope, and obtain the knowledge graph subset g of the knowledge graph G, specifically: The knowledge graph subset g is calculated by the following formula: g=FFN(Concat i∈1,…,h head i ) in K i ,V i After W i K ,W i V The variants of the original information obtained after linear transformation of three learnable matrices, h represents the number of heads of multi-head attention, FFN and Concat represent deep network and feature connection respectively; K, V are the query, key and value in the attention mechanism, is the characteristic dimension of K, mainly to prevent With K i The dot product value of pos is too large; q and pos e It is the position code of the service request feature q and the entity feature e.

5. The service fusion method based on multi-jump multi-modal knowledge graph according to claim 1, characterized in that: A service fusion reinforcement learning model is constructed within the scope of the knowledge graph subset g, and the state, action, state transfer and reward mechanism of reinforcement learning in the service fusion process are defined for the intelligent agent that executes the service fusion strategy, specifically: Use S = {s1,s2,…,s t } to represent the state space in the service request process, where Describes the status of the service fusion process, t represents the number of multi-level jumps between the relationships between entities in the knowledge graph, and e t is the entity visited at step t, e s and r q They are service entities and service relationships. and ε t Indicates e t Adjacent entities and relationships; Use A={A1,A2,…,A t , STOP} to represent the action space in the service request process, where STOP means the action is terminated, A t Indicates that t The edge set of the remaining entities connected, and extend the service path to the next service entity e t+1 ; The complete process can be expressed as follows: A t ={(e t+1 ,r t+1 )|(e t ,r t+1 ,e t+1 )∈g} It means in e t With e t+1 There is a relationship t+1 When it is taken as an action A t ; In order to avoid infinite jumps in service fusion and query process, the stop action STOP is executed when the query step number t increases to the maximum step number T; Use P = {P1, P2, ..., P r } to represent the state transition set in the service fusion and query process, where P r Indicates that from the current state s t Reach the next state s through the service relationship r t+1 ; The complete process can be expressed as follows: In order to better encourage agents to achieve service integration of different types and sources, accuracy rewards, distance rewards, and diversity rewards are designed respectively: Accuracy reward R d The purpose is to encourage intelligent acquisition of accurate related service entities, thereby achieving service integration of different types and sources; the specific process is shown in the following formula: When the correct service entity is found, the reward is 1; when the service fusion is wrong and a service entity that does not match the query is found, the reward is two service entities e t ,e d Semantic cosine similarity to avoid the problem of too sparse rewards; Distance Reward R step The purpose is to encourage the agent to complete service fusion under a limited number of jump steps T to avoid excessive service fusion; the specific process is shown in the following formula: Where k represents the number of jump steps used in the current service fusion process. If it is less than the specified compensation range, a reward is given, and if it exceeds the range, a penalty is given to help the model converge better; Diversity Rewards div The purpose is to encourage the agent to explore more action spaces and strike a balance between exploration and utilization, so as to ensure that the knowledge graph can be fully explored during the learning process, and the existing knowledge can be used to maximize the cumulative reward and avoid the local optimal service fusion of the system; the specific process is shown in the following formula: Where L represents the number of known reasoning paths, and represents the state transition probability distribution in the state transition set P in the service fusion and query process, μ 2 and Represents the variance of the state transition probability distribution to balance the dimensions of the two; The final reward function is as follows: R=R d +R step +R div 。 6. A service fusion method based on multi-jump multimodal knowledge graph as described in claim 2 or 5, characterized in that: The agent is trained offline using the service fusion reinforcement learning model and the knowledge graph G to form a policy network after offline training. The input of the policy network is the knowledge graph subset feature expression Z composed of the knowledge graph subset g and the service request feature q, which is specifically: Using the knowledge graph Construct missing triples (e s ,r,? ),(e s ,? ,e d ), (?,r,e d ) as a service request, where "?" represents the service entity or service relationship that needs to be fused; the training goal is to infer the missing service entity or relationship element through a relationship path that is shorter or equal to k hops, where k is an integer not less than 1; The knowledge graph subset g and the service request feature q are used to construct a knowledge graph subset feature expression Z, the formula is: Z = Concat [g: q], where Concat represents feature connection; Use a feedforward neural network to implement the policy network π θ , the policy network inputs the feature expression Z of the knowledge graph subset and outputs the next action with the highest executable probability to realize the interaction between the knowledge graph and the agent; as shown in the following formula: π θ (and t |with t )=softmax(A t (W2Relu(Z))) Where Relu represents the activation function, a t ∈A t Denotes the set A at step t t Any one of; For the policy network π θ To train: The following formula is used as the training objective to maximize the cumulative reward of the model and obtain the optimal strategy network: Where E represents expectation, e s ,r,e d is a triple in the knowledge graph, representing the starting entity, relationship, and target entity in the graph respectively; π θ is a policy network based on parameters θ, which determines the probability of choosing an action at each time step; a 1, …,a t is based on the policy network π θ The action sequence selected at each time step; [R(S t |e s ,r)] represents the agent in a given initial service entity e s , in the case of service relationship r, according to the policy network π θ Select action sequence a 1, …,a t The final state S obtained after t The reward J(θ) represents the agent's strategy network π under all possible starting entity, relationship and target entity triples in the graph. θ The expected cumulative reward of the action is selected at each time step; finally, by maximizing J(θ), the cumulative reward of the model is maximized to obtain the optimal policy network.

7. A service fusion system based on multi-jump multi-modal knowledge graph, characterized in that: The system comprises: The feature encoding unit is used to input the multimodal cross-border property service data knowledge graph G and the service request Q, use the pre-trained model to encode the service request Q and the multimodal data in the knowledge graph G, and extract the service request feature q, entity feature e, relationship feature r, text description feature td, image semantic feature p, and video semantic feature v; The feature fusion unit is used to perform matrix multiplication on the entity feature e, text description feature td, image semantic feature p, and video semantic feature v, so as to map them to the same semantic space and obtain the service entity feature The service entity feature set of all service entities is recorded as A feature filtering unit is used to filter the service entity feature set Input into the Transformer model, and use the service request feature q to filter the knowledge graph G, narrow the service search scope, and obtain the knowledge graph subset g of the knowledge graph G; A reinforcement learning model construction unit, used to construct a service fusion reinforcement learning model within the scope of the knowledge graph subset g, and define the state, action, state transfer and reward mechanism of reinforcement learning in the service fusion process for the intelligent agent that executes the service fusion strategy; An offline training unit, used to perform offline training on the agent using the service fusion reinforcement learning model and the knowledge graph G to form a policy network after offline training, wherein the input of the policy network is a knowledge graph subset feature expression Z consisting of the knowledge graph subset g and the service request feature q; The online training and service fusion unit is used to update the offline trained policy network online during the service query process. During the updating process, the intelligent agent, under the guidance of the reward mechanism, continuously establishes connections between different service entities, thereby completing the relationship between different service entities in the multimodal cross-border property service data knowledge graph G, and then realizing service fusion.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images