Knowledge database construction method and device, electronic equipment and storage medium

Abstract

This invention provides a method for constructing a knowledge database. The method includes: acquiring a dataset of images to be processed, comprising multiple images to be processed; performing entity recognition on the images to be processed to obtain at least one entity combination of the images, the entity combination including two target entities; constructing at least one knowledge data corresponding to the entity combination based on the two target entities; constructing a knowledge dataset of the images to be processed based on the at least one knowledge data corresponding to the entity combination, with each image to be processed corresponding to one knowledge dataset; and constructing a knowledge database corresponding to the dataset of images to be processed based on the knowledge datasets of all images to be processed. In this invention, the knowledge data not only contains information about the entities themselves but also implicitly contains relationships between entities. The constructed knowledge database expands the dimensions and accuracy of image recognition, thereby improving the accuracy of image recognition results.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and in particular to a method, apparatus, electronic device, and storage medium for constructing a knowledge database. Background Technology

[0002] With the rapid development of artificial intelligence technology, image recognition technology has been widely applied in various fields. A knowledge base is a database storing rich background information, specifically designed to provide additional related information for identified image entities, enhancing the depth of the image recognition system's understanding of the content. Current image recognition technologies are typically limited to simple identification and description of image content, failing to fully capture the multidimensional information within images. This results in simplistic descriptions and insufficient content richness, leading to inaccurate image retrieval. Therefore, existing image recognition technologies suffer from insufficient accuracy in image recognition results. Summary of the Invention

[0003] This invention provides a method for constructing a knowledge database, aiming to address the problem of insufficient accuracy in existing image recognition technologies. By performing entity recognition on the image to be processed, it can comprehensively capture the information in the image, not limited to a single entity, but identifying multiple target entities and forming entity combinations. Based on the identified entity combinations, at least one knowledge database can be constructed. This knowledge not only includes information about the entities themselves but also implicitly contains relationships between entities, generating richer and more accurate descriptions. The constructed knowledge database expands the dimensions and accuracy of image recognition, thereby improving the accuracy of image recognition results.

[0004] In a first aspect, embodiments of the present invention provide a method for constructing a knowledge database, the method comprising the following steps:

[0005] Obtain a dataset of images to be processed, which includes multiple images to be processed;

[0006] Entity recognition is performed on the image to be processed to obtain at least one entity combination of the image to be processed, the entity combination including two target entities;

[0007] Based on the two target entities, at least one knowledge data corresponding to the entity combination is constructed;

[0008] Based on at least one of the knowledge data corresponding to the entity combination, a knowledge dataset of the image to be processed is constructed, and each image to be processed corresponds to one knowledge dataset.

[0009] Based on the knowledge dataset of all the images to be processed, a knowledge database corresponding to the image dataset to be processed is constructed.

[0010] Optionally, the step of performing entity recognition on the image to be processed to obtain at least one combination of entities in the image to be processed includes:

[0011] Entity recognition is performed on the image to be processed to obtain multiple target entities in the image to be processed;

[0012] Among multiple target entities, every two target entities are combined to obtain at least one entity combination of the image to be processed.

[0013] Optionally, the step of performing entity recognition on the image to be processed to obtain multiple target entities of the image to be processed includes:

[0014] Visual entities in the image to be processed are obtained by performing visual recognition on the image using a multimodal large model.

[0015] If there is no text region in the image to be processed, then multiple target entities in the image to be processed are determined based on the visual entities.

[0016] If there is a text region in the image to be processed, then extract the text image containing the text from the image to be processed.

[0017] Perform text recognition on the text image to obtain the text entities in the image to be processed;

[0018] Based on the visual entities and the text entities, multiple target entities in the image to be processed are identified.

[0019] Optionally, the step of constructing at least one knowledge data corresponding to the entity combination based on the two target entities includes:

[0020] Relationship identification is performed on two target entities in the entity combination to obtain at least one entity relationship between the two target entities;

[0021] Based on the entity combination and at least one entity relationship, at least one knowledge data corresponding to the entity combination is constructed, and each knowledge data corresponds to one entity relationship.

[0022] Optionally, the step of performing relationship identification on the target entity pair to obtain at least one entity relationship corresponding to the target entity includes:

[0023] In the image to be processed, relationship reasoning is performed on the two target entities in the target entity pair to obtain at least one association relationship between the two target entities;

[0024] Based on at least one association between two target entities, at least one entity relationship corresponding to each target entity is determined, and each association corresponds to one entity relationship.

[0025] Optionally, the step of constructing at least one knowledge data corresponding to the entity combination based on the two target entities includes:

[0026] Associating two target entities in the entity combination with at least one entity relation corresponding to the entity combination yields at least one knowledge triplet corresponding to the entity combination, with each entity relation corresponding to one knowledge triplet.

[0027] The knowledge triple is determined as the knowledge data corresponding to the entity combination.

[0028] Optionally, the step of constructing a knowledge database corresponding to the dataset of images to be processed based on the knowledge dataset of all the images to be processed includes:

[0029] Based on the knowledge dataset of all the images to be processed, a node graph is constructed. In the node graph, each image to be processed is a separate node, each target entity is a separate node, and two nodes belonging to the same knowledge dataset are connected by connecting edges.

[0030] Based on the node graph, and using entity information as the query condition, a knowledge database corresponding to the image dataset to be processed is constructed.

[0031] Secondly, embodiments of the present invention also provide a knowledge database construction apparatus, the knowledge database construction apparatus comprising:

[0032] The acquisition module is used to acquire a dataset of images to be processed, which includes multiple images to be processed.

[0033] The processing module is used to perform entity recognition on the image to be processed to obtain at least one entity combination of the image to be processed, wherein the entity combination includes two target entities;

[0034] The first construction module is used to construct at least one knowledge data corresponding to the entity combination based on the two target entities;

[0035] The second construction module is used to construct a knowledge dataset for the image to be processed based on at least one knowledge data corresponding to the entity combination, wherein each image to be processed corresponds to one knowledge dataset.

[0036] The third construction module is used to construct a knowledge database corresponding to the dataset of images to be processed based on the knowledge dataset of all the images to be processed.

[0037] Thirdly, embodiments of the present invention provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps in the knowledge database construction method provided in embodiments of the present invention.

[0038] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in the knowledge database construction method provided in the embodiments of the present invention.

[0039] In this embodiment of the invention, a dataset of images to be processed is obtained, which includes multiple images to be processed. Entity recognition is performed on the images to be processed to obtain at least one entity combination of the images, where each entity combination includes two target entities. Based on the two target entities, at least one knowledge data corresponding to the entity combination is constructed. Based on the at least one knowledge data corresponding to the entity combination, a knowledge dataset of the images to be processed is constructed, with each image to be processed corresponding to one knowledge dataset. Based on the knowledge datasets of all images to be processed, a knowledge database corresponding to the image dataset is constructed. By performing entity recognition on the images to be processed, information in the images can be comprehensively captured, not limited to a single entity, but identifying multiple target entities and forming entity combinations. Based on the identified entity combinations, at least one knowledge data can be constructed. This knowledge not only includes information about the entities themselves but also implicitly contains relationships between entities, generating richer and more accurate descriptions. The constructed knowledge database expands the dimensions and accuracy of image recognition, thereby improving the accuracy of image recognition results. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a flowchart of a knowledge database construction method provided in an embodiment of the present invention;

[0042] Figure 2 This is a schematic diagram of an image preprocessing process provided by an embodiment of the present invention;

[0043] Figure 3 This is a schematic diagram illustrating the construction of a knowledge database according to an embodiment of the present invention;

[0044] Figure 4 This is a schematic diagram illustrating the maintenance of a knowledge database according to an embodiment of the present invention;

[0045] Figure 5 This is a schematic diagram of the structure of a knowledge database construction device provided in an embodiment of the present invention;

[0046] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0047] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0048] like Figure 1 As shown, Figure 1 This is a flowchart of a knowledge database construction method provided by an embodiment of the present invention. The knowledge database construction method includes the following steps:

[0049] 101. Obtain the dataset of images to be processed.

[0050] In this embodiment of the invention, the aforementioned knowledge database construction method can be applied to an image recognition platform. This image recognition platform can be built on a server or a distributed server, and includes an image interface (for user uploads), a knowledge database, and a knowledge database construction program. The image interface can be used to acquire image data, which can be continuous frame images or video stream data. The knowledge database construction program can be used to implement each step of the knowledge database construction method. The knowledge database is specifically designed to provide additional association information for the identified image entities, thereby improving the image recognition system's understanding of the content.

[0051] The aforementioned image dataset contains multiple images, each containing multiple identifiable entities. This image dataset can be user-uploaded or downloaded from a user-provided download link.

[0052] In one possible embodiment, the image dataset to be processed can be obtained through user uploads or online downloads. This image dataset includes initial images. The initial images are then preprocessed to obtain the images to be processed. The preprocessing described above can be referenced... Figure 2 , Figure 2 This is a schematic flowchart of an image preprocessing method provided by an embodiment of the present invention. The specific image preprocessing is as follows:

[0053] Considering that scaling smaller images can lead to information loss and distortion, ultimately resulting in poorer recognition performance, we can perform conditional filtering on the initial images in the image dataset. We can determine whether the width or height of the initial image is greater than or equal to a preset value, and remove images that do not meet the conditions, while images that meet the conditions can proceed to subsequent steps.

[0054] The above filtering conditions are used to select and retain initial images whose width and height are greater than or equal to a preset number of pixels, and to remove initial images whose width and height are less than a preset number of pixels. For example, it is used to select and retain initial images whose width and height are greater than or equal to 100 pixels, and to remove initial images whose width and height are less than 100 pixels.

[0055] Images that meet the criteria from the image dataset are scaled proportionally. The images are scaled along their longer side, ensuring the scaled image maintains the same aspect ratio as the original. For example, assuming the input requirement is an image with a height × width of 224 × 224 pixels, if the longer side of the image is 500 pixels, the scaling ratio is 224 / 500 ≈ 0.448. Therefore, the original width of 300 pixels will be scaled to 300 × 0.448 ≈ 134 pixels. This ensures that the main objects in the image are not distorted.

[0056] It should be noted that the image after proportional scaling may still not meet the preset values, so further filtering can be performed. Images that meet the filtering criteria will then undergo the final processing.

[0057] For images that meet the selection criteria, if their width or height is smaller than the model's input size, blank areas need to be padded around the image to make it fit the model's input size. To standardize the image pixel values ​​and improve the model's training and inference efficiency, this embodiment maps all pixel values ​​to between 0 and 1. This makes the weight changes during model training smoother and the optimization process easier to converge. The finally processed images are stored in a dataset to obtain the dataset of images to be processed; the final processed images are the images to be processed.

[0058] In this embodiment, images that do not meet the criteria are removed through a conditional filtering step, ensuring that only images with sufficient pixel information are processed. Simultaneously, proportional scaling and filling of blank areas ensure the integrity of the image content, preventing poor recognition performance due to lost or distorted image information.

[0059] 102. Perform entity recognition on the image to be processed to obtain at least one combination of entities in the image.

[0060] In this embodiment of the invention, the content of the image to be processed can be converted into structured data using a multimodal large model, and integrated to form an entity combination, which includes two target entities. The aforementioned multimodal large model is an artificial intelligence model capable of simultaneously processing and comprehensively analyzing multiple different types of data (such as text, images, audio, video, etc.).

[0061] Specifically, entity recognition can be performed on the image to be processed using a multimodal large model. The entities that the multimodal large model can output can include people's names, place names, organization names, object names, scene names, etc. The entities output by the multimodal large model are used as target entities.

[0062] When there are multiple target entities, the multiple target entities are combined in pairs to obtain a corresponding number of entity combinations. For an image to be processed, there can be at least one entity combination. When there are multiple entity combinations, any two entity combinations are different.

[0063] 103. Based on two target entities, construct at least one knowledge data corresponding to the entity combination.

[0064] In this embodiment of the invention, for an entity combination, the degree of association between two target entities can be calculated. Based on the degree of association between the two target entities, the two target entities are associated, and the successfully associated entity combination is determined as a knowledge data.

[0065] Alternatively, the entity relationship between two target entities can be inferred. By analyzing this relationship, the two target entities can be associated, and the successfully associated entity combination can be identified as a knowledge data point. Considering that different entity relationships may exist between the same two entities, an entity combination can correspond to at least one knowledge data point. For example, in the entity combination (man, clothes), the entity relationships could be "wear," "take off," "take," "take," etc.

[0066] 104. Based on at least one knowledge data corresponding to the entity combination, construct the knowledge dataset of the image to be processed.

[0067] In this embodiment of the invention, for each image to be processed, there may be at least one entity combination, and each entity combination may have at least one knowledge data. Therefore, the image to be processed may have at least one knowledge data. The knowledge data belonging to the image to be processed is added to the corresponding set to obtain the knowledge dataset of the image to be processed.

[0068] The aforementioned knowledge dataset is used to store the knowledge data corresponding to the image to be processed. When building the knowledge database later, the corresponding knowledge data can be extracted from the knowledge dataset corresponding to the image to be processed.

[0069] 105. Based on the knowledge dataset of all images to be processed, construct the knowledge database corresponding to the dataset of images to be processed.

[0070] In this embodiment of the invention, after obtaining the knowledge dataset of all images to be processed, the images to be processed can be associated with the corresponding knowledge dataset. Specifically, the knowledge data in the knowledge dataset is associated with the corresponding image to be processed to form a storable data entry. This storable data is then stored in the corresponding data table or database to obtain the corresponding knowledge database. Image recognition using the knowledge database can retrieve knowledge data related to the image and images related to the knowledge. The knowledge database can be updated using new knowledge data.

[0071] Alternatively, the images to be processed and their corresponding knowledge datasets can be node-based to construct a graph database, such as the Neo4j graph database. Based on the entity information in the graph database, queries are performed in the knowledge base to obtain relevant results. Subsequently, the query results are associated with the existing entity information and batch-updated to the entity nodes in the graph database to achieve content expansion.

[0072] In this embodiment of the invention, a dataset of images to be processed is obtained, which includes multiple images to be processed. Entity recognition is performed on the images to be processed to obtain at least one entity combination of the images, where each entity combination includes two target entities. Based on the two target entities, at least one knowledge data corresponding to the entity combination is constructed. Based on the at least one knowledge data corresponding to the entity combination, a knowledge dataset of the images to be processed is constructed, with each image to be processed corresponding to one knowledge dataset. Based on the knowledge datasets of all images to be processed, a knowledge database corresponding to the image dataset is constructed. By performing entity recognition on the images to be processed, information in the images can be comprehensively captured, not limited to a single entity, but identifying multiple target entities and forming entity combinations. Based on the identified entity combinations, at least one knowledge data can be constructed. This knowledge not only includes information about the entities themselves but also implicitly contains relationships between entities, generating richer and more accurate descriptions. The constructed knowledge database expands the dimensions and accuracy of image recognition, thereby improving the accuracy of image recognition results.

[0073] It is understood that in the specific implementation of this application, data such as image data, knowledge data, and user data are involved. When the embodiments in this application are applied to specific products or technologies, user permission or consent is required. Furthermore, the collection, use, and processing of related data, as well as the training, deployment, and invocation of algorithm models, must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0074] Optionally, in the step of performing entity recognition on the image to be processed to obtain at least one combination of entities in the image to be processed, entity recognition can be performed on the image to be processed to obtain multiple target entities in the image to be processed; among the multiple target entities, every two target entities are combined to obtain at least one combination of entities in the image to be processed.

[0075] In this embodiment of the invention, entity recognition can be performed on the image to be processed using a multimodal large model to obtain multiple target entities from the image. These target entities can be entities such as person names, place names, organization names, object names, and scene names.

[0076] When there are multiple target entities, the multiple target entities are combined in pairs to obtain a corresponding number of entity combinations. For an image to be processed, there can be at least one entity combination. When there are multiple entity combinations, any two entity combinations are different.

[0077] In one possible embodiment, entities can be combined based on features such as distance, size, and color. For example, two entities that are less than a preset distance can be combined, two entities that are similar in size can be combined, and two entities that are similar or opposite in color can be combined.

[0078] Optionally, in the step of performing entity recognition on the image to be processed to obtain multiple target entities in the image to be processed, a multimodal large model can be used to perform visual recognition on the image to be processed to obtain visual entities in the image to be processed; if there is no text region in the image to be processed, then multiple target entities in the image to be processed are determined based on the visual entities; if there is a text region in the image to be processed, then the text image containing the text is extracted from the image to be processed; text recognition is performed on the text image to obtain text entities in the image to be processed; and multiple target entities in the image to be processed are determined based on the visual entities and the text entities.

[0079] In this embodiment of the invention, a multimodal large model can be used to perform visual recognition on the image to be processed, and entities in the image to be processed can be identified through visual features such as shape, color, and texture. These entities are visual entities.

[0080] After identifying all visual entities in the image to be processed, all visual entities can be used as target entities, thus obtaining target recognition with the same number of visual entities.

[0081] In the process of visual recognition of images by a multimodal large model, it is also possible to identify whether there are text regions in the image. If no text regions exist, multiple target entities in the image are directly determined based on the previously identified visual entities. If text regions exist, the corresponding images (small images) can be extracted. Specifically, image processing techniques such as image segmentation and edge detection can be used to extract text regions from the image to ensure accurate extraction. After extracting the text images, the text regions can be provided to an OCR text recognition model to accurately extract the text content within those regions. By recognizing text entities in the image using OCR technology and associating the entities in the text with the entities in the image, complete image-text knowledge data is formed, expanding the dimensions and accuracy of image recognition.

[0082] In one possible implementation, the visual entities identified by the multimodal large model may contain textual descriptions, which can further enrich the entity's description. Using the visual entity location information returned by the multimodal large model, the corresponding image region can be provided to the OCR text recognition model to accurately extract the text content within that region. Combining this textual information with the entity combinations generated by the multimodal large model can more comprehensively improve the image's knowledge data, enhancing the understanding and richness of the image's content description.

[0083] In the multimodal large model, the visual entity identified is "aluminum can", and the OCR text recognition shows that the description on the aluminum can is "cola". Therefore, the entity combination can be (aluminum can, cola).

[0084] Optionally, in the step of constructing at least one knowledge data corresponding to the entity combination based on two target entities, the two target entities in the entity combination can be subjected to relationship identification to obtain at least one entity relationship between the two target entities; based on the entity combination and at least one entity relationship, at least one knowledge data corresponding to the entity combination can be constructed, and each knowledge data corresponds to one entity relationship.

[0085] Optionally, in the step of performing relationship identification on the target entity pair to obtain at least one entity relationship corresponding to the target entity, relationship reasoning can be performed on the two target entities in the target entity pair in the image to be processed to obtain at least one association relationship between the two target entities; based on at least one association relationship between the two target entities, at least one entity relationship corresponding to the target entity is determined, and each association relationship corresponds to one entity relationship.

[0086] In this embodiment of the invention, relation recognition aims to identify and extract semantic relationships that exist between entities. Within a semantic context, the purpose of relation recognition is to discover at least one entity relationship between two target entities. These entity relationships can be spatial relationships (such as "next to", "above"), attribute relationships (such as "is the color of", "has the shape of", "is the brand of"), or more complex logical relationships (such as "because of", "if", "then").

[0087] Entity relationships can be inferred based on a multimodal large model to obtain the entity relationships between two target entities. Specifically, the input prompt for the multimodal large model can be: (image to be processed), entity 1 and entity 2. Please determine which entity relationships entity 1 and entity 2 have in the image to be processed and output the entity relationships.

[0088] Optionally, in the step of constructing at least one knowledge data corresponding to an entity combination based on two target entities, the two target entities in the entity combination can be associated with at least one entity relation corresponding to the entity combination to obtain at least one knowledge triplet corresponding to the entity combination, with each entity relation corresponding to one knowledge triplet; the knowledge triplet is determined as the knowledge data corresponding to the entity combination.

[0089] In this embodiment of the invention, the aforementioned knowledge triples can be represented as (entity 1, entity relation, entity 2). A multimodal large model converts the preprocessed image content into structured data and outputs image-related knowledge triples. The model identifies entities that may be output, including names of people, places, organizations, objects, and scenes, and builds the model based on contextual information. It also identifies the types or categories these entities belong to. Later, when expanding the image content through a knowledge base, the type or category of the entity allows for more accurate content matching.

[0090] Multimodal large-scale model training typically doesn't directly address the specific identities of individuals (such as names or positions), but rather relies on training with massive amounts of image-text pairing data. If a person in an image is a public figure (such as a well-known actor or politician), the model might infer their profession or field (such as "actor" or "politician"). Each image can correspond to one or more knowledge triples, which comprehensively describe the objects in the image and their relationships. For example: (hand, holding, can). For instance, if the multimodal large-scale model identifies a can, and OCR text identifies the description on the can as Pepsi, then through relational reasoning within the large-scale model, the relationship between the can and Pepsi can be refined, generating a new knowledge triple (can, brand, Pepsi).

[0091] Optionally, the step of constructing a knowledge database corresponding to the knowledge dataset of all images to be processed can be carried out by constructing a node graph based on the knowledge dataset of all images to be processed. In the node graph, each image to be processed is treated as a separate node, each target entity is treated as a separate node, and two nodes belonging to the same knowledge data are connected by connecting edges. Based on the node graph, the knowledge database corresponding to the knowledge dataset of images to be processed is constructed using entity information as the query condition.

[0092] In this embodiment of the invention, the node graph described above is a data structure that can effectively represent the relationships between entities, facilitating querying and reasoning. When constructing the node graph, each image to be processed is considered a separate node, and each target entity extracted from the image is also considered a separate node.

[0093] Two nodes belonging to the same knowledge data are connected by a connecting edge. Here, "belonging to the same knowledge data" refers to two target entities that have been identified and associated in the previous steps. For example, if two entities, "person" and "chair," are identified in an image, and a "sitting" relationship is determined between them, then in the node graph, the "person" node and the "chair" node will be connected by a connecting edge representing the "sitting" relationship. Connecting edges can also be called relationship edges, and relationship edges may include additional attributes, such as a description of the relationship or a confidence level.

[0094] The storage method for the above-mentioned images and knowledge triples can be accomplished using the node and edge structure of a graph database (such as Neo4j).

[0095] Based on entity information in the graph database, the query conditions are used to match the results in the knowledge base. Then, the query results are associated with the existing entity information and updated in batches to the entity nodes in the graph database to achieve content expansion.

[0096] In one possible implementation, the knowledge base can be extended with image entity content and data persistence, such as... Figure 3 and Figure 4 As shown, Figure 3 This is a schematic diagram illustrating the construction of a knowledge database according to an embodiment of the present invention;

[0097] Figure 4 This is a schematic diagram illustrating the maintenance of a knowledge database according to an embodiment of the present invention. Combined with... Figure 3 and Figure 4 Explanation:

[0098] By constructing a node graph and using entity information as query conditions, we can effectively integrate and utilize the knowledge data extracted from the images to be processed, thereby building a powerful and flexible knowledge database and effectively improving the accuracy of image recognition.

[0099] like Figure 5 As shown, an embodiment of the present invention provides a knowledge database construction apparatus, which includes:

[0100] The acquisition module 501 is used to acquire a dataset of images to be processed, which includes multiple images to be processed.

[0101] The processing module 502 is used to perform entity recognition on the image to be processed to obtain at least one entity combination of the image to be processed, wherein the entity combination includes two target entities;

[0102] The first construction module 503 is used to construct at least one knowledge data corresponding to the entity combination based on the two target entities;

[0103] The second construction module 504 is used to construct a knowledge dataset of the image to be processed based on at least one knowledge data corresponding to the entity combination, wherein each image to be processed corresponds to one knowledge dataset.

[0104] The third construction module 505 is used to construct a knowledge database corresponding to the dataset of images to be processed based on the knowledge dataset of all the images to be processed.

[0105] Optionally, the processing module 502 is further configured to perform entity recognition on the image to be processed to obtain multiple target entities of the image to be processed; and to combine every two target entities to obtain at least one entity combination of the image to be processed.

[0106] Optionally, the processing module 502 is further configured to perform visual recognition on the image to be processed using a multimodal large model to obtain visual entities in the image to be processed; if there is no text region in the image to be processed, then based on the visual entities, determine multiple target entities in the image to be processed; if there is a text region in the image to be processed, then extract a text image containing text from the image to be processed; perform text recognition on the text image to obtain text entities in the image to be processed; and determine multiple target entities in the image to be processed based on the visual entities and the text entities.

[0107] Optionally, the first construction module 503 is further configured to identify the relationship between two target entities in the entity combination to obtain at least one entity relationship between the two target entities; and to construct at least one knowledge data corresponding to the entity combination based on the entity combination and at least one entity relationship, wherein each knowledge data corresponds to one entity relationship.

[0108] Optionally, the first construction module 503 is further configured to perform relationship reasoning on two target entities in the target entity pair in the image to be processed, to obtain at least one association relationship between the two target entities; and based on the at least one association relationship between the two target entities, to determine at least one entity relationship corresponding to the target entity, wherein each association relationship corresponds to one entity relationship.

[0109] Optionally, the first construction module 503 is further configured to associate two target entities in the entity combination with at least one entity relationship corresponding to the entity combination to obtain at least one knowledge triplet corresponding to the entity combination, wherein each entity relationship corresponds to one knowledge triplet; and to determine the knowledge triplet as the knowledge data corresponding to the entity combination.

[0110] Optionally, the third construction module 505 is further configured to construct a node graph based on the knowledge dataset of all the images to be processed, wherein each image to be processed is a separate node, each target entity is a separate node, and two nodes belonging to the same knowledge data are connected by connecting edges; based on the node graph, a knowledge database corresponding to the image dataset to be processed is constructed using entity information as the query condition.

[0111] like Figure 6 As shown, this embodiment of the invention also provides an electronic device, including a processor, which can execute any of the above-described knowledge database construction methods.

[0112] Specifically, it includes a processor 601 and a memory 602, as well as a computer program stored in the memory 602 and capable of running on the processor 601, which executes the knowledge database construction method, wherein:

[0113] The processor 601 executes the calculator program containing the knowledge database construction method stored in memory 602, performing the following steps:

[0114] Obtain a dataset of images to be processed, which includes multiple images to be processed;

[0115] Entity recognition is performed on the image to be processed to obtain at least one entity combination of the image to be processed, the entity combination including two target entities;

[0116] Based on the two target entities, at least one knowledge data corresponding to the entity combination is constructed;

[0117] Based on at least one of the knowledge data corresponding to the entity combination, a knowledge dataset of the image to be processed is constructed, and each image to be processed corresponds to one knowledge dataset.

[0118] Based on the knowledge dataset of all the images to be processed, a knowledge database corresponding to the image dataset to be processed is constructed.

[0119] Optionally, the entity recognition of the image to be processed performed by the processor 601 to obtain at least one combination of entities in the image to be processed includes:

[0120] Entity recognition is performed on the image to be processed to obtain multiple target entities in the image to be processed;

[0121] Among multiple target entities, every two target entities are combined to obtain at least one entity combination of the image to be processed.

[0122] Optionally, the entity recognition performed by the processor 601 on the image to be processed to obtain multiple target entities of the image to be processed includes:

[0123] Visual entities in the image to be processed are obtained by performing visual recognition on the image using a multimodal large model.

[0124] If there is no text region in the image to be processed, then multiple target entities in the image to be processed are determined based on the visual entities.

[0125] If there is a text region in the image to be processed, then extract the text image containing the text from the image to be processed.

[0126] Perform text recognition on the text image to obtain the text entities in the image to be processed;

[0127] Based on the visual entities and the text entities, multiple target entities in the image to be processed are identified.

[0128] Optionally, the process executed by processor 601 to construct at least one knowledge data corresponding to the entity combination based on the two target entities includes:

[0129] Relationship identification is performed on two target entities in the entity combination to obtain at least one entity relationship between the two target entities;

[0130] Based on the entity combination and at least one entity relationship, at least one knowledge data corresponding to the entity combination is constructed, and each knowledge data corresponds to one entity relationship.

[0131] Optionally, the process of processor 601 performing relation identification on the target entity pair to obtain at least one entity relation corresponding to the target entity includes:

[0132] In the image to be processed, relationship reasoning is performed on the two target entities in the target entity pair to obtain at least one association relationship between the two target entities;

[0133] Based on at least one association between two target entities, at least one entity relationship corresponding to each target entity is determined, and each association corresponds to one entity relationship.

[0134] Optionally, the process executed by processor 601 to construct at least one knowledge data corresponding to the entity combination based on the two target entities includes:

[0135] Associating two target entities in the entity combination with at least one entity relation corresponding to the entity combination yields at least one knowledge triplet corresponding to the entity combination, with each entity relation corresponding to one knowledge triplet.

[0136] The knowledge triple is determined as the knowledge data corresponding to the entity combination.

[0137] Optionally, the processor 601 executes the process of constructing a knowledge database corresponding to the dataset of images to be processed based on the knowledge dataset of all the images to be processed, including:

[0138] Based on the knowledge dataset of all the images to be processed, a node graph is constructed. In the node graph, each image to be processed is a separate node, each target entity is a separate node, and two nodes belonging to the same knowledge dataset are connected by connecting edges.

[0139] Based on the node graph, and using entity information as the query condition, a knowledge database corresponding to the image dataset to be processed is constructed.

[0140] This invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the knowledge database construction method provided in this invention and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0141] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0142] The above description discloses only preferred embodiments of the present invention and should not be construed as limiting the scope of the present invention. Therefore, equivalent variations made in accordance with the claims of the present invention are still within the scope of the present invention.

Claims

1. A method for constructing a knowledge database, characterized in that, The method includes the following steps: Obtain a dataset of images to be processed, which includes multiple images to be processed; Entity recognition is performed on the image to be processed to obtain at least one entity combination of the image to be processed, the entity combination including two target entities; Based on the two target entities, at least one knowledge data corresponding to the entity combination is constructed; Based on at least one of the knowledge data corresponding to the entity combination, a knowledge dataset of the image to be processed is constructed, and each image to be processed corresponds to one knowledge dataset. Based on the knowledge dataset of all the images to be processed, a knowledge database corresponding to the image dataset to be processed is constructed.

2. The knowledge database construction method as described in claim 1, characterized in that, The step of performing entity recognition on the image to be processed to obtain at least one combination of entities in the image to be processed includes: Entity recognition is performed on the image to be processed to obtain multiple target entities in the image to be processed; Among multiple target entities, every two target entities are combined to obtain at least one entity combination of the image to be processed.

3. The knowledge database construction method as described in claim 2, characterized in that, The step of performing entity recognition on the image to be processed to obtain multiple target entities in the image to be processed includes: Visual entities in the image to be processed are obtained by performing visual recognition on the image using a multimodal large model. If there is no text region in the image to be processed, then multiple target entities in the image to be processed are determined based on the visual entities. If there is a text region in the image to be processed, then extract the text image containing the text from the image to be processed. Perform text recognition on the text image to obtain the text entities in the image to be processed; Based on the visual entities and the text entities, multiple target entities in the image to be processed are identified.

4. The knowledge database construction method as described in claim 2, characterized in that, The step of constructing at least one knowledge data corresponding to the entity combination based on the two target entities includes: Relationship identification is performed on two target entities in the entity combination to obtain at least one entity relationship between the two target entities; Based on the entity combination and at least one entity relationship, at least one knowledge data corresponding to the entity combination is constructed, and each knowledge data corresponds to one entity relationship.

5. The knowledge database construction method as described in claim 4, characterized in that, The step of performing relationship identification on the target entity pair to obtain at least one entity relationship corresponding to the target entity includes: In the image to be processed, relationship reasoning is performed on the two target entities in the target entity pair to obtain at least one association relationship between the two target entities; Based on at least one association between two target entities, at least one entity relationship corresponding to each target entity is determined, and each association corresponds to one entity relationship.

6. The knowledge database construction method as described in claim 4, characterized in that, The step of constructing at least one knowledge data corresponding to the entity combination based on the two target entities includes: Associating two target entities in the entity combination with at least one entity relation corresponding to the entity combination yields at least one knowledge triplet corresponding to the entity combination, with each entity relation corresponding to one knowledge triplet. The knowledge triple is determined as the knowledge data corresponding to the entity combination.

7. The knowledge database construction method according to any one of claims 1 to 6, characterized in that, The knowledge database corresponding to the dataset of images to be processed is constructed based on the knowledge dataset of all the images to be processed, including: Based on the knowledge dataset of all the images to be processed, a node graph is constructed. In the node graph, each image to be processed is a separate node, each target entity is a separate node, and two nodes belonging to the same knowledge dataset are connected by connecting edges. Based on the node graph, and using entity information as the query condition, a knowledge database corresponding to the image dataset to be processed is constructed.

8. A knowledge database construction apparatus, characterized in that, The knowledge database construction device includes: The acquisition module is used to acquire a dataset of images to be processed, which includes multiple images to be processed. The processing module is used to perform entity recognition on the image to be processed to obtain at least one entity combination of the image to be processed, wherein the entity combination includes two target entities; The first construction module is used to construct at least one knowledge data corresponding to the entity combination based on the two target entities; The second construction module is used to construct a knowledge dataset for the image to be processed based on at least one knowledge data corresponding to the entity combination, wherein each image to be processed corresponds to one knowledge dataset. The third construction module is used to construct a knowledge database corresponding to the dataset of images to be processed based on the knowledge dataset of all the images to be processed.

9. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the knowledge database construction method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the knowledge database construction method as described in any one of claims 1 to 7.

Images