Artificial intelligence-based information processing method, apparatus, device, and storage medium

Abstract

The present disclosure provides an artificial intelligence-based information processing method, device and equipment and a storage medium, relates to the technical field of computers, in particular to the technical fields of large models, data processing, artificial intelligence and the like. The specific implementation scheme is as follows: a content understanding mode corresponding to an information type of to-be-understood information is used to extract a plurality of sub-information from the to-be-understood information; the plurality of sub-information is converted into knowledge expression information used for retrieval according to information modalities; the knowledge expression information of the plurality of sub-information having semantic correlation is stored in association to obtain a knowledge set; the knowledge set includes a scalar and a vector.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, particularly to the fields of large models, data processing, and artificial intelligence. Background Technology

[0002] Currently, AI (Artificial Intelligence) has demonstrated significant technological advantages in text processing. In long text scenarios, the context window has expanded to millions of tokens, enabling the parsing of multiple documents simultaneously and achieving end-to-end analysis.

[0003] However, the core shortcomings remain prominent. In scenarios with a large amount of content to be understood, or with a lot of redundant and invalid content, the accuracy of locating key information decreases significantly as the complexity of understanding increases. Factual errors in the generated content may lead to mistakes in professional scenarios, or even cause serious consequences. Summary of the Invention

[0004] This disclosure provides an information processing method, apparatus, device, and storage medium based on artificial intelligence.

[0005] According to one aspect of this disclosure, an information processing method based on artificial intelligence is provided, comprising: The content understanding method corresponding to the information type of the information to be understood is used to extract multiple sub-information from the information to be understood; Multiple pieces of information are converted into knowledge representation information for retrieval according to information modalities; Knowledge representation information that is semantically related to multiple sub-informations is stored together to obtain a knowledge set; the knowledge set includes scalars and vectors.

[0006] According to another aspect of this disclosure, an information processing apparatus based on artificial intelligence is provided, comprising: The extraction module is used to extract multiple sub-information from the information to be understood by adopting the content understanding method corresponding to the information type of the information to be understood; The conversion module is used to convert multiple sub-information into knowledge representation information for retrieval according to information modality; The association module is used to associate and store knowledge representation information of multiple semantically related sub-information to obtain a knowledge set; the knowledge set includes scalars and vectors.

[0007] According to another aspect of this disclosure, an electronic device is provided, comprising: At least one processor; and The memory is communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform any of the methods described in the present disclosure.

[0008] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform any of the methods according to embodiments of this disclosure.

[0009] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements any of the methods according to embodiments of this disclosure.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 This is a flowchart illustrating an artificial intelligence-based information processing method according to an embodiment of the present disclosure; Figure 2 This is a flowchart illustrating the process of extracting multiple sub-information from information to be understood according to an embodiment of the present disclosure; Figure 3 This is a flowchart illustrating the process of extracting multiple sub-information from information to be understood according to another embodiment of the present disclosure; Figure 4 This is a flowchart illustrating the process of converting multiple pieces of sub-information into knowledge representation information for retrieval according to an embodiment of the present disclosure; Figure 5 This is a flowchart illustrating the implementation of an intelligent response to a query request according to an embodiment of the present disclosure; Figure 6 This is a schematic diagram of the overall process of an information processing method based on artificial intelligence according to an embodiment of the present disclosure; Figure 7 This is a schematic diagram of the structure of an artificial intelligence-based information processing device according to an embodiment of the present disclosure; Figure 8 This is a block diagram of an electronic device used to implement the artificial intelligence-based information processing method of the embodiments of this disclosure. Detailed Implementation

[0012] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0013] The terms “first,” “second,” etc., used in this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion, such as including a series of steps or units. A method, system, product, or apparatus is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0014] It should be noted that, unless it is explicitly stated that there is a sequential order of execution between different operations, or that there is a sequential order of execution between different operations in terms of technical implementation, the execution order between multiple operations may not be significant, and multiple operations may be executed simultaneously.

[0015] Leveraging the powerful content understanding capabilities and self-learning characteristics of artificial intelligence, this disclosure provides an AI-based information processing method. This method is applicable to intelligent question-and-answer scenarios. For example, it can assist in content understanding of long documents, assist in content understanding of multimodal information, and provide question-and-answer support in intelligent store creation.

[0016] Especially in the Q&A scenario of intelligent store building, the questions often involve specific product or service needs, requiring a higher level of accuracy in the answers. For example, in traditional marketing campaigns, the expression and implementation of merchant knowledge heavily rely on specific content formats and agreed-upon types, typically using a dedicated landing page as the core carrier, and requiring professional website building capabilities for knowledge creation and publishing. Merchants also need to independently select suitable content types for their campaigns, leading to problems such as complex document formats, cumbersome manual data entry processes, and high costs associated with content adaptation and deployment. In this model, the landing page content serves only as a single carrier of information, its core value limited to formal display, making deep reuse and flexible application difficult.

[0017] With the development and maturation of large language model technology, the core attribute of content has upgraded from an information carrier to a cognitive asset that can be deeply understood and structured. The core carrier of marketing interaction has also gradually shifted from traditional landing pages to intelligent dialogue and intelligent agent interaction scenarios, resulting in a fundamental change in knowledge retrieval logic. Unlike the traditional model where merchants define knowledge types and display formats, intelligent agent scenarios require dynamic matching and accurate response to relevant knowledge based on real-time user needs, placing higher demands on multi-source, multi-format, and multi-modal merchant knowledge.

[0018] However, the relevant technologies struggle to perform in-depth analysis, unified processing, and efficient retrieval of multi-format and multi-modal merchant knowledge, resulting in high barriers to entry for merchant knowledge, high update and maintenance costs, and low distribution efficiency, making it unsuitable for the actual needs of intelligent marketing scenarios.

[0019] The information processing method based on artificial intelligence provided in this disclosure, for the landing page of a merchant's store, will not only display knowledge after completion, but will also deeply understand and reuse its content to complete intelligent question and answer.

[0020] Similarly, the method provided in this disclosure can also handle complex knowledge carriers in other scenarios, thereby improving the efficiency and accuracy of knowledge retrieval.

[0021] like Figure 1 The diagram shown illustrates the flow chart of the artificial intelligence-based information processing method provided in this disclosure, including the following: S101, using the content understanding method corresponding to the information type of the information to be understood, extract multiple sub-information from the information to be understood; where different information types correspond to different content understanding methods.

[0022] Information to be understood refers to the original information that requires content analysis and knowledge-based processing. In this embodiment, the information type of the information to be understood mainly refers to the type of information carrier, including but not limited to documents, network elements, and websites.

[0023] Each information type supports single-modal information, such as text information, image information, audio information, video information, etc., and also supports multimodal information.

[0024] To improve the accuracy of retrieval and response, this embodiment employs a pre-set content understanding method corresponding to different information types. This can be understood as follows: based on the information type corresponding to the information to be understood, a content understanding method adapted to that information type is used to parse and break down the information to be understood, extracting multiple independent sub-information pieces with clear semantic meanings from the information to be understood.

[0025] S102, convert multiple sub-information into knowledge representation information for retrieval according to information modality.

[0026] Information modalities refer to the different forms in which information is presented and expressed, reflecting the characteristics and attributes of the information itself. Examples include text modalities, image modalities, audio modalities, and video modalities.

[0027] During implementation, for sub-information of different modalities, a vectorization method matching its modality can be used for conversion, so that sub-information of different modalities can form a unified, standardized and searchable knowledge expression information form, providing a foundation for subsequent retrieval and matching.

[0028] S103, associate and store the knowledge representation information of multiple semantically related sub-information to obtain a knowledge set; the knowledge set includes scalars and vectors.

[0029] Semantic association refers to the inherent connection and relevance between sub-information based on the semantic level. It can reflect the mutual relationship between sub-information in terms of meaning, logic, etc., rather than just based on the surface form or grammatical structure.

[0030] During implementation, semantic association analysis is performed on the knowledge representation information of multiple transformed sub-information items. Knowledge representation information of multiple semantically related sub-information items is linked and stored together to construct a knowledge set. This knowledge set contains both scalar and vector data. Scalar data supports traditional keyword retrieval, including attributes, types, identifiers, classification tags, and structural information used to represent sub-information. Vector data is generated based on the GTE (General Text Embedding) high-performance vector model and is used to represent the deep semantic features of sub-information, such as text vectors and image vectors. By combining and storing scalar and vector data and associating semantically related sub-information, the knowledge set possesses the characteristics of being structured, semantic, and supporting RAG (Retrieval-Augmented Generation) hybrid retrieval, thereby improving the accuracy and efficiency of subsequent information queries and material acquisition.

[0031] In this embodiment, by extracting multiple sub-information using content understanding methods corresponding to the information type of the information to be understood, complex information to be understood can be broken down and transformed into sub-information that is easier to process and analyze, thereby deeply mining the key content within the information to be understood. By converting multiple sub-information into knowledge representation information for retrieval according to information modality, sub-information of different modalities can be processed in a unified manner, allowing information of various types and modalities to be retrieved within the same framework, thereby improving the efficiency and accuracy of information retrieval. By associating and storing the knowledge representation information of semantically related sub-information to form a knowledge set, forming a knowledge set that integrates scalars and vectors, it is possible to achieve structured parsing, unified representation, and semantic association of multi-source, multi-modal information, thereby improving the accuracy of information processing, retrieval efficiency, and knowledge utilization effect.

[0032] In this embodiment of the disclosure, when the information type of the information to be understood is an editable file compatible with multimodal information, the content understanding method corresponding to the information type of the information to be understood is used to extract multiple sub-information from the information to be understood. This can be achieved based on the following method: In the case of an editable file whose information type is compatible with multimodal information, extract at least one of the following sub-information and the semantic structure information of each sub-information in the information to be understood, on a paragraph-by-paragraph basis: text content, multimedia resources, tabular data, and figure caption text.

[0033] Editable files compatible with multimodal information refer to document formats that support the coexistence of multiple modalities such as text, images, audio, video, tables, and charts, and can be edited, modified, parsed, and saved by users or applications. Examples include Word (Microsoft Word) documents, WPS (WPS Office) documents, and online document editors.

[0034] A paragraph is a naturally formed section of text, usually separated by line breaks or similar delimiters. In practice, using paragraphs as units, at least one of the following sub-information can be extracted from the information to be understood: (1) Text content; In other words, plain text information contained in a paragraph includes at least one of the following: body description, headings, footnotes, and quotations. Text content is the basic form of information in editable files. During extraction, the original semantics of the text can be preserved, without changing the text's expression logic and core meaning. For example, document semantic information such as heading levels, list structures, and style tags can be preserved.

[0035] (2) Multimedia resources; This refers to various multimedia information embedded or associated within the paragraph, including but not limited to images, audio clips, and video clips. During extraction, deep parsing based on OOXML (Office Open XML, an open XML format) can accurately extract multimedia resources such as images and videos embedded in the document and establish multi-dimensional positional relationships to ensure that the semantic association between multimedia resources and corresponding text content is not lost.

[0036] (3) Tabular data; This refers to tabular information contained within or semantically related to the paragraph, including table headers, row data, column data, and cell content. During extraction, the table can be parsed in a structured manner, preserving its structure, such as row and column relationships, merged tables, and data format, and breaking down the table data into individually identifiable and searchable sub-information units.

[0037] (4) Caption text.

[0038] This refers to the descriptive text corresponding to the visual resources such as images and charts embedded in the paragraph. It may include at least one of the following: figure number, figure title, and annotations of elements within the figure. During extraction, the figure caption text can be associated with and recorded with the corresponding multimedia resources to ensure a clear semantic correspondence between the two.

[0039] Among them, the semantic structure information of each sub-information in the information to be understood refers to the structured information that transcends the visual presentation level and is used to describe at least one of the following in the information to be understood: logical attribution, type attribute, hierarchical relationship, positional relationship, semantic association and contextual dependency relationship. It is used to represent the semantic organization form between each sub-information and between the sub-information and the editable file.

[0040] In this embodiment, sub-information such as text content, multimedia resources, tabular data, and figure captions are extracted on a per-segment basis. This enables the structured and semantic breakdown and parsing of the content of editable files compatible with multimodal information. By extracting semantic structure information, the semantic context of each sub-information within the information to be understood can be clarified, providing a semantic foundation for subsequently converting sub-information into knowledge representation information and storing it in association. This improves the semantic coherence and accuracy of knowledge points in the knowledge set.

[0041] In this embodiment of the disclosure, when the information type of the information to be understood is an editable file compatible with multimodal information, and the sub-information extracted from the information to be understood is text content, the text content is text extracted based on semantic boundaries, and the semantic boundaries are defined at the sentence granularity and / or paragraph granularity.

[0042] Semantic boundaries are the limits used to divide text regions that have relatively independent semantics.

[0043] Among them, the semantic boundary defined at the sentence level can be judged by the semantic completeness of natural sentences. By identifying the punctuation marks and semantic pause markers of sentences, independent and semantically complete individual sentences can be divided as the extraction unit of text content.

[0044] Semantic boundaries defined at the paragraph level can be determined by the core semantics of a paragraph, treating all text content within the paragraph that revolves around the same core theme as a complete extraction unit. When a paragraph contains textual content involving semantic shifts or topic transitions, it can be further subdivided into multiple paragraph-level text extraction units based on semantic shift markers.

[0045] In implementation, semantic boundary delineation can be performed at the sentence level, the paragraph level, or both levels, depending on the text complexity and semantic relevance of the information to be understood. This allows for flexible adaptation to different text organization formats in editable files, ensuring the extracted text content is semantically complete and logically clear. For example, for longer paragraphs, extraction can be performed first by paragraph, and then further subdivided and extracted from sentences within each paragraph that are semantically independent. Specific implementation methods are not limited in this embodiment.

[0046] In this embodiment of the disclosure, the semantic structure information of the text content within the information to be understood includes at least one of the following: (1) Paragraph identifiers to which the text content belongs; Paragraph identifiers are information used to uniquely identify the paragraph to which text content belongs. They can be a number, a name, or other identifier that clearly distinguishes different paragraphs. They are used to achieve precise location and association of text content with its corresponding paragraphs.

[0047] (2) Chapter path; Chapter paths indicate the position of text content within the chapter structure of an editable document that supports multimodal information. In a document with chapter divisions, each chapter has a specific hierarchical relationship. Chapter paths clearly show the chapter level to which the text content belongs, thus reflecting the structural relationship of the text content within the overall document.

[0048] (3) Style tags; Style tags are used to describe the format and layout style of text content, including but not limited to font, font size, color, bold, italics, underline, paragraph formatting and other layout style information, which are used to reflect the emphasis, distinction level and format characteristics of text content in the document.

[0049] (4) The set of sub-elements associated with paragraph text.

[0050] The set of sub-elements associated with a paragraph text refers to the set of other elements associated with that paragraph text. This includes, but is not limited to, sub-information such as images, tables, captions, and multimedia resources associated with the paragraph, used to represent semantic relationships such as descriptions, correspondences, and references between the text content and other modal sub-information.

[0051] In this embodiment, text content is extracted based on semantic boundaries defined at the sentence and / or paragraph granularity. This allows for better semantic segmentation of the text, making the extracted text content more consistent with semantic logic and facilitating an accurate understanding of the text's meaning and information essence. Furthermore, the semantic structure information of the text content, including paragraph identifiers, chapter paths, style tags, and sets of associated sub-elements, fully preserves the text content's hierarchy, position, format, and relationships with other sub-elements within the document. This provides a highly complete structural query basis for subsequent knowledge representation, vector generation, and retrieval matching, thereby improving the accuracy and traceability of information processing.

[0052] In this embodiment of the disclosure, when the information type of the information to be understood is an editable file compatible with multimodal information, and the sub-information extracted from the information to be understood is a multimedia resource, the semantic structure information of the multimedia resource includes at least one of the following: (1) Resource identifiers for multimedia resources; Resource identifiers are unique identifiers assigned to each extracted multimedia resource to distinguish different multimedia resources in the information to be understood, such as different images, audio, and video, and to avoid confusion between different multimedia resources.

[0053] (2) Related paragraph markers; The paragraph identifier associated with a multimedia resource is a unique identifier corresponding to a paragraph that has a direct semantic relationship with the multimedia resource. It is used to clarify the paragraph affiliation of the multimedia resource and to indicate the semantic relationship between the multimedia resource and the corresponding paragraph text, such as explanation, supplement, or corroboration.

[0054] (3) Character offset; Character offset refers to the specific positional information of multimedia resources within the information to be understood, using characters as the unit. It includes the start character offset and the end character offset, and is used to accurately locate the insertion position of multimedia resources in a document. It can accurately locate the position of the locator in a paragraph, avoid the loss of the positional information of multimedia resources due to a rough understanding of the content, retain a more accurate position, and can better understand the information conveyed by multimedia resources by combining position and context. At the same time, it provides accurate positional references for subsequent document backtracking, content verification, and information rearrangement.

[0055] (4) Anchoring method; Anchoring methods refer to the embedding and association methods of multimedia resources in editable files, including but not limited to three core types: embedded, linked, and floating. Embedded means the multimedia resource is directly embedded within the document, saved synchronously with the document body, and can be viewed directly within the document. Linked means the multimedia resource is not directly embedded in the document; its external storage path or access link is saved within the document, requiring the external resource to be accessed through the link. Floating means the multimedia resource can freely adjust its display position within the document, not fixed with the text layout.

[0056] (5) The path to the chapter; Chapter paths indicate the hierarchical relationship of multimedia resources within the information to be understood, that is, which chapter or section of the document the multimedia resource belongs to. This clearly presents the hierarchical position of the multimedia resource in the overall document structure and reflects its relationship with the overall semantic framework of the document.

[0057] (6) Associated figure captions.

[0058] In other words, when a multimedia resource has a corresponding caption text, the caption identifier is a unique identifier used to record the caption text corresponding to that multimedia resource. This enables a precise association between the multimedia resource and the caption text, clarifies the explanatory relationship between the two, and ensures that the multimedia resource and its explanatory text can be synchronously associated when storing knowledge in a subsequent context. This avoids semantic gaps and guarantees the semantic integrity of the knowledge set.

[0059] In this embodiment of the disclosure, by extracting semantic structural information such as resource identifiers, paragraph identifiers, character offsets, anchoring methods, chapter paths, and associated figure identifiers of multimedia resources, the location, hierarchical relationship, and associated context of multimedia resources in the document can be accurately located. The semantic binding and structural association between multimedia resources and text paragraphs, figure annotations, and chapters are fully preserved, thereby establishing multi-dimensional associations for multimedia resources. This provides reliable structural support for subsequent unified knowledge expression information, semantic retrieval, and multimodal information alignment, and improves the accuracy, completeness, and association matching efficiency of multimodal information processing.

[0060] In this embodiment of the disclosure, when the information type of the information to be understood is an editable file compatible with multimodal information, and the sub-information extracted from the information to be understood is tabular data, the semantic structure information of the tabular data includes at least one of the following: (1) The paragraph identifier to which it belongs; The paragraph identifier refers to the unique identifier that identifies the paragraph in the document to which the table data belongs. It is used to indicate the semantic relationship between the table data and the text content, such as explanation or supplement.

[0061] (2) Table hierarchy path in multi-level nesting cases; In multi-level nesting, the table hierarchy path refers to the hierarchical position and nesting relationship of a table in a multi-level nested table structure, and is used to reflect the subordinate, nested, and contained relationships between tables.

[0062] (3) Row and column data; Row and column data refer to the specific content contained in each row and column of a table. They represent the basic data information of the table, including various types of data such as text, numbers, and dates in the cells.

[0063] (4) Table structure; The table structure describes the overall layout and organization of the table, including but not limited to at least one of the following: the number of rows, the number of columns, cell merging information, border styles, cell alignment, and structured descriptions of the header area.

[0064] (5) Nested sub-table identifiers.

[0065] That is, when there are nested sub-tables in the table, the nested sub-table identifier is used to uniquely identify each sub-table in order to distinguish sub-tables at different levels and in different nesting positions.

[0066] In this embodiment of the disclosure, by extracting semantic structural information such as the paragraph identifier to which the table data belongs, the table hierarchy path under multiple nesting, row and column data, table structure and nested sub-table identifiers, the position, hierarchy and internal organization of the table in the document can be completely preserved. This enables refined parsing and structured representation of complex nested tables, providing an accurate and complete data foundation for subsequent unified knowledge expression information, vector generation and structured retrieval, and improving the accuracy and usability of complex table information processing.

[0067] In this embodiment of the disclosure, when the information type of the information to be understood is an editable file compatible with multimodal information, and the sub-information extracted from the information to be understood is caption text, the semantic structure information of the caption text includes at least one of the following: (1) Associated multimedia resource identifier; The associated multimedia resource identifier refers to the unique identifier of the multimedia resource that has a unique correspondence with the caption text, and is used to indicate the binding relationship between the caption text and the corresponding multimedia resource.

[0068] (2) Associated paragraph identifiers.

[0069] The paragraph identifier refers to the unique identifier of the paragraph to which the caption text belongs in the information to be understood. It is used to indicate the location of the caption text, that is, the semantic relationship between the caption text and the paragraph text.

[0070] In this embodiment of the disclosure, by extracting semantic structure information such as multimedia resource identifiers and paragraph identifiers corresponding to the image caption text, a semantic binding relationship can be established between the image caption text and the associated multimedia resources and the text paragraphs to which it belongs. This improves the completeness and accuracy of the semantic correspondence between multimodal information, provides a reliable data foundation for subsequent unified knowledge expression information, semantic association storage and accurate retrieval, and enhances the accuracy of multimodal information understanding and matching.

[0071] In this embodiment of the disclosure, when the information type is a non-editable file, multiple sub-information is extracted from the information to be understood by adopting the content understanding method corresponding to the information type of the information to be understood. This can be achieved based on the following methods: In the case of a non-editable document, based on the visually divided regions in the information to be understood, at least one of the following sub-information and the layout structure information of each sub-information in the information to be understood are extracted from the information to be understood: basic text unit; semantic block unit; object unit.

[0072] Non-editable files are those whose internal content, such as text and images, cannot be added to, deleted from, modified, or split using conventional editing tools. Examples include PDF (Portable Document Format) files and image-based documents. Understandably, whether a file is editable can be identified based on its file extension. This helps improve the accuracy of information type identification. For instance, a PDF file that is not editable by default but is actually editable will still be identified as non-editable, allowing for the appropriate use of content understanding methods to interpret the content and organize it into knowledge-based information.

[0073] During implementation, due to the inherently uneditable nature of the file, direct manipulation of the text is not possible. Therefore, sub-information can be extracted based on visually defined regions of the information to be understood. These regions are different parts divided according to the visual presentation of the file, such as the layout of text and images. For example, a page may have a title area, a body text area, and an image area.

[0074] In this embodiment of the disclosure, based on visually segmented region blocks in the information to be understood, at least one of the following sub-information is extracted from the information to be understood: (1) The basic text unit is the smallest indivisible text granularity entity in document layout analysis; This smallest text-granularity entity is the atomic foundation for building the logical structure of a document. It represents a text fragment that cannot be further broken down into smaller but still has independent expressive capabilities. It usually corresponds to a character with its own language, a word or short phrase composed of multiple visually continuous characters (such as idioms or proverbs).

[0075] (2) A semantic block unit, consisting of at least one region block of character content used to express complete semantics; Since the text content in an uneditable file may be scattered across multiple visual regions, the text in a single region may not be able to express complete semantics. Therefore, by integrating the character content of one or more semantically related regions to form semantic block units, we can ensure that the extracted text content has complete semantic expression and avoid semantic breaks caused by visual region division.

[0076] A region block consists of multiple basic text units. A complete semantic block may include one or more region blocks.

[0077] (3) Object unit, used to represent non-character type elements in the information to be understood.

[0078] This refers to all elements that are visually presented but do not contain text information, including but not limited to images, charts, lines, stamps, QR codes, icons, tables, etc., in non-editable files. The extraction object unit can achieve comprehensive coverage of multimodal information in non-editable files, avoiding the omission of valid non-character information when only extracting text information.

[0079] In the case of non-editable files, the layout structure information of each sub-information within the information to be understood refers to a set of metadata describing the spatial location, arrangement, and geometric relationships of visual elements on the page. Examples include the page number of the sub-information, its coordinate range within the page, and its relative position to other areas.

[0080] In this embodiment of the disclosure, by visually dividing the uneditable document into regions and extracting basic text units, semantic block units, object units, and corresponding layout structure information, the document content of the uneditable document can be transformed into identifiable and processable structured sub-information. This preserves the visual layout and physical positional relationship of the document while achieving the separation and unified representation of the smallest text granularity, complete semantic units, and non-character objects. This provides accurate basic data for subsequent knowledge expression information conversion, vector generation, and associated storage, thereby improving the adaptability, accuracy, and completeness of processing uneditable document information.

[0081] In this embodiment of the disclosure, when the information type is a non-editable file, and the sub-information extracted from the information to be understood is a basic text unit, the layout structure information of the basic text unit includes at least one of the following: (1) Page number; Page numbers refer to the specific page number of a non-editable document containing a basic text unit, used to indicate the page affiliation of that basic text unit in a multi-page document.

[0082] (2) Block location information; The location information of a region block refers to the coordinates, range, size, and relative position of the visual region block to which the basic text unit belongs on the page, which is used to represent the physical distribution of the basic text unit on the page layout.

[0083] (3) Region type; Region type refers to the layout type of the visual area block to which the basic text unit belongs, such as title area, body text area, footer area, side area, header and footer area, etc. It is used to distinguish different functional areas of the layout and improve the accuracy of subsequent semantic understanding and knowledge classification.

[0084] (4) Area block identifier.

[0085] A region block identifier is a unique identifier assigned to each visual region block to distinguish different region blocks within the same page. This ensures that the correspondence between basic text units and their respective region blocks is clear and unique, facilitating subsequent structured storage and fast indexing.

[0086] In this embodiment of the disclosure, by extracting page numbers, region block location information, region type, region block identifiers, and other layout structure information of basic text units, the position and attributes of the smallest text unit in an uneditable document can be located and uniquely identified in the layout. The physical layout, region affiliation, and visual distribution characteristics of the text in the document are fully preserved, enabling fine-grained and structured parsing of the content of the uneditable document. This provides a reliable layout positioning basis for subsequent semantic block merging, object association, knowledge expression information conversion, and accurate retrieval, thereby improving the accuracy, completeness, and locatability of uneditable document information processing.

[0087] In this embodiment of the disclosure, when the information type is a non-editable file, and the sub-information extracted from the information to be understood is a semantic block unit, the layout structure information of the semantic block unit includes at least one of the following: (1) Construct at least one region block identifier with complete semantics; A semantic block is a unique identifier for a visual region that has a semantic association with the semantic block unit and can itself constitute a complete semantic fragment. This can be understood as follows: when a complete semantic unit includes multiple region blocks, the complete semantic unit and the identifiers of these multiple region blocks are associated and stored to provide structural information for retrieval.

[0088] (2) Includes basic text unit identifiers; The included basic text unit identifiers refer to the unique set of identifiers for all basic text units contained in this semantic block unit.

[0089] (3) Preceding region block identifier; A preceding region block identifier is a unique identifier for a visual region block that precedes the semantic block unit in the layout of a non-editable document and has a potential semantic relationship with it. This identifier is used to indicate the order of semantic block units in the layout and to clarify their positional association with preceding related region blocks.

[0090] (4) Subsequent area block identifiers.

[0091] The subsequent region block identifier refers to a unique identifier for a visual region block located after the region block to which the semantic block unit belongs, and which has a potential semantic relationship with the semantic block unit, within the layout of a non-editable document. Corresponding to the preceding region block identifier, this identifier is used to clarify the positional relationship between the semantic block unit and subsequent related region blocks.

[0092] In implementation, when a semantic block unit includes multiple region blocks, a key region block can be selected and its preceding region block identifier and / or following region block identifier can be recorded. Alternatively, the preceding region block identifier and / or following region block identifier can be recorded for each region block.

[0093] In this embodiment of the disclosure, by extracting the semantic complete region block identifier, the basic text unit identifier, and the preceding and subsequent region block identifiers associated with the semantic block unit, the composition structure, association relationship, and page position characteristics of the semantic block unit can be clearly defined. The original page logic and semantic association in the non-editable file are fully preserved, providing reliable structural support for converting the semantic block unit into searchable knowledge expression information and associating it with other sub-information for storage. This further improves the structured information processing flow of the non-editable file and enhances the practicality and retrieval accuracy of the knowledge set.

[0094] In this embodiment of the disclosure, when the information type is a non-editable file, and the sub-information extracted from the information to be understood is an object unit, the layout structure information of the object unit includes at least one of the following: (1) Page number; Page number refers to the page number on which an object unit is located, used to indicate the page to which the object unit belongs in a multi-page non-editable document.

[0095] (2) Area block identifier; A region block identifier is a unique identifier assigned to the visual region block where an object unit is located. It is used to indicate the correspondence between the object unit and its region block, facilitating the location, indexing, and management of the object unit.

[0096] (3) Object type; Object type refers to the specific type of the object unit, such as non-character elements like images, charts, graphics, seals, QR codes, barcodes, icons, and tables. These elements are used to distinguish different categories of object units and provide a type basis for subsequent targeted processing and knowledge representation information conversion.

[0097] (4) Context block identifier.

[0098] A contextual block identifier is a unique identifier of a block that is adjacent to the object unit in terms of its position on the page and is semantically related to it. It is used to establish a contextual association between the object unit and the surrounding text content and to preserve the semantic context information of the object unit in the document.

[0099] In this embodiment, extracting page number, region block identifier, object type, and context region block identifier of the object unit is beneficial for uniquely identifying and locating the position and category of non-character objects in the uneditable document. At the same time, it establishes the layout association relationship between the object and its surrounding text and region blocks, realizing the structured and associative management of various object elements in the document. This improves the clarity of the ownership and context of multimodal content in the layout, provides a stable and reliable structural foundation for subsequent unified knowledge expression information, semantic alignment and retrieval matching, and enhances the integrity and accuracy of information processing in uneditable documents.

[0100] In this embodiment of the disclosure, when the information type is a webpage file, a content understanding method corresponding to the information type of the information to be understood is adopted to extract multiple sub-information from the information to be understood, such as... Figure 2 As shown, this can be achieved based on the following steps: S201, when the information type is a web page file, determine the node density of each node in the document object model tree of the web page file; the node density is directly proportional to the number of characters contained in the node, inversely proportional to the depth of the node in the document object model tree, and inversely proportional to the number of links contained in the node.

[0101] Web page files refer to network document files written based on Hypertext Markup Language, Style Description Language, and Scripting Language, which can be loaded, parsed, and rendered in a browser, and are compatible with carrying multiple modal information such as text, images, audio, video, tables, hyperlinks, style information, and interactive controls.

[0102] During implementation, the node density of each node in the document object model tree corresponding to the webpage file is determined, which can be described by expression (1): (1) In expression (1), This represents the node density of the nth node; This represents the number of characters contained in the nth node; This represents the depth of the nth node in the document object model tree; This represents the number of links contained in the nth node; Represents the natural constant; This represents a logarithmic function used to compress the impact of the number of links on node density.

[0103] The more characters a node contains, the richer the information it carries, and the higher its node density. A node's depth in the model tree represents its position relative to the root node; greater depth indicates a more detailed position within the webpage structure, suggesting potentially lower importance and lower node density. A node containing more links is more likely used for navigation or linking to other pages rather than providing core content, also resulting in lower node density.

[0104] S202, filter out nodes whose density is greater than the density threshold to obtain the remaining nodes.

[0105] The density threshold is a pre-set value used to determine whether the density of nodes is too high. When the density of a node exceeds this threshold, it indicates that the node may contain too much link or navigation information, or that the information is of low importance and may be some distracting information.

[0106] During implementation, the node density of each node is compared with a preset density threshold. Nodes with a density greater than the density threshold are filtered out, and the remaining nodes with a density that meets the requirements are retained.

[0107] S203. Based on the large model, perform semantic analysis on the remaining nodes, filter out nodes whose correlation with the reference target is lower than the correlation threshold, and obtain optimized nodes; each optimized node is used as sub-information.

[0108] Based on the large model, semantic analysis is performed on the remaining nodes. The correlation between the remaining nodes and the preset reference target is calculated. Nodes with a correlation below the correlation threshold are filtered out to obtain optimized nodes that retain semantic relevance. This ensures that the final output of the large model is highly relevant to the reference target and maximizes the information value density.

[0109] During implementation, nodes with a correlation degree lower than the correlation degree threshold with the reference target are filtered out to obtain optimized nodes, which can be described by formula (2): (2) In formula (2), This indicates the degree of correlation between the remaining nodes and the reference target; This represents the weight of the k-th sentence within the node; This represents the k-th sentence extracted from the node; This represents the semantic vector of the k-th sentence within the node; A semantic vector representing the reference target; This indicates the degree of similarity between the semantic vector of the k-th sentence and the semantic vector of the reference target in the semantic space.

[0110] In this embodiment, the node density of each node in the Document Object Model (DOM) tree of the web page file is determined. By comprehensively considering the number of characters a node contains, its depth within the DOM tree, and the number of links it contains, the information content and importance of each node can be fully measured. By filtering out nodes with a density greater than a density threshold, nodes with excessive characters, shallow depth, or too many links, potentially containing a large amount of redundant information, can be removed, thus initially selecting a more valuable set of nodes. Semantic analysis of the remaining nodes using a large model further filters out nodes with a relevance lower than a relevance threshold to further locate information closely related to the reference target. The resulting optimized nodes are used as sub-information to effectively reduce noise and irrelevant information in the web page file, focusing on core content highly relevant to the reference target, improving the accuracy and efficiency of information extraction, and providing a more valuable data foundation for subsequent information processing and analysis.

[0111] In this embodiment of the disclosure, when the information type is a website, a content understanding method corresponding to the information type of the information to be understood is adopted to extract multiple sub-information from the information to be understood, such as... Figure 3 As shown, it may include the following: S301, when the information type is a website, extract the details page information of the target object from the website.

[0112] A website is a collection of online information consisting of multiple interconnected web pages that are accessed through a unified domain name or access portal.

[0113] The target object refers to the pre-defined subject that needs to be extracted and processed, such as goods, articles, users, and organizations.

[0114] A target object's detail page refers to a webpage specifically designed to display complete information about that target object. It is a key page on a website that carries the core information of the target object, such as the product detail page on an e-commerce website or the article detail page on a news website.

[0115] S302, extract information from the details page according to preset information categories to obtain sub-information corresponding to each information category.

[0116] Predefined information categories are information classification standards predefined based on the attributes of the target object, information processing needs, and application scenarios. These categories are used to structurally break down the information on the details page, ensuring that the extracted sub-information has clear category attributes and semantic references, facilitating subsequent knowledge representation, information conversion, and associated storage. For example, if the target object is a product on an e-commerce website, the predefined information categories could include product name, product price, product specifications, manufacturer, product description, and after-sales guarantee; if the target object is an article on a news website, the predefined information categories could include article title, author, publication time, main content, keywords, and source.

[0117] In this embodiment of the disclosure, for scenarios where the information type is a website, the detail page information of the target object is extracted from the website, and then standardized and extracted according to a preset information category to obtain sub-information of the corresponding category. This enables the classification, parsing, and standardized extraction of unstructured content in the website, retains information related to the target object, and removes irrelevant pages and redundant content. This transforms the information from messy web page content into sub-information with a clear structure and well-defined categories, providing a high-quality data foundation for subsequent knowledge representation information conversion, vector generation, and related retrieval. This significantly improves the targeting, accuracy, and usability of website-type information processing.

[0118] Based on the preceding explanation, when the information type is a website, step S301 extracts the details page information of the target object from the website, such as... Figure 3 As shown, it may include the following: S3011, perform a screenshot operation on the website's webpage to obtain a webpage screenshot.

[0119] S3012 performs visual effects analysis on webpage screenshots using a large model, extracting multiple content blocks based on the details page definition from the webpage screenshots.

[0120] Visual effects analysis refers to the ability to analyze the visual features of webpage screenshots, such as layout partitions, color differences, element shapes, and white space intervals, based on large-scale model image recognition and layout analysis capabilities, to identify relatively independent visual areas, i.e., content blocks, in the webpage.

[0121] In this embodiment of the disclosure, extracting multiple content blocks based on the details page definition from a webpage screenshot may include at least one of the following: (1) Main product display; The main product display usually occupies the top position of the page, including a clear product title and at least one clear main image.

[0122] (2) Specifications and parameters table; The specifications table is used to display optional features such as product size, color, and model.

[0123] (3) Describe the block in detail; The detailed description section is a product feature description with a mix of text and images.

[0124] (3) Price display area; The price display area includes currency symbols and numerical prices.

[0125] (4) Purchase operating components; The purchase process includes "Buy Now" and "Add to Cart" buttons.

[0126] (5) User evaluation module.

[0127] The user rating module includes star ratings or a list of reviews.

[0128] S3013, if the webpage contains the target feature based on multiple content blocks and the webpage does not meet the exclusion conditions, the webpage is determined to be detail page information.

[0129] During implementation, if a webpage is determined to contain the target features and does not meet the exclusion criteria, the webpage is identified as the details page information of the target object, and the complete information of the webpage is extracted as the details page information. If a webpage does not contain the target features or meets any of the exclusion criteria, the webpage is identified as not a details page and is not extracted to avoid interference from invalid information.

[0130] Among them, target features are the core visual and content features used to define the information on the details page, ensuring that the extracted webpage is a details page that can fully display the information of the target object, and may specifically include: (1) The main image containing the target object; The main image refers to the image used to visually showcase the core form and characteristics of the target object. It is one of the core features that distinguishes a product detail page from other pages such as list pages and navigation pages. For example, the main image of an e-commerce product detail page or the cover image of an article detail page. By identifying the main image, one can initially determine the relevance of the webpage to the target object.

[0131] (2) Multiple content blocks contain detailed description blocks.

[0132] The detailed description section refers to the text or image-text combination area used to elaborate on the attributes, characteristics, and content of the target object. It is the key area of ​​the details page that carries the core information of the target object, such as the product parameter description area of ​​a product details page or the main text area of ​​an article details page. The existence of this section can further confirm that the webpage is a details page and ensure that the webpage contains complete information about the target object.

[0133] Exclusion criteria are used to further filter non-detail pages, avoiding misclassifying pages that are similar in form to detail pages but have different functions as detail pages. Specifically, they include at least one of the following: (1) The webpage screenshot contains pagination navigation; Pagination navigation refers to navigation elements used to navigate to different pages of the same content. The presence of pagination navigation on a webpage indicates that the webpage may be a list page or contain content across multiple pages, which contradicts the characteristic of detail pages, which are typically single-page displays of information.

[0134] (2) Webpage screenshots are categorized as instruction manuals; Articles in the form of instruction manuals refer to general instructions, tutorials, rules, etc., which do not focus on detailed information about a single target object and do not conform to the function of a target object's detail page. Therefore, they do not belong to the detail page category.

[0135] (3) The webpage screenshot is a screenshot of the website's homepage.

[0136] The core function of a website homepage is navigation and displaying an overall overview of the website, rather than focusing on detailed information about a single target object. Therefore, it does not possess the core characteristics of a detail page and is thus excluded.

[0137] In this embodiment of the disclosure, screenshots of website webpages are taken, and a large model is used to perform visual feature and content block analysis on the webpage screenshots. Based on target features such as whether the main image of the target object or the detailed description block is included, as well as exclusion conditions such as whether there is pagination navigation, instruction manual articles, or homepage, the target object's detail page information is automatically identified and extracted. This is to effectively filter out non-target pages such as list pages, homepages, navigation pages, and instruction manual pages, improve the accuracy of detail page information extraction, and provide a reliable data source for subsequent classification and extraction of sub-information.

[0138] Furthermore, after determining the details page information of the target object, step S302 extracts the details page information according to preset information categories, obtaining the sub-information corresponding to each information category, such as... Figure 3 As shown, it may include the following: S3021, Extract multiple images of the target object from the details page information.

[0139] During implementation, image recognition algorithms can be used to scan the details page information, identify and extract all multiple images related to the target object. The extracted images of the target object are then subjected to format filtering and conversion, size adjustment, and extreme size filtering to ensure that the image meets the standards.

[0140] S3022, Extract the header image of the target object from multiple images.

[0141] The header image is a representative image in the details page, used to highlight the core features of the target object and is a key visual element for quickly understanding the target object.

[0142] In practice, a preliminary screening can be conducted first. By relying on a multimodal large model combined with Embedding vector retrieval technology, the extracted images can be understood and converted into text, and then corresponding vectors can be generated. At the same time, the similarity between each image vector and the main image can be calculated, and the images with the highest similarity ranking can be selected for subsequent fine ranking.

[0143] During implementation, multiple images are selected through a coarse screening process, which can be described by expression (3): (3) In expression (3), This represents the k-th image. Semantic similarity with the main image T; This represents the k-th image. The corresponding semantic vector; This represents the semantic vector corresponding to the main image T; This represents the L2 norm.

[0144] Next, in the fine sorting stage, the VLM (Vision-Language Model) model based on CoT (Chain of Thought) is adopted. Through the Pointwise Scoring mechanism, the images after the initial screening are further refined to determine the optimal header image.

[0145] During implementation, the optimal header image is determined through a fine-tuning process, which can be described by expression (4): (4) In expression (4), This represents the final header image; This represents the image set constructed from the Top M images obtained from the coarse screening; This represents the k-th image among the M images obtained after coarse screening; This represents the comprehensive score of the k-th image calculated based on the image-by-image scoring mechanism; this comprehensive score can be based on the k-th image obtained in expression (3). It is calculated by combining at least one of the following information: semantic similarity with the main image T, similarity of the image's composition size with the main image T, and format similarity. Indicates in In the set, find the one that makes The image with the largest value Use it as the header image.

[0146] Finally, the selected header image is cropped to maximize the preservation of the product's prominent areas in the image while adapting to the display ratio requirements based on existing technology, thus completing the final processing of the header image.

[0147] S3023: Extract key fields from the details page information based on the industry-customized template for the target audience.

[0148] Industry-specific templates are pre-designed field extraction templates based on the characteristics, information expression habits, and actual information processing needs of the target industry. These templates include the core key fields of the target industry to efficiently extract crucial fields and avoid extracting irrelevant or missing core fields. For example, if the target is e-commerce products, the template might include key fields such as product name, price, specifications, manufacturer, production date, and after-sales policy. If the target is news articles, the template might include key fields such as article title, author, publication date, source, keywords, and abstract.

[0149] During implementation, based on the field features in the industry-customized template, key fields are accurately extracted from the text content of the details page information through text matching, semantic extraction, and other methods. The extraction result of each key field is treated as an independent sub-information to ensure the structure and relevance of the sub-information.

[0150] S3024, based on the large model, align the images and text in the details page information to obtain integrated image and text information.

[0151] During implementation, the semantic understanding and matching capabilities of the large model can be used to perform semantic analysis on all images and text content in the details page, establish the association between images and corresponding explanatory text, and achieve semantic alignment of images and text.

[0152] During implementation, the images and text in the details page are aligned based on the large model to obtain integrated image and text information, which can be described by expression (5): (5) In expression (5), This indicates the final selected image (header image) that best matches the target text paragraph. The image on the details page is interpreted by the large model as descriptive text. The semantic vector generated later; This indicates a text paragraph on the details page. The generated semantic vector; Cosine similarity is used to measure the degree of semantic matching between the semantic vector corresponding to the image description and the semantic vector of the text. This represents the hard keyword match score between the image and the text; This represents the weighting coefficient of the keyword matching item, used to adjust the proportion of keyword matching in the total score.

[0153] S3025, multiple sub-information including header image, key fields and integrated text and image information.

[0154] Through the above four steps, the obtained header image, key fields, and integrated text and image information are summarized and integrated to form a complete set of multiple sub-information items with a unified format.

[0155] In this embodiment, by extracting the target object image from the details page information and filtering the header image, extracting key fields based on industry-customized templates, and then using a large model to achieve semantic alignment between the image and text, integrated image and text information is formed. Finally, sub-information such as header image, key fields, and integrated image and text information is obtained, so as to enable structured and semantic parsing of multimodal information in the details page, realize the semantic association and unified expression of images and text, provide high-quality multimodal data for subsequent knowledge conversion, associated storage, and efficient retrieval, and significantly improve the accuracy, adaptability, and practicality of information extraction.

[0156] In this embodiment of the disclosure, when the information type is a plain text document, multiple sub-information is extracted from the information to be understood using a content understanding method corresponding to the information type of the information to be understood. This can be achieved based on the following methods: When the information type is a plain text document, extract line characters from the plain text document to obtain multiple sub-information composed of multiple line characters, and record the line break format.

[0157] A plain text document is a document that contains only character content, such as text, numbers, and symbols, and does not contain non-character elements such as images, tables, or multimedia resources. Examples include TXT (Text File) documents and unformatted text files.

[0158] Line-by-line extraction refers to using line breaks in a plain text document as the dividing line to split the document content into several consecutive character segments. Each character segment corresponds to one line of content, that is, one line of characters, and each line of characters is a sub-information.

[0159] Line break formatting refers to the way line breaks are identified in plain text documents (such as carriage return, line feed, etc.) and the distribution pattern of line breaks.

[0160] During implementation, the plain text document can be parsed line by line, extracting line characters one by one. Each extracted line character is treated as an independent sub-information, ultimately resulting in multiple sub-information items composed of multiple line characters. Simultaneously, the line break formatting of the plain text document is recorded during the process of extracting line characters and generating multiple sub-information items.

[0161] In this embodiment of the disclosure, for plain text document-type information to be understood, the method of extracting line characters to generate sub-information and synchronously recording the content understanding method of line break format helps to accurately and efficiently extract the core content in the plain text document, while completely preserving the original format characteristics of the plain text document. This allows for better adaptation to the subsequent need to convert the sub-information into knowledge expression information for retrieval and to perform associated storage to form a knowledge set.

[0162] In this embodiment of the disclosure, when the information type is a table file, multiple sub-information is extracted from the information to be understood using the content understanding method corresponding to the information type of the information to be understood. This can be achieved based on the following methods: When the information type is a table file, extract the table structure and table content from the table file; multiple sub-information includes table structure and table content.

[0163] That is, extract the table structure and table content from the table file, and treat the table structure and table content as multiple sub-information items.

[0164] The table structure refers to the overall layout and organization of the table file, used to fully reproduce the table's framework.

[0165] During implementation, extraction can be achieved through parsing table boundary definitions and original format specifications. Specifically, this includes parsing the table's data area range (Dimension), named ranges, filter areas (AutoFilter), frozen pane positions, and table object definitions to accurately identify the table's valid data boundaries and business semantic partitions, avoiding interference from blank areas on the effective structure. At the same time, it extracts cell number formats (date, percentage, amount, etc.), conditional formatting rules, and data validation constraints. Based on format features such as bold, fill color, borders, and frozen positions, it intelligently identifies the header row and data area boundaries, uncovers hidden business rules and field value constraints, and finally forms a complete table structure containing boundary information, format specifications, header definitions, and business partitions, providing structural support for table content parsing and subsequent retrieval.

[0166] Table content refers to the actual data information contained in each cell of the table, which is used to reflect the actual business meaning and data meaning expressed by the table.

[0167] During implementation, the system can comprehensively analyze table boundary information and format specifications to accurately extract specific data from each cell within the table. It can also use numeric format codes to accurately restore data types such as numeric dates, percentages, and amounts, and remove redundant content such as blank cells and invalid data. Finally, it outputs a structured table containing complete metadata such as column names, data types, value ranges, and business partitions, ensuring the semantic integrity and accuracy of the table data.

[0168] In this embodiment of the disclosure, by extracting the table structure and table content separately, the structural features of the table can be fully preserved while maintaining the integrity of the table data. This allows the extracted sub-information to have clear structural and content attributes, providing structural and data support for the subsequent conversion of sub-information into knowledge representation information for retrieval and for associated storage.

[0169] In this embodiment of the disclosure, for any information type, multiple sub-information items are converted into knowledge representation information for retrieval according to information modality, such as... Figure 4 As shown, it includes the following: S401: For the text content in multiple sub-information, extract scalars from the text content, and perform content understanding on the text content based on the large model to obtain the vector of the text content.

[0170] In practice, a text quality assessment model can be used to filter out low-value content such as blank content, invalid and redundant information, and data with disordered format. Then, valuable fragments, question-and-answer pairs, and service recommendation content can be extracted from the text content that has passed the initial quality screening. Finally, the text content can be refined and semantically encoded based on the large text model to obtain the vector corresponding to the text content.

[0171] S402, for multimedia resources in multiple sub-information, generate a vector of multimedia resources.

[0172] During implementation, the cross-modal semantic alignment capability of the technology and multimodal big model is used to perform content recognition and knowledge extraction on multimedia resources. After processing from dimensions such as classification tags and content summaries, corresponding multimedia vectors are generated, thereby unifying the sub-information of different modalities into knowledge representation information in the form of scalar and vector combinations that can be used for retrieval.

[0173] In this embodiment, scalars are extracted from text content and text vectors are generated using a large model for sub-information of different information modalities. Multimedia resources are processed by the large model to obtain corresponding vectors. This can uniformly convert heterogeneous information such as text and multimedia into structured knowledge representation information of scalars and vectors, so as to enable the vectorization, standardization and unification of multimodal information representation, providing a unified data foundation for subsequent semantic association storage and efficient retrieval, and effectively improving the accuracy and efficiency of cross-modal information retrieval.

[0174] Furthermore, after the knowledge set is constructed, when a user's query request is received, such as... Figure 5 As shown, intelligent responses to query requests can be achieved based on the following steps: S501, in response to a query request, retrieves the corresponding response material from the knowledge set based on a hybrid retrieval mode.

[0175] Hybrid retrieval mode combines the advantages of scalar retrieval and vector retrieval to achieve multi-dimensional and precise recall. On one hand, it utilizes scalar information from the knowledge set for precise matching of key fields such as attributes and identifiers, ensuring the accuracy and relevance of search results. On the other hand, it uses vector information from the knowledge set for semantic similarity calculation, achieving content retrieval that is semantically relevant to the query request at a deeper level, balancing recall and relevance. Through this hybrid retrieval method, highly relevant and complete information is selected from the knowledge set as response materials.

[0176] S502: Input the response materials into the large model and generate the response content for the query request.

[0177] The retrieved response materials are input into the large model, which then understands, integrates, summarizes, and organizes the response materials into natural language. Based on the intent of the query request, it generates a standardized, semantically fluent, and accurate response, which is then output to the user.

[0178] In this embodiment of the disclosure, in response to a query request, a hybrid retrieval mode is used to retrieve matching response materials from the knowledge set. The response materials are then input into a large model to generate response content. This allows for the combination of the semantic understanding capabilities of vector retrieval and the precise matching advantages of scalar retrieval, quickly locating highly relevant materials from the knowledge set that integrates scalar and vector data. The large model then understands, integrates, and generates natural language from the materials, thereby achieving precise understanding of the query intent, efficient retrieval of relevant knowledge, and accurate and standardized response content. This significantly improves the response speed, matching accuracy, and response quality of information queries.

[0179] In this embodiment of the disclosure, retrieving the response material corresponding to the query request from the knowledge set can be implemented as follows: For editable files in the knowledge set, response materials are selected based on the semantic structure information of each sub-information.

[0180] Based on the above description, the semantic structure information corresponding to each sub-information in an editable file includes the semantic structure information corresponding to the text content, multimedia resources, tabular data, and figure caption text. This information can be used to represent key information such as paragraph affiliation, hierarchical relationship, associated objects, and positional features of each sub-information in the original editable file.

[0181] During the retrieval process, the user's query request can be parsed first to clarify the query intent and core needs. Then, based on the query intent, the semantic structure information of the source information of editable documents in the knowledge set can be matched. By filtering the semantic structure information, the sub-information related to the query request can be quickly located. Then, combined with other information, the response materials that meet the query requirements can be accurately filtered from the remaining information.

[0182] In this embodiment of the disclosure, for the knowledge set of editable file types, the response materials are filtered based on the semantic structure information of each sub-information. This can make full use of the structural and semantic relevance of the editable file sub-information, avoid interference from irrelevant content, and ensure that the retrieved materials are highly matched with the query intent in terms of structure and semantics. This improves the accuracy and relevance of the response materials and provides solid support for generating accurate and reliable response content in the future.

[0183] In this embodiment of the disclosure, retrieving the response material corresponding to the query request from the knowledge set can be implemented as follows: For non-editable files in the knowledge set, select response materials based on the layout structure information of each sub-information.

[0184] Based on the content described above, the layout structure information corresponding to each sub-information in a non-editable file includes the layout structure information corresponding to each basic text unit, semantic block unit, and object unit.

[0185] During the retrieval process, based on the information type and content requirements corresponding to the query request, and the layout structure information of each sub-information in the non-editable file, different sub-information can be structurally filtered to locate the target sub-information that has a high degree of matching with the query request, is structurally regular, and has complete information. Then, the remaining information can be further refined based on other conditions to obtain the response materials.

[0186] In this embodiment of the disclosure, for non-editable files, the response materials are filtered based on the layout structure information of each sub-information. This can make full use of the unique layout features of non-editable files, quickly filter irrelevant content, accurately locate content units that match the query requirements, effectively improve the retrieval accuracy and reliability of image-based and complex-formatted documents, and provide high-quality materials with clear structure and precise positioning for the subsequent generation of accurate and standardized response content.

[0187] In this embodiment of the disclosure, retrieving the response material corresponding to the query request from the knowledge set can be implemented as follows: For the target object in the knowledge set, select response materials based on the industry classification tags and integrated text and image content of the target object.

[0188] In other words, upon receiving a user's query request, the system identifies the target object relevant to the query from the knowledge set. Based on the industry category tags corresponding to the target object, it performs preliminary screening of the information within the knowledge set, eliminating content irrelevant to the current industry field and narrowing the search scope. Building on this, it further combines the text and image content of the target object and extracts response materials that semantically match the query request, are complete in content, and conform to the prescribed format, according to preset matching rules or similarity calculation methods, from the pre-screened information.

[0189] In this embodiment of the disclosure, for target objects in the knowledge set, response materials are filtered based on the classification tags of their respective industries and the integrated text and image content. This can combine the constraints of industry classification with deep matching of text and image semantics, which helps to quickly locate target object information that is highly relevant to the query needs, improve the professionalism, relevance and relevance of the search results, and make the filtered materials more in line with industry standards and semantically complete, providing reliable support for generating accurate and professional response content in the future.

[0190] In summary, the overall flowchart of the information processing method based on artificial intelligence provided in this disclosure embodiment is as follows: Figure 6 As shown, it includes three parts: the parsing stage, the understanding stage, and the retrieval stage. (1) In the parsing stage, the content comprehension method corresponding to the information type of the information to be understood is adopted to extract multiple sub-information from the information to be understood; The information processing method based on artificial intelligence provided in this embodiment has comprehensive parsing capabilities across document types. It is compatible with plain text fragments, txt format files, Word, PDF, Excel and other text documents, as well as diverse content formats such as images, videos, and URLs (single page / entire website), realizing a unified parsing entry point for multi-format documents.

[0191] To address core issues such as high memory consumption, high processing latency, and risk of process interruption in large file processing, the information processing method based on artificial intelligence provided in this disclosure constructs a large file and streaming input optimization processing mechanism based on a distributed object storage service. This mechanism has three core capabilities: block reading, incremental parsing, and breakpoint resumption, to optimize resource consumption, efficiency, and stability in large file processing. The block reading strategy relies on the distributed object storage service to adopt pagination or block reading mode for large documents, avoiding memory overload caused by full loading and adapting to efficient processing of documents of different sizes. The incremental parsing capability supports the parser to output intermediate processing results page by page / block by block, allowing the upstream system to simultaneously perform index building and data storage operations, significantly reducing overall processing latency through parallel execution of "parsing-storage". The breakpoint resumption mechanism automatically records the checkpoint progress after each page or data block is processed, enabling recovery from the most recent progress after abnormal interruption, ensuring the continuity of the parsing process and data integrity.

[0192] To further improve the adaptability and resource controllability of chunked reading, the information processing method based on artificial intelligence provided in this embodiment of the disclosure designs a refined execution strategy on the basis of the core mechanism: First, it implements Range request chunking based on the HTTP Range protocol natively supported by the distributed object storage service, and pulls file content in segments with configurable block size (default 5MB, supports dynamic adjustment from 1MB to 20MB), strictly controlling the memory usage of a single operation at the block size level; Second, it formulates an intelligent chunking strategy, which adaptively adjusts the chunking granularity according to the total file size and network environment: increasing the block size for small files under 50MB to reduce the number of requests, and reducing the block size for large files over 500MB or in weak network environments to improve the transmission success rate; Third, it increases traffic control capabilities, supports setting a single-link bandwidth limit of 100KB / s to 100MB / s to avoid large file downloads crowding out business bandwidth; At the same time, before parsing, it uses a lightweight HEAD (HEAD method) request to pre-obtain metadata such as file size and ETag (Entity Tag) for calculating the total number of blocks, generating a processing plan, and subsequent data integrity verification.

[0193] To address the issues of data integrity and process reliability in block-based processing, the AI-based information processing method provided in this disclosure adds multi-dimensional verification and assurance mechanisms: During the data reading phase, each data block is immediately calculated and recorded with its MD5 value (Message-Digest Algorithm 5) after reading. During recovery processing, processed blocks can be selectively verified to ensure error-free single-block data. In the storage phase, idempotent processing is achieved through the unique identifier of block_id + sequence_index, automatically ignoring duplicate write operations and avoiding data redundancy. In the task completion phase, a full verification process is triggered, comparing the amount of stored content with the processing progress recorded by the checkpoint to ensure overall data integrity and completeness, ultimately achieving final consistency in large file processing.

[0194] In the case of an editable file that is compatible with multimodal information, at least one of the following sub-information and the semantic structure information of each sub-information in the information to be understood are extracted from the information to be understood, on a paragraph-by-paragraph basis: text content, multimedia resources, tabular data, and figure caption text.

[0195] In the case of non-editable documents, based on the visually divided regions of the information to be understood, at least one of the following sub-information and the layout structure information of each sub-information in the information to be understood are extracted: basic text unit, which is the smallest indivisible text granularity entity in document layout analysis; semantic block unit, which is character content composed of at least one region block used to express complete semantics; object unit, which is used to represent non-character type elements in the information to be understood.

[0196] When the information type is a web page file, determine the node density of each node in the document object model tree of the web page file; the node density is directly proportional to the number of characters contained in the node, inversely proportional to the depth of the node in the document object model tree, and inversely proportional to the number of links contained in the node; filter out nodes whose node density is greater than the density threshold to obtain the remaining nodes; perform semantic analysis on the remaining nodes based on the large model, filter out nodes whose relevance to the reference target is lower than the relevance threshold to obtain optimized nodes; each optimized node is used as a sub-information.

[0197] When the information type is a website, extract the details page information of the target object from the website; extract the details page information according to the preset information categories to obtain the sub-information corresponding to each information category.

[0198] When the information type is a plain text document, extract line characters from the plain text document to obtain multiple sub-information composed of multiple line characters, and record the line break format.

[0199] When the information type is a table file, extract the table structure and table content from the table file; multiple sub-information includes table structure and table content.

[0200] (2) In the understanding stage, multiple sub-information items are converted into knowledge representation information for retrieval according to information modalities. For the text content in the multiple sub-information items, scalars are extracted from the text content, and content understanding is performed on the text content based on the large model to obtain the text content vector. For the multimedia resources in the multiple sub-information items, a multimedia resource vector is generated.

[0201] (3) In the retrieval stage, knowledge expression information of multiple semantically related sub-information is associated and stored to obtain a knowledge set; based on the hybrid retrieval mode, the corresponding response material for the query request is retrieved in the knowledge set, and the response material is input into the large model to generate the response content of the query request.

[0202] In summary, the information processing method based on a large model provided in this disclosure has the following advantages: 1. Expand the boundaries of product knowledge processing: Support the full-process processing of multiple document formats such as text, images, videos, and URLs (single page / entire website), breaking the limitations of traditional products that only support a single or a few formats, enabling products to have more comprehensive knowledge access capabilities and adapt to the diverse knowledge management needs of merchants.

[0203] 2. Improve knowledge processing efficiency and quality: By using mechanisms such as adaptive segmentation, incremental parsing, and breakpoint resume, the problem of high memory consumption and high latency in large file processing is solved; relying on the GTE model and multimodal model, accurate knowledge extraction, deep understanding and efficient vectorization are achieved, ensuring the efficiency and accuracy of knowledge processing.

[0204] 3. Enhance core retrieval capabilities: Construct a scalar + vector hybrid retrieval mechanism, combining metadata precise matching and semantic similarity retrieval to significantly improve the accuracy and efficiency of knowledge retrieval in multiple scenarios, and meet the real-time knowledge retrieval needs in intelligent dialogue scenarios.

[0205] 4. Reduce user barriers and costs: Simplify the knowledge onboarding process for merchants, support custom segmentation rules and knowledge extraction, and enable the uploading and management of diverse knowledge without professional technical skills; at the same time, reduce product operation costs through technical optimization and enhance commercial competitiveness.

[0206] 5. Ensure security, compliance, and business adaptability: To address the security needs of merchants' private domain knowledge, avoid relying on mature third-party models and independently implement end-to-end processing; adapt to commercial risk control rules and answer routine requirements to ensure that the product meets business compliance standards.

[0207] Based on the same technical concept, this disclosure also provides an information processing device 700 based on artificial intelligence, such as... Figure 7 As shown, it includes: The extraction module 701 is used to extract multiple sub-information from the information to be understood by adopting the content understanding method corresponding to the information type of the information to be understood. The conversion module 702 is used to convert multiple sub-information into knowledge representation information for retrieval according to information modality; The association module 703 is used to associate and store knowledge representation information of multiple semantically related sub-information to obtain a knowledge set; the knowledge set includes scalars and vectors.

[0208] In some embodiments, the extraction module includes: The first extraction unit, when the information type is an editable document compatible with multimodal information, extracts at least one of the following sub-information and the semantic structure information of each sub-information in the information to be understood, on a paragraph-by-paragraph basis: Text content, multimedia resources, tabular data, and figure captions.

[0209] In some embodiments, the text content is text extracted based on semantic boundaries, which are defined at the sentence level and / or paragraph level. The semantic structure information of the text content includes at least one of the following: The collection of paragraph identifiers, chapter paths, style tags, and child elements associated with the paragraph text to which the text content belongs.

[0210] In some embodiments, the semantic structure information of the multimedia resource includes at least one of the following: The resource identifier, associated paragraph identifier, character offset, anchoring method, chapter path, and associated figure caption identifier of the multimedia resource.

[0211] In some embodiments, the semantic structure information of the tabular data includes at least one of the following: The paragraph identifier, the table hierarchy path in the case of multiple nesting, row and column data, table structure, and nested sub-table identifiers.

[0212] In some embodiments, the semantic structure information of the caption text includes at least one of the following: Associated multimedia resource identifiers and associated paragraph identifiers.

[0213] In some embodiments, the extraction module includes: The second extraction unit, when the information type is a non-editable file, extracts at least one of the following sub-information and the layout structure information of each sub-information from the information to be understood based on visually divided regions in the information to be understood: The basic text unit is the smallest indivisible text granularity entity in document layout analysis; A semantic block unit is a character content composed of at least one region block used to express complete semantics; An object unit is used to represent non-character elements in the information to be understood.

[0214] In some embodiments, the layout structure information of the basic text unit includes at least one of the following: Page number, region block location information, region type, region block identifier.

[0215] In some embodiments, the layout structure information of the semantic block unit includes at least one of the following: Construct at least one region block identifier, the basic text unit identifier contained therein, the preceding region block identifier, and the subsequent region block identifier to form a complete semantics.

[0216] In some embodiments, the layout structure information of the object unit includes at least one of the following: Page number, region block identifier, object type, context region block identifier.

[0217] In some embodiments, the extraction module includes: The third extraction unit is used to determine the node density of each node in the document object model tree of the web page file when the information type is a web page file. The node density is directly proportional to the number of characters contained in the node, inversely proportional to the depth of the node in the document object model tree, and inversely proportional to the number of links contained in the node. The filtering unit is used to filter out nodes whose node density is greater than the density threshold, and obtain the remaining nodes; The first sub-information extraction unit is used to perform semantic analysis on the remaining nodes based on the large model, filter out nodes whose correlation with the reference target is lower than the correlation threshold, and obtain optimized nodes; each optimized node is used as sub-information.

[0218] In some embodiments, the extraction module includes: The fourth extraction unit is used to extract the details page information of the target object from the website when the information type is a website; The second sub-information extraction unit is used to extract information from the details page according to preset information categories, and obtain sub-information corresponding to each information category.

[0219] In some embodiments, the fourth extraction unit is specifically used for: Take a screenshot of a webpage from a website. Visual effects analysis of webpage screenshots is performed using a large model to extract multiple content blocks based on the details page definition from the webpage screenshots; If a webpage is determined to contain the target features based on multiple content blocks, and the webpage does not meet the exclusion criteria, then the webpage is identified as detail page information. Target features include: a main image containing the target object, and multiple content blocks containing detailed description blocks; Exclusion criteria include at least one of the following: The webpage screenshot shows pagination navigation; Webpage screenshots are categorized as articles in the form of instruction manuals; The screenshot is of the website's homepage.

[0220] In some embodiments, the second sub-information extraction unit is specifically used for: Extract multiple images of the target object from the details page information; Extract the header image of the target object from multiple images; Based on the industry-customized template for the target audience, extract key fields from the details page information; Based on the large model, the images and text in the details page are aligned to obtain integrated image and text information; Multiple sub-information items include header image, key fields, and integrated text and image information.

[0221] In some embodiments, the extraction module includes: The fifth extraction unit is used when the information type is a plain text document. It extracts line characters from the plain text document to obtain multiple sub-information composed of multiple line characters, and records the line break format.

[0222] In some embodiments, the extraction module includes: The sixth extraction unit is used to extract the table structure and table content from the table file when the information type is a table file. Multiple sub-information items include table structure and table content.

[0223] In some embodiments, the conversion module includes: The first transformation unit is used to extract scalars from the text content in multiple sub-information, and perform content understanding on the text content based on the large model to obtain the vector of the text content. The second conversion unit is used to generate a vector of multimedia resources from multiple sub-information.

[0224] In some embodiments, a retrieval module is further included, for: In response to a query request, the system retrieves the corresponding response material from the knowledge set based on a hybrid retrieval mode. Input the response materials into the large model to generate the response content for the query request.

[0225] In some embodiments, the retrieval module is specifically used for: For editable files in the knowledge set, response materials are selected based on the semantic structure information of each sub-information.

[0226] In some embodiments, the retrieval module is specifically used for: For non-editable files in the knowledge set, select response materials based on the layout structure information of each sub-information.

[0227] In some embodiments, the retrieval module is specifically used for: For the target object in the knowledge set, select response materials based on the industry classification tags and integrated text and image content of the target object.

[0228] The specific functions and examples of each module and submodule of the apparatus in this disclosure can be found in the relevant descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.

[0229] The acquisition, storage, and application of any type of information, such as user personal information, involved in the technical solutions disclosed herein comply with relevant laws and regulations and do not violate public order and good morals.

[0230] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0231] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0232] like Figure 8As shown, device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 802 or a computer program loaded from storage unit 808 into random access memory (RAM) 803. RAM 803 may also store various programs and data required for the operation of device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.

[0233] Multiple components in device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of monitors, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0234] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as AI-based information processing methods. For example, in some embodiments, the AI-based information processing method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the AI-based information processing method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform AI-based information processing methods by any other suitable means (e.g., by means of firmware).

[0235] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0236] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0237] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0238] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0239] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0240] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0241] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0242] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An information processing method based on artificial intelligence, comprising: Using a content understanding method corresponding to the information type of the information to be understood, multiple sub-information is extracted from the information to be understood; The multiple pieces of information are converted into knowledge representation information for retrieval according to information modalities; Knowledge representation information of multiple semantically related sub-information is associated and stored to obtain a knowledge set; the knowledge set includes scalars and vectors.

2. The method according to claim 1, wherein, The method of content understanding corresponding to the information type of the information to be understood extracts multiple sub-information from the information to be understood, including: In the case where the information type is an editable file compatible with multimodal information, at least one of the following sub-information and the semantic structure information of each sub-information in the information to be understood are extracted from the information to be understood, taking natural paragraphs as units: Text content, multimedia resources, tabular data, and figure captions.

3. The method according to claim 2, wherein, The text content is text extracted based on semantic boundaries, which are defined at the sentence level and / or paragraph level. The semantic structure information of the text content includes at least one of the following: The text content belongs to the paragraph identifier, chapter path, style tags, and the set of child elements associated with the paragraph text.

4. The method according to claim 2, wherein, The semantic structure information of the multimedia resources includes at least one of the following: The multimedia resource includes its resource identifier, associated paragraph identifier, character offset, anchoring method, chapter path, and associated caption identifier.

5. The method according to claim 2, wherein, The semantic structure information of the table data includes at least one of the following: The paragraph identifier, the table hierarchy path in the case of multiple nesting, row and column data, table structure, and nested sub-table identifiers.

6. The method according to claim 2, wherein, The semantic structure information of the caption text includes at least one of the following: Associated multimedia resource identifiers and associated paragraph identifiers.

7. The method according to claim 1, wherein, The method of content understanding corresponding to the information type of the information to be understood extracts multiple sub-information from the information to be understood, including: In the case where the information type is a non-editable file, based on the visually divided regions in the information to be understood, at least one of the following sub-information and the layout structure information of each sub-information in the information to be understood are extracted from the information to be understood: The basic text unit is the smallest indivisible text granularity entity in document layout analysis; A semantic block unit is a character content composed of at least one region block used to express complete semantics; An object unit is used to represent a non-character element in the information to be understood.

8. The method according to claim 7, wherein, The layout structure information of the basic text unit includes at least one of the following: Page number, region block location information, region type, region block identifier.

9. The method according to claim 7, wherein, The layout structure information of the semantic block unit includes at least one of the following: Construct at least one region block identifier, the basic text unit identifier contained therein, the preceding region block identifier, and the subsequent region block identifier to form a complete semantics.

10. The method according to claim 7, wherein, The layout structure information of the object unit includes at least one of the following: Page number, region block identifier, object type, context region block identifier.

11. The method according to claim 1, wherein, The method of content understanding corresponding to the information type of the information to be understood extracts multiple sub-information from the information to be understood, including: When the information type is a web page file, the node density of each node in the document object model tree of the web page file is determined; the node density is directly proportional to the number of characters contained in the node, inversely proportional to the depth of the node in the document object model tree, and inversely proportional to the number of links contained in the node. Filter out nodes whose density is greater than the density threshold to obtain the remaining nodes; Based on the large model, semantic analysis is performed on the remaining nodes to filter out nodes whose correlation with the reference target is lower than the correlation threshold, thus obtaining optimized nodes; each optimized node is used as sub-information.

12. The method according to claim 1, wherein, The method of content understanding corresponding to the information type of the information to be understood extracts multiple sub-information from the information to be understood, including: When the information type is a website, extract the details page information of the target object from the website; The details page information is extracted according to preset information categories to obtain sub-information corresponding to each information category.

13. The method according to claim 12, wherein, The step of extracting the details page information of the target object from the website includes: Take a screenshot of the webpage on the website to obtain a webpage screenshot; Visual effects analysis of the webpage screenshots is performed using a large model to extract multiple content blocks based on the details page definition from the webpage screenshots; If the webpage contains the target features based on the multiple content blocks, and the webpage does not meet the exclusion conditions, then the webpage is determined to be the details page information. The target features include: a main image containing the target object, and a detailed description block included in the plurality of content blocks; The exclusion criteria include at least one of the following: The webpage screenshot contains pagination navigation; The webpage screenshots are categorized as articles in the form of instruction manuals; The screenshot is a screenshot of the website's homepage.

14. The method according to claim 13, wherein, The step of extracting information from the details page according to preset information categories to obtain sub-information corresponding to each information category includes: Extract multiple images of the target object from the details page information; Extract the header image of the target object from the multiple images; Based on the industry-customized template of the target object, extract key fields from the details page information; Based on the large model, the images and text in the details page information are aligned to obtain integrated image and text information; The multiple pieces of information include the header image, the key fields, and the integrated text and image information.

15. The method according to claim 1, wherein, The method of content understanding corresponding to the information type of the information to be understood extracts multiple sub-information from the information to be understood, including: When the information type is a plain text document, line characters are extracted from the plain text document to obtain multiple sub-information composed of multiple line characters, and the line break format is recorded.

16. The method according to claim 1, wherein, The method of content understanding corresponding to the information type of the information to be understood extracts multiple sub-information from the information to be understood, including: If the information type is a table file, extract the table structure and table content from the table file; The multiple pieces of information include the table structure and the table content.

17. The method according to any one of claims 1-16, wherein, The step of converting the multiple sub-information into knowledge representation information for retrieval according to information modalities includes: For the text content in the multiple sub-information, a scalar is extracted from the text content, and the text content is understood based on a large model to obtain a vector of the text content; For the multimedia resources in the multiple sub-information, a vector of the multimedia resources is generated.

18. The method of claim 17, further comprising: In response to a query request, the system retrieves the corresponding response material from the knowledge set based on a hybrid retrieval mode. The provided response materials are input into the large model to generate the response content for the query request.

19. The method according to claim 18, wherein, The step of retrieving the response material corresponding to the query request from the knowledge set includes: For the editable files in the knowledge set, the response materials are selected based on the semantic structure information of each sub-information.

20. The method according to claim 18, wherein, The step of retrieving the response material corresponding to the query request from the knowledge set includes: For the non-editable files in the knowledge set, the response materials are selected based on the layout structure information of each sub-information.

21. The method according to claim 18, wherein, The step of retrieving the response material corresponding to the query request from the knowledge set includes: For the target object in the knowledge set, the response materials are selected based on the industry classification tags and integrated text and image content of the target object.

22. An information processing device based on artificial intelligence, comprising: The extraction module is used to extract multiple sub-information from the information to be understood by adopting the content understanding method corresponding to the information type of the information to be understood; The conversion module is used to convert the multiple sub-information into knowledge representation information for retrieval according to information modality; The association module is used to associate and store knowledge representation information of multiple semantically related sub-information to obtain a knowledge set; The knowledge set includes scalars and vectors.

23. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-21.

24. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-21.

25. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-21.

Images