Text detection method and system

Abstract

The invention relates to the technical field of text detection, in particular to a text detection method and system. The method comprises the following steps: acquiring a text use demand; obtaining a rich text historical file according to a text use requirement; analyzing the rich text historical file, and determining structured multi-type rich texts; determining detection demand guide information according to the text use demand and the structured multi-type rich text; judging whether multimedia fusion content exists in the detection demand guide information or not; if the multimedia fusion content exists, determining a detection result information set based on the multimedia fusion content; and carrying out structured text output on the detection result information set. Accuracy and reliability of data are guaranteed, and interference of irrelevant data is reduced. The information is more convenient to analyze and process, the practical value of the information is improved, the readability and the understandability of a detection result information set are improved, a user can quickly obtain the information, a computer can conveniently process and analyze the information, and support is provided for subsequent data mining and application.

Description

Technical Field

[0001] The present application relates to the technical field of text detection, and in particular to a text detection method and system. Background Art

[0002] In the information content recording system, content providers need to ensure that the recorded information is accurate and reproducible for subsequent use and analysis. In this process, they need to record various information in detail, including event status, processing steps, key elements involved, and problems encountered. This information is crucial for subsequent content management and in-depth analysis.

[0003] Traditional recording methods often rely on unstructured free rich text. Although this recording method is flexible, the historical archives recorded by content providers have obvious deficiencies in consistency and comparability. Key information in free rich text is difficult to detect quickly and accurately, making it difficult to extract and integrate the required information, which in turn affects the efficiency of subsequent data analysis.

[0004] In areas such as news content provision and the establishment of corporate knowledge bases, accurate and structured information records are equally important. They help improve the efficiency of content management and provide strong support for subsequent analysis and utilization. Summary of the invention

[0005] The present application provides a text detection method and system to solve the above problems.

[0006] In a first aspect, the present application provides a text detection method, the method comprising:

[0007] Acquire text usage requirements; acquire rich text history archives based on the text usage requirements; parse the rich text history archives to determine structured multi-type rich texts; determine detection requirement-oriented information based on the text usage requirements and the structured multi-type rich texts; determine whether there is multimedia fusion content in the detection requirement-oriented information; if so, determine a detection result information set based on the multimedia fusion content; and output the detection result information set in structured text.

[0008] Through the above technical scheme, rich text historical archives are obtained in a targeted manner according to the text usage requirements, which ensures the accuracy and reliability of the data and reduces the interference of irrelevant data; by parsing the rich text historical archives, deeply understanding its content and structure, and determining the structured multi-type rich text, information can be extracted more specifically, improving the efficiency of information extraction, ensuring the accuracy and relevance of the detection demand-oriented information, and making the extracted detection demand-oriented information more convenient for subsequent data analysis and processing; it is identified whether there is multimedia fusion content in the detection demand-oriented information. If so, the multimedia fusion content is marked, and the compiled detection result information set is convenient for subsequent use and analysis, thereby improving the practical value of the information; structured text output improves the readability and comprehensibility of the detection result information set, facilitates users to quickly obtain information, and facilitates computer processing and analysis, providing support for subsequent data mining and application.

[0009] Optionally, determining the detection result information set based on the multimedia fusion content includes: dividing the multimedia fusion content into text data and image data; parsing the text data to determine the paragraph line information and semantic information of the text data; parsing the image data to determine the image coordinates; determining the target image and the target image coordinates referring to the corresponding text according to the semantic information, the image coordinates and the paragraph line information; determining the valid text component represented by the target image according to the target image and the semantic information; determining the detection result information set according to the valid text component, the semantic information, the paragraph line information and the target image coordinates.

[0010] Through the above technical solution, the multimedia fusion content is effectively divided into text data and image data, ensuring the accurate extraction of text data and image data, avoiding information loss, and providing a basis for the subsequent processing of these two types of data. Parsing the text data to obtain its detailed structural information (paragraph line information) and semantic information is helpful to understand the text layout, and provides a basis for the positional relationship judgment for the subsequent determination of the relationship between text and image. Parsing the image data to obtain its coordinate information plays an important role in the subsequent extraction of effective text components and providing accurate target images and their coordinates corresponding to the text content; extracting the effective text components represented by the target image, supplementing the deficiencies of the text data, and providing supplementary data for the subsequent construction of the detection result information set. Forming a detection result information set containing rich information such as text, images, coordinates, etc., provides convenience for subsequent output and use, meets user needs, improves the consistency and comparability of archive content records, and quickly and accurately detects, extracts and integrates required information.

[0011] Optionally, determining the target image and target image coordinates referring to the corresponding text according to the semantic information, the image coordinates and the paragraph line information includes: calculating the association degree between each text data and each image data based on the semantic information, the image coordinates and the paragraph line information, and determining the target image according to the association degree, referring to the following formula:

[0012] I sel =I j |I j ∈I,R ij >T;

[0013] Among them, I sel is the target image; I j is the jth data point of the image data; I is the set of image data; R ij is the degree of relevance;

[0014] According to the correlation degree, the target image coordinates are determined, referring to the following formula:

[0015]

[0016] Among them, C sel is the target image coordinate; for j * The image in the image data pointed to by the index; j * is the index of the target image corresponding to the text; j is the index of any image in the set of image data; R ij For the degree of relevance.

[0017] Through the above technical solution, the correlation between each text data and each image data is calculated, providing a calculation basis for the subsequent determination of the target image and the target image coordinates. According to the correlation, the target image most relevant to the data point of the text data at the location of the key information in the detected structured multi-type rich text is determined, providing a basis for the subsequent extraction of the target image coordinates. According to the correlation, the coordinates of the target image most relevant to the text data point at the location of the key information in the detected structured multi-type rich text are determined, providing accurate image location information for subsequent data analysis, information extraction and integration. Through the above calculation method, it is possible to accurately find images and texts with a strong correlation, thereby avoiding the image from being distorted and the image interpretation failing to express the true meaning of being placed here.

[0018] Optionally, the calculating the association degree between each text data and each image data based on the semantic information, the image coordinates and the paragraph line information includes: extracting a keyword set from the set of text data; extracting a keyword set from the set of image data; calculating the size of the intersection of the keyword set in the set of text data and the keyword set in the set of image data to obtain a keyword matching score, referring to the following formula:

[0019] K rat (K Ti ,K lj )=|K Ti ∩K Ij |;

[0020] Among them, K rat is the keyword matching score; K Ti is the set of keywords in the text data set; K j is a set of keywords in a set of image data;

[0021] Convert the set of text data into text semantic vectors; convert the set of image data into image semantic vectors; calculate the cosine similarity between the text semantic vector and the image semantic vector to obtain the semantic matching score, refer to the following formula;

[0022]

[0023] Where: V rat (V Ti ,V lj ) is the cosine similarity; dot(V Ti ,V Ij ) is the dot product formula of text semantic vector and image semantic vector; ||V Ti || is the length of the text semantic vector; ||V Ij || is the length of the image semantic vector;

[0024] Calculate the proximity between the target image coordinates and the paragraph line information corresponding to the keyword set in any text data to obtain a position relationship score; obtain the association between each text data and each image data according to the keyword matching score, the semantic matching score and the position relationship score, referring to the following formula:

[0025]

[0026] Among them, R ij is the correlation degree; ε is the weight coefficient of the position relationship score; K rat (K Ti ,K lj ) is the keyword matching score; is the weight coefficient of semantic matching score; V rat (V Ti ,V lj ) is the semantic matching score; η is the weight coefficient of the position relationship score; S core Score the positional relationship.

[0027] Through the above technical solution, combined with the keyword set of text and image, the key information in the archive content record can be extracted more accurately, which is more comprehensive and accurate than the method that relies solely on text or image. The calculation of the correlation takes into account the keywords, semantics and positional relationships between text and image, which helps to identify and consistent information in the archive content record, thereby improving the comparability between data. By calculating the correlation between text and image, the required information can be quickly and accurately located, avoiding the tedious process of manual search and sorting in unstructured free rich text, greatly improving the efficiency of data analysis; using the above formula to calculate the correlation makes the information in the archive content record easier to integrate and use; by adjusting the weight coefficients corresponding to the keyword matching score, semantic matching score and position relationship score, it can flexibly adapt to different application scenarios and needs, which makes this method have good scalability and adaptability.

[0028] Optionally, the calculation of the proximity between the target image coordinates and paragraph line information corresponding to a keyword in a keyword set in any text data to obtain a positional relationship score includes: parsing the proximity between the target image coordinates and paragraph line information corresponding to a keyword in a keyword set in any text data to obtain the difference between the row numbers of the text and the image, the difference between the column numbers of the text and the image, and the surrounding situation between the text and the image; obtaining the surrounding distance of the text on the X-axis of the image and the surrounding distance of the text on the Y-axis of the image according to the surrounding situation of the image and the text; calculating the positional relationship score according to the difference in row numbers, the difference in column numbers, the surrounding distance on the X-axis, and the surrounding distance on the Y-axis, referring to the following formula:

[0029]

[0030] Among them, α is the weight coefficient of the absolute value of the difference between row numbers; D row is the absolute value of the difference in row numbers; β is the weight coefficient of the difference in column numbers; D col is the difference in column numbers; γ is the weight coefficient of the surrounding distance on the X-axis; is the wrapping distance on the X axis; δ is the weight coefficient of the wrapping distance on the Y axis; is the wrap-around distance on the Y axis.

[0031] Through the above technical solution, by carefully analyzing the relative positions of text and images in the document, including the difference in line numbers, column numbers and surrounding conditions, the text information closely related to the image can be more accurately located; the calculation of the position relationship score takes into account the spatial layout relationship between text and image, which helps to identify and compare the consistency information in the archival content record. Even in different archival content records, as long as the spatial relationship between text and image is similar, their position relationship scores will be similar, thereby improving the comparability between data. By calculating the position relationship score, we can quickly and accurately locate the text information closely related to the image, avoiding the tedious process of manual search and sorting in unstructured free rich text, and greatly improving the efficiency of data analysis. This method is not only applicable to text data, but also combines image data to support the fusion analysis of multimedia data, which is particularly important for the analysis of image information such as photos that contain a large amount of key information in archival content records.

[0032] Optionally, determining the valid text components represented by the target image based on the target image and the semantic information includes: parsing the target image to determine whether the target image has text content; if so, extracting the image text content of the target image; interpreting based on the image text content and the semantic information of the image screen connection context, extracting the valid text components in the image through the image text content, image interpretation information and the semantic information; if not, interpreting based on the semantic information of the image screen connection context, and using the image interpretation information as the valid text component.

[0033] Through the above technical scheme, not only the text content that may exist in the image is considered, but also the information conveyed by the image itself is considered, which improves the comprehensiveness of information extraction. By extracting the text content in the image and combining it with semantic information, the unstructured free rich text can be converted into a more structured and understandable form, which helps to better understand and utilize the information in the archive content record. Consistent key information is extracted from different types of archive content records (including text and images), thereby improving the consistency and comparability of archive content records, and also helps to compare and analyze between multiple archive content records to find potential problems and improvement points. The process of obtaining the above effective text components is highly applicable to image data, supports the fusion analysis of multimedia data, and is particularly important for the analysis of image information such as photos contained in archive content records. By extracting the effective text components in the image, the key information in the archive content record can be located more quickly and accurately, which avoids the tedious process of manual search and sorting in unstructured free rich text, thereby optimizing the efficiency of data analysis.

[0034] Optionally, the semantic information based on the image and screen connection context is interpreted, and the image interpretation information is used as a valid text component, including: interpreting based on the image and screen connection context to obtain interpretation information; calculating the similarity between the interpretation information and the semantic information; if the similarity is greater than a preset threshold, the interpretation information is used as a valid text component.

[0035] Through the above technical solution, the image and contextual semantic information are combined for interpretation, and the specific meaning of the image in the archive content record is understood more accurately, so as to extract more accurate information. At the same time, not only the text content in the image is considered, but also the information conveyed by the image itself is considered, which improves the comprehensiveness of information extraction; unstructured free rich text and image information are converted into structured interpretation information, which is convenient for subsequent data analysis and processing. Structured interpretation information improves the consistency and comparability of archive content records, making it easier to compare and analyze between multiple archive content records; by extracting effective text components in the image, the key information in the archive content record can be more accurately located.

[0036] Optionally, determining a detection result information set according to the valid text components, the semantic information, the paragraph line information and the target image coordinates includes: determining a distribution state of the valid text components based on the valid text components according to the target image coordinates, the paragraph line information and the semantic information; performing multimodal information fusion on the valid text components and the semantic information according to the distribution state to obtain a detection result information set.

[0037] Through the above technical solution, the effective text components, semantic information, paragraph line information and target image coordinates are combined to more accurately determine the distribution of text in the image, thereby improving the accuracy of information extraction. The unstructured free rich text and image information are converted into a structured detection result information set to facilitate subsequent data analysis and processing. Through multimodal information fusion, the effective text components are integrated with the semantic information of the text to form a more unified, rich and comprehensive detection result information set.

[0038] Optionally, the structured text output of the detection result information set includes: filtering the detection result information set according to query information to obtain a filtered information set; obtaining a preset structured template; filling the structured template with information through the filtered information set to obtain a streamlined information set; and structured text output of the streamlined detection result information set.

[0039] Through the above technical solution, the test result information set is screened according to the query information, and the required information is quickly and accurately located, avoiding the tedious process of manual search in unstructured text, and improving the efficiency of subsequent processing. The screened information is filled with a preset structured template to ensure the unified format and standard of the output information, improve the consistency and comparability of the archive content records, make the information clearer and easier to understand, and facilitate subsequent data analysis. It can be directly used for report generation, data visualization or further data processing and analysis. The structured text output allows users to more clearly understand the specific circumstances and background in the archive records, which helps to make more accurate judgments and decisions. Because the information has been screened and structured, the efficiency and accuracy of data analysis have been improved.

[0040] In a second aspect, the present application provides a text detection system, the system comprising:

[0041] A rich text history archive acquisition module is used to acquire text usage requirements; and acquire rich text history archives according to the text usage requirements;

[0042] A structured multi-type rich text parsing module, used to parse the rich text historical archives and determine the structured multi-type rich text;

[0043] A detection requirement-oriented information determination module, used to determine detection requirement-oriented information according to the text usage requirements and the structured multi-type rich text;

[0044] A multimedia content determination module, used to determine whether there is multimedia fusion content in the detection demand-oriented information;

[0045] A detection result information set construction module, for determining the detection result information set based on the multimedia fusion content, if any;

[0046] The structured text output module is used to output the detection result information set in the form of structured text. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0048] Figure 1 A schematic diagram of an application scenario provided for an embodiment of the present application;

[0049] Figure 2A flowchart of a text detection method provided in one embodiment of the present application;

[0050] Figure 3 A schematic diagram of the structure of a text detection system provided in one embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0052] In addition, the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article, unless otherwise specified, generally means that the associated objects before and after are in an "or" relationship.

[0053] The embodiments of the present application are further described in detail below in conjunction with the drawings in the specification.

[0054] The way to record archive content often relies on unstructured free rich text. Although this recording method is flexible, the consistency and comparability of the rich text historical archives recorded by the users are extremely low. The key information in the free rich text cannot be quickly and accurately detected, so the required information cannot be extracted and integrated, making subsequent data analysis extremely difficult.

[0055] Based on this, the present application provides a text detection method and system to obtain text usage requirements; based on the text usage requirements, targeted acquisition of rich text historical archives ensures data accuracy and reliability and reduces interference from irrelevant data; parses the rich text historical archives to determine structured multi-type rich texts; by parsing the archives, an in-depth understanding of their content and structure is achieved, and after determining the structured multi-type rich texts, information can be extracted more targetedly to improve information extraction efficiency; based on the text usage requirements and the structured multi-type rich texts, detection demand-oriented information is determined; the accuracy and relevance of the detection demand-oriented information are ensured, and the extracted detection demand-oriented information is more convenient for subsequent data analysis and Processing; judging whether there is multimedia fusion content in the detection demand-oriented information; being able to identify the multimedia fusion content in the detection demand-oriented information, such as images, text and other information forms, and marking the multimedia fusion content to facilitate subsequent targeted processing and analysis; if it exists, then based on the multimedia fusion content, determining the detection result information set, the sorted detection result information set is more convenient for subsequent use and analysis, and improves the practical value of the information; outputting the detection result information set in structured text; structured text output improves the readability and comprehensibility of the information, facilitates users to quickly obtain information, facilitates computer processing and analysis, and provides support for subsequent data mining and application.

[0056] Figure 1 A schematic diagram of an application scenario provided by the present application, when detecting key information quickly and accurately in a free rich text, the method provided by the present application is applied, the method provided by the present application is applied to any server, the server, the text query terminal system and the archive content recording system interact, and the text usage demand is obtained from the text query terminal system; according to the text usage demand, the rich text historical archives are obtained from the archive content recording system in a targeted manner, the accuracy and reliability of the data are guaranteed, and the interference of irrelevant data is reduced; by parsing the rich text historical archives, the content and structure are deeply understood, and after determining the structured multi-type rich text, information can be extracted more targetedly, the efficiency of information extraction is improved, the accuracy and relevance of the detection demand-oriented information are ensured, and the extracted detection demand-oriented information is more convenient for subsequent data analysis and processing; identify whether there is multimedia fusion content in the detection demand-oriented information, if there is, mark the multimedia fusion content, and organize the detection result information set, which is convenient for subsequent use and analysis, and improves the practical value of the information; the structured text output improves the readability and comprehensibility of the detection result information set, which is convenient for users to quickly obtain information, convenient for computers to process and analyze, and provides support for subsequent data mining and application. The specific implementation method can refer to the following embodiments.

[0057] Figure 2This is a flowchart of a text detection method provided in an embodiment of the present application. The method of this embodiment can be applied to the server in the above scenario. Figure 2 As shown, the method includes:

[0058] S201. Obtain text usage requirements; and obtain rich text historical archives based on the text usage requirements.

[0059] The text usage requirements may be specific requirements such as regular inspection, calling, content extraction or summarization of rich text historical archives. Rich text historical archives can be detailed records of an event, usually in the form of text or images.

[0060] Specifically, the text usage requirements of users are obtained from the text query terminal system, and relevant rich-text historical archives are retrieved from the archive content recording system based on key information in the text usage requirements.

[0061] S202, parsing rich text historical archives to determine structured multi-type rich texts.

[0062] Structured multi-type rich text can be a specific structure of organization and information hierarchy. Structured multi-type rich text can include multiple parts such as title, summary, specific details, problem record, treatment measures, result feedback, etc. Each part may contain specific information fields, formats or images.

[0063] Specifically, since the acquired rich-text historical archives have complex contents and may be composed of multiple historical archive content records, their structure is generally filled in according to a fixed template, but their internal contents vary greatly due to different users, recording issues and other specific circumstances. Therefore, it is necessary to identify the acquired rich-text historical archives, obtain the overall structure of the rich text, and determine the key information and format expressed in the content of each structure.

[0064] S203. Determine detection demand-oriented information based on text usage requirements and structured multi-type rich text.

[0065] The detection demand-oriented information can be specific information leading to the detection results extracted from the rich text historical archives according to the text usage requirements; the detection demand-oriented information includes multiple record types such as pure text type, pure image type or a combination of text and image type.

[0066] Specifically, when a rich-text historical archive is generated each time a record is generated, multiple issues may be recorded in parallel at the same time, resulting in the existence of records of several other issues and actual situations that are not strongly relevant to the current text usage needs; at the same time, there may be multiple historical archive content records available for use in a rich-text historical archive; therefore, it is necessary to combine the text usage needs and structured multi-type rich texts to determine the location of the key information in the structured multi-type rich texts that need to be detected, so as to extract the specific information that leads to the detection results.

[0067] S204: Determine whether there is multimedia fusion content in the detection demand-oriented information.

[0068] Multimedia fusion content can be other media forms of content in addition to text contained in rich text historical archives, such as a combination of text and images, etc. Multimedia fusion content includes types such as pure image type or image-text combination type. Specifically, sometimes in order to accurately record the current problem and reproduce the problem later to formulate corresponding plans or materials, images taken of the current problem are usually interspersed in the text. These images play a vital role in describing the specific situation of the problem, such as an event + image that occurred at a certain time. In this process, if only the text is detected and recognized, the specific situation cannot be obtained, such as the state description of the news scene and many other situations that can only be determined based on the image content. Therefore, it is necessary to traverse and analyze the detection demand-oriented information. If the detection demand-oriented information contains image content, it is judged that the structured multi-type rich text content where the detection demand-oriented information is located contains multimedia fusion content; if the detection demand-oriented information does not contain image content, it is judged that the structured multi-type rich text content where the detection demand-oriented information is located does not contain multimedia fusion content.

[0069] S205: If it exists, determine the detection result information set based on the multimedia fusion content.

[0070] The detection result information set can be a set of information related to the detection result sorted out according to the text usage requirements and the multimedia fusion content. Specifically, if it is detected that there is multimedia fusion content in the demand-oriented information, then according to the type and characteristics of the multimedia content, the characteristics are mainly the layout characteristics of the image and text, such as the upper and lower surround or text surround, and the existence of the picture name reference, the picture name should be replaced with the image and its characteristics should be determined. On this basis, the corresponding detection result is determined, and the detection result is combined with the relevant text information to form a detection result information set. In other implementations, if the presence of multimedia fusion content in the demand-oriented information is not detected, the text in the detection demand-oriented information is directly detected to determine the detection result information set, and structured text output is performed in the subsequent process.

[0071] S206: Output the detection result information set in structured text.

[0072] The structured text may be a text format in which the detection result information set is arranged in a predetermined format and organization. The predetermined format and organization may be preset in different structured forms according to different requirements so as to be used by different ports.

[0073] Specifically, in the test result information, the content recorded in the original archive content record may be complex, which may be complicated to read or subsequently process and difficult to understand, so the text and multimedia content in the test result information set are structured to form an output format that is easy to read and understand.

[0074] Through the method provided in this embodiment, rich text historical archives are obtained in a targeted manner according to text usage requirements, thereby ensuring the accuracy and reliability of the data and reducing the interference of irrelevant data; by parsing the rich text historical archives, deeply understanding its content and structure, and determining the structured multi-type rich text, information can be extracted more specifically, improving the efficiency of information extraction, ensuring the accuracy and relevance of the detection demand-oriented information, and making the extracted detection demand-oriented information more convenient for subsequent data analysis and processing; identifying whether there is multimedia fusion content in the detection demand-oriented information, and if so, marking the multimedia fusion content, and sorting out the detection result information set, which is convenient for subsequent use and analysis, and improves the practical value of the information; structured text output improves the readability and comprehensibility of the detection result information set, facilitates users to quickly obtain information, and facilitates computer processing and analysis, providing support for subsequent data mining and application.

[0075] In some embodiments, the multimedia fusion content is divided into text data and image data; the text data is parsed to determine the paragraph line information and semantic information of the text data; the image data is parsed to determine the image coordinates; based on the semantic information, image coordinates and paragraph line information, the target image and target image coordinates referring to the corresponding text are determined; based on the target image and the semantic information, the valid text component represented by the target image is determined; based on the valid text component, the semantic information, the paragraph line information and the target image coordinates, a detection result information set is determined.

[0076] Text data can be the free rich text part in the rich text historical archive, including all text information recorded by the user, such as problem description, solution, etc. Image data can be images or other visual elements contained in the multimedia fusion content, which are associated with the text data and used to assist in explaining or supplement the text information. Paragraph line information can be the structural information of the text data, including the division of paragraphs, the arrangement of lines, etc., which is used to determine the organizational structure and layout of the text. Semantic information can be the meaning and content in the text data, including keywords, phrases, sentences and the relationship between them, which is used to understand the information and intention conveyed by the text. Image coordinates can be the position information of the image data in the multimedia fusion content, including the starting point, size and direction of the image, etc., which is used to determine the relative position relationship between the image and the text data. The target image can be an image in the image data associated with the text content at the location of the key information in the detected structured multi-type rich text. These images may contain intuitive displays or supplementary explanations of the text content. The target image coordinates can be the specific position information of the target image in the multimedia fusion content, including its starting point, size and direction, etc., which is used to accurately locate the target image. The effective text component may be text information extracted from the target image by image analysis technology, and such text information is of great significance for understanding the image content or the text data associated therewith.

[0077] Specifically, for multimedia fusion content, since these images are generally taken directly or simply annotated and stored in documents, if only the text and image are processed separately, the content expressed by the image will be distorted in the separate processing method that can be achieved in the existing technology, that is, the context will be lost, and the meaning of the image being placed in that position cannot be fully and appropriately reflected, and more content cannot be interpreted in the image. The multimedia fusion content is traversed and divided according to the data type (text or image) in the multimedia fusion content. The text data is extracted from the multimedia fusion content using text extraction technology, and the image data is extracted from the multimedia fusion content using image extraction technology. The extracted text data is parsed to identify the paragraph and line structure and determine the paragraph line information. The text data is semantically analyzed using natural language processing tools to extract semantic information. The extracted image data is parsed to identify the image coordinates of the image in the multimedia fusion content. The reference relationship between the image and the text is analyzed by combining the semantic information, image coordinates and paragraph line information to determine the target image that refers to the corresponding text and the target image coordinates of the target image in the original multimedia fusion content. The target image is further analyzed and the text content represented by the target image is analyzed by combining the semantic information, that is, the effective text component represented by the target image is extracted. The detection result information set is constructed by combining the effective text component, semantic information, paragraph line information and target image coordinates. The detection result information set is structured for subsequent output and use.

[0078] Through the method provided by this embodiment, the multimedia fusion content is effectively divided into text data and image data, ensuring the accurate extraction of text data and image data, avoiding information loss, and providing a basis for the subsequent processing of these two types of data. Parsing the text data to obtain its detailed structural information (paragraph line information) and semantic information is helpful to understand the text layout, and provides a basis for the subsequent determination of the relationship between the text and the image. Parsing the image data to obtain its coordinate information plays an important role in the subsequent extraction of effective text components to provide accurate target images and their coordinates corresponding to the text content; extracting the effective text components represented by the target image, supplementing the deficiencies of the text data, and providing supplementary data for the subsequent construction of the detection result information set. Forming a detection result information set containing rich information such as text, image, coordinates, etc., provides convenience for subsequent output and use, meets user needs, improves the consistency and comparability of archive content records, and quickly and accurately detects, extracts and integrates required information.

[0079] In some embodiments, the association degree between each text data and each image data is calculated based on the semantic information, image coordinates and paragraph line information, and the target image is determined according to the association degree, referring to the following formula (1):

[0080] Isel =I j |I j ∈I,R ij >T (1)

[0081] Among them, I sel is the target image; I j is the jth data point of the image data; I is the set of image data; R ij is the degree of relevance;

[0082] According to the degree of association, the target image coordinates are determined, referring to the following formula (2):

[0083]

[0084] Among them, C sel is the target image coordinate; for j * The image in the image data pointed to by the index; j * is the index of the target image corresponding to the text; j is the index of any image in the set of image data; R ij For the degree of relevance.

[0085] The correlation degree can be an indicator to measure the degree of correlation between the data points of text data and the data points of image data. The index of the target image can be the unique position information corresponding to the image data point in the image data set whose correlation degree with the specific text data point is greater than a threshold value, and the threshold value needs to be set based on experimental data or experience. The image data set can be a data set containing multiple image data points, each of which contains certain image information and possible metadata.

[0086] Specifically, in the search for the relationship between image and text, how to judge the correlation between image and text is crucial. For example, the text in the previous structure cannot be used to interpret the image in the current structure. This will cause a distorted interpretation of the image content. Therefore, a certain relationship strength between the image and the text is required to interpret the image based on the semantic information of the text, which means that the image needs to select one or more paragraphs of text as the background for image interpretation. The specific implementation method is as follows: Combine semantic information, image coordinates and paragraph line information, analyze the potential relationship between text data and image data, use the correlation calculation method to calculate the correlation between each text data and each image data, and obtain a correlation matrix, where each element in the correlation matrix represents the degree of correlation between a data point of text data and a data point of image data. Traverse the correlation matrix to find the image data point whose correlation with the text data at the location of the key information in the detected structured multi-type rich text is greater than the threshold, and determine it as the target image. Use formula (1) to calculate the index of the target image, and select the corresponding target image from the image data set according to the index. Using the target image as an index, the corresponding target image coordinates are extracted from the image data set according to formula (2).

[0087] Through the method provided in this embodiment, the correlation between each text data and each image data is calculated, providing a calculation basis for the subsequent determination of the target image and the target image coordinates. According to the correlation, the target image that is most relevant to the data point of the text data at the location of the key information in the detected structured multi-type rich text is determined, providing a basis for the subsequent extraction of the target image coordinates. According to the correlation, the coordinates of the target image that is most relevant to the text data point at the location of the key information in the detected structured multi-type rich text are determined, providing accurate image location information for subsequent data analysis, information extraction and integration. The above calculation method can accurately find the strong correlation between the image and the text, thereby avoiding the image from being distorted and the image interpretation failing to express the true meaning of being placed here.

[0088] In some embodiments, a keyword set is extracted from a set of text data; a keyword set is extracted from a set of image data; and the intersection size of the keyword set in the set of text data and the keyword set in the set of image data is calculated to obtain a keyword matching score, referring to the following formula (3):

[0089] K rat (K Ti ,K lj )=|K Ti ∩K Ij | (3)

[0090] Among them, K rat K is the keyword matching score; Tiis the set of keywords in the text data set; K j is a set of keywords in a set of image data;

[0091] Convert the set of text data into text semantic vectors; convert the set of image data into image semantic vectors; calculate the cosine similarity between the text semantic vector and the image semantic vector to obtain a semantic matching score, refer to the following formula (4);

[0092]

[0093] Where: V rat (V Ti ,V lj ) is the cosine similarity; dot(V Ti ,V Ij ) is the dot product formula of text semantic vector and image semantic vector; ||V Ti || is the length of the text semantic vector; ||V Ij || is the length of the image semantic vector;

[0094] Calculate the proximity between the target image coordinates and the paragraph line information corresponding to the keyword set in any text data to obtain the position relationship score; according to the keyword matching score, semantic matching score and position relationship score, obtain the association between each text data and each image data, refer to the following formula (5):

[0095]

[0096] Among them, R ij is the correlation degree; ε is the weight coefficient of the position relationship score; K rat (K Ti ,K lj ) is the keyword matching score; is the weight coefficient of semantic matching score; V rat (V Ti ,V lj ) is the semantic matching score; η is the weight coefficient of the position relationship score; S core Score the positional relationship.

[0097] The keyword set in the text data set may be a representative or important vocabulary set extracted from the text data. The keyword set in the image data set may be a keyword set further extracted from the text information extracted from the image data by image recognition or OCR technology. The intersection size of the keyword set may be the number of keywords in common in the keyword sets of the text data and the image data. The keyword matching score may be a score calculated based on the keyword intersection size, which is used to measure the degree of matching between the text and the image at the keyword level. The text semantic vector may be a vector form converted from the text data to represent the semantic information of the text. The image semantic vector may be a vector form converted from the image data to represent the semantic information of the image. The cosine similarity may be a measure of the similarity of two vectors in direction, which is used to compare the similarity of the text and image semantic vectors. The semantic matching score may be a score calculated based on the similarity of the text and image semantic vectors, which is used to measure their matching degree at the semantic level. The length of the text semantic vector may be the Euclidean length of the text semantic vector in a multidimensional space. The length of the image semantic vector may be the Euclidean length of the image semantic vector in the multidimensional space. The dot product formula may be the result of summing the corresponding elements of two vectors after multiplication, which is used to calculate the similarity between the vectors. The proximity may be a measure of the proximity between the positional relationship of the text and the image. The positional relationship score may be a score calculated based on the positional relationship between the text and the image in the multimedia fusion content, which is used to measure their relevance in spatial layout.

[0098] Specifically, in the process of determining the degree of association, three dimensions can be used to quantify the above three dimensions, namely, the degree of keyword matching between the image and the text, the degree of semantic matching between the image and the text, and the positional relationship between the image and the text. Specifically, the keywords in the set of text data can be extracted by a keyword extraction algorithm to form a keyword set of the set of text data; the keywords in the set of image data can be extracted by image recognition technology; the intersection size of the keyword set of each text data and the keyword set of each image data is calculated according to formula (3); the intersection size is recorded as the keyword matching score. The text data is converted into a text semantic vector using a pre-trained text embedding model; the image data is converted into an image semantic vector using a pre-trained image embedding model; the cosine similarity between the text semantic vector and the image semantic vector is calculated according to formula (4), and the cosine similarity is recorded as the semantic matching score. The paragraph line information corresponding to the keyword set in the text data is extracted; the proximity between the target image coordinates and the paragraph line information is calculated, and the proximity is recorded as the positional relationship score. According to the keyword matching score, the semantic matching score and the positional relationship score, the degree of association is calculated according to formula (5).

[0099] Through the method provided by this embodiment, the key information in the archive content record is extracted more accurately by combining the keyword set of text and image, which is more comprehensive and accurate than the method that relies solely on text or image. The calculation of the correlation takes into account the keywords, semantics and positional relationships between text and image, which helps to identify the consistency information in the archive content record, thereby improving the comparability between data. By calculating the correlation between text and image, the required information can be located quickly and accurately, avoiding the tedious process of manual search and sorting in unstructured free rich text, greatly improving the efficiency of data analysis; using the above formula to calculate the correlation makes the information in the archive content record easier to integrate and use; by adjusting the weight coefficients corresponding to the keyword matching score, semantic matching score and positional relationship score, it can flexibly adapt to different application scenarios and needs, which makes the method have good scalability and adaptability.

[0100] In some embodiments, the proximity between the target image coordinates and the paragraph line information corresponding to the keywords in the keyword set in any text data is analyzed to obtain the difference between the row numbers where the text and the image are located, the difference between the column numbers where the text and the image are located, and the surrounding situation between the text and the image; according to the surrounding situation of the image and the text, the surrounding distance of the text on the X axis of the image and the surrounding distance of the text on the Y axis of the image are obtained;

[0101] The positional relationship score is calculated based on the difference in row numbers, column numbers, the wrapping distance on the X axis, and the wrapping distance on the Y axis, referring to the following formula (6):

[0102]

[0103] Among them, α is the weight coefficient of the absolute value of the difference between row numbers; D row is the absolute value of the difference in row numbers; β is the weight coefficient of the difference in column numbers; D col is the difference in column numbers; γ is the weight coefficient of the surrounding distance on the X-axis; is the wrapping distance on the X axis; δ is the weight coefficient of the wrapping distance on the Y axis; is the wrap-around distance on the Y axis.

[0104] The difference in row numbers may be the difference between the row number where the target image is located and the row number of the paragraph corresponding to the keyword in the text data. The difference in column numbers may be the difference between the column number where the target image is located and the column number of the paragraph corresponding to the keyword in the text data. The wrapping condition may be whether the text wraps around the image and the direction and degree of the wrapping. The wrapping distance may be the distance between the text and the image boundary when the text wraps around the image.

[0105] Specifically, in the process of quantifying the positional relationship between the image and the text, it is necessary to consider the layout features of the image and the text, such as the features of the upper and lower surround or text surround; in addition, there is also the state of indirect use of the image, such as the case of the reference of the image name. Specifically, the image name should be replaced with the image to determine its features. The specific quantification process of the above features can be referred to as follows: Extract the coordinate information of the target image, including its line number and column number in the document. For each keyword in the text data, find its corresponding paragraph line information, including the line number and column number. Calculate the line number difference and column number difference between the target image coordinates and the paragraph line information corresponding to each keyword. Analyze the surround situation between the text and the image, that is, whether the text is surrounded by the image, and the direction and degree of the surround. According to the surround situation of the text and the image, determine the relative position of the text on the X-axis and Y-axis of the image. Calculate the offset distance of the text on the X-axis relative to the center of the image as the surround distance on the X-axis. Calculate the offset distance of the text on the Y-axis relative to the center of the image as the surround distance on the Y-axis. According to the difference in row numbers, column numbers, the wrapping distance on the X axis and the wrapping distance on the Y axis, the position relationship score is calculated using formula (6) and can be adjusted according to actual conditions.

[0106] Through the method provided by this embodiment, by carefully analyzing the relative positions of text and image in the document, including the difference in line number, column number and surrounding situation, the text information closely related to the image can be more accurately located; the calculation of the position relationship score takes into account the spatial layout relationship between the text and the image, which helps to identify and the consistency information in the archival content record. Even in different archival content records, as long as the spatial relationship between the text and the image is similar, their position relationship scores will be similar, thereby improving the comparability between the data. By calculating the position relationship score, we can quickly and accurately locate the text information closely related to the image, avoiding the tedious process of manual search and sorting in unstructured free rich text, and greatly improving the efficiency of data analysis. This method is not only applicable to text data, but also combines image data to support the fusion analysis of multimedia data, which is particularly important for the analysis of image information such as photos that contain a large amount of key information in archival content records.

[0107] In some embodiments, the target image is parsed to determine whether the target image has text content; if so, the image text content of the target image is extracted; an interpretation is performed based on the image text content and the semantic information of the image screen in the context, and valid text components in the image are extracted through the image text content, image interpretation information and semantic information; if not, an interpretation is performed based on the semantic information of the image screen in the context, and the image interpretation information is used as a valid text component.

[0108] The text content can be the recognizable text content in the image. The image text content can be the readable text content extracted from the target image. The interpretation information can be the interpretation based on the image picture and contextual semantic information when the target image does not contain readable text content or the text content is insufficient to express complete information.

[0109] Specifically, when detecting the text content of an image, consider whether the image contains the annotations described above, use image recognition technology to identify the text area in the image, and determine whether there is recognizable text content; if there is text content, use image recognition tools to extract the text content in the target image, pre-process the extracted text content, combine the extracted image text content, image interpretation and contextual semantic information, perform text analysis, identify and extract key information related to the archive content record as an effective text component. If there is no text content, analyze the screen content of the target image, identify the subject, state, scene and other elements in the image, combine the contextual semantic information, interpret and describe the image content, and organize the image interpretation information into text form as an effective text component.

[0110] By the method provided by this embodiment, not only the text content that may exist in the image is considered, but also the information conveyed by the image itself is considered, thereby improving the comprehensiveness of information extraction. By extracting the text content in the image and combining it with semantic information, the unstructured free rich text can be converted into a more structured and understandable form, which helps to better understand and utilize the information in the archive content record. Consistent key information is extracted from different types of archive content records (including text and images), thereby improving the consistency and comparability of the archive content records, and also helping to compare and analyze between multiple archive content records to find potential problems and improvement points. The process obtained by the above-mentioned effective text components is highly applicable to image data, supports the fusion analysis of multimedia data, and is particularly important for the analysis of image information such as photos contained in the archive content records. By extracting the effective text components in the image, the key information in the archive content record can be located more quickly and accurately, which avoids the cumbersome process of manually searching and collating in the unstructured free rich text, thereby optimizing the efficiency of data analysis.

[0111] In some embodiments, interpretation is performed based on the image screen in conjunction with the context to obtain interpretation information; the similarity between the interpretation information and the semantic information is calculated; if the similarity is greater than a preset threshold, the interpretation information is used as a valid text component.

[0112] The similarity may be the degree of similarity between the interpretation information and the semantic information. The preset threshold may be a condition for determining whether the similarity is satisfied, obtained from experimental data or experience, and stored in a preset database.

[0113] Specifically, in the image interpretation information, the interpretation information needs to be related to the image content to avoid ambiguity, so it is necessary to check the similarity of the interpretation information; the specific implementation method is: analyze the screen content of the target image and identify the elements in the image; combine the contextual semantic information to parse the specific meaning of the image in the archive content record, generate interpretation information, and describe the meaning represented by the image and its association with the context. The image interpretation information and contextual semantic information are preprocessed with text, and the similarity between the interpretation information and the semantic information is calculated using the text similarity calculation algorithm to obtain a similarity score, which indicates the matching degree between the interpretation information and the semantic information. A similarity threshold is set as the standard for judging whether the interpretation information is valid, and the calculated similarity score is compared with the threshold; if the similarity score is greater than the threshold, the interpretation information is considered valid; otherwise, the interpretation information is considered invalid; if the interpretation information is judged to be valid, it is used as a valid text component of the archive content record.

[0114] Through the method provided by this embodiment, the image and contextual semantic information are combined for interpretation, and the specific meaning of the image in the archival content record is understood more accurately, thereby extracting more accurate information. At the same time, not only the text content in the image is considered, but also the information conveyed by the image itself is considered, which improves the comprehensiveness of information extraction; the unstructured free rich text and image information are converted into structured interpretation information, which is convenient for subsequent data analysis and processing. The structured interpretation information improves the consistency and comparability of the archival content record, making it easier to compare and analyze multiple archival content records; by extracting the effective text components in the image, the key information in the archival content record can be more accurately located.

[0115] In some embodiments, based on the valid text components, the distribution status of the valid text components is determined according to the target image coordinates, paragraph line information, and semantic information; multimodal information fusion is performed on the valid text components and semantic information according to the distribution status to obtain a detection result information set.

[0116] The effective text component distribution state can be the distribution and arrangement of effective information in the text. Multimodal information fusion can be the integration and fusion of information from different modalities to form a more comprehensive and richer information representation, such as text, images, etc.

[0117] Specifically, the interpretation information of an image is generally not a word, but a paragraph or multiple paragraphs of content. Its relevance to the text, the specific placement of each paragraph in the text, and many other factors that need to match the semantic environment need to be integrated again based on the text semantic information and paragraph line information; the distribution status of each valid text component in the text is determined based on the target image coordinates, paragraph line information, and semantic information. Among them, the valid text components are associated with the semantic information to ensure that each valid text component has a suitable corresponding position so as to better integrate with the semantic environment; according to the distribution status of the valid text components, the multimodal information fusion technology is used to fuse the valid text components and the semantic information of the text into a unified information set.

[0118] By combining the effective text components, semantic information, paragraph line information and target image coordinates, the distribution state of the text in the image can be determined more accurately, thereby improving the accuracy of information extraction. The unstructured free rich text and image information are converted into a structured detection result information set, which is convenient for subsequent data analysis and processing. Through multimodal information fusion, the effective text components are integrated with the semantic information of the text to form a more unified, rich and comprehensive detection result information set.

[0119] In some embodiments, the detection result information set is filtered according to the query information to obtain a filtered information set; a preset structured template is obtained; the structured template is filled with information through the filtered information set to obtain a streamlined information set; and the streamlined detection result information set is output as a structured text.

[0120] The filtered information set may be an information set obtained after filtering and processing the detection result information set. The preset structured template may be a structured format for filling information, and may include key fields of archive content records, such as recording time, user, event content, etc. It may be stored in a preset database. Information filling may be a process of filling the extracted key information into a preset structured template.

[0121] Specifically, determine the specific content of the query information, including keywords, time range, event type, etc. Traverse the detection result information set and perform a matching check on each information item to see if it meets the conditions of the query information. Filter out the information items that meet the conditions to form a filtered information set. Design and preset a structured template based on the specific needs and format requirements of the archive content record. Store the preset structured template in the database for subsequent use. Traverse the filtered information set and extract the key data in each information item. Fill the extracted key data into the corresponding fields of the structured template. Repeat the above steps until all filtered information items have been processed to obtain a streamlined information set. Convert the streamlined information set into text format. Ensure that the text format meets the preset structured template requirements. Output the structured text to the specified location or file.

[0122] Through the method provided by this embodiment, the detection result information set is screened according to the query information, and the required information is quickly and accurately located, avoiding the cumbersome process of manual search in unstructured text, and improving the efficiency of subsequent processing. The screened information is filled with a preset structured template to ensure the unified format and standard of the output information, improve the consistency and comparability of the archive content records, make the information clearer and easier to understand, and provide convenience for subsequent data analysis. It can be directly used for report generation, data visualization or further data processing and analysis. The structured text output allows users to understand the specific circumstances and background of the event more clearly, which helps to make more accurate judgments and decisions. Because the information is screened and structured, the efficiency and accuracy of data analysis are improved.

[0123] Figure 3 A structural diagram of a text detection system provided in an embodiment of the present application is shown in FIG. Figure 3 As shown, the text detection system 300 of this embodiment includes: a rich text history archive acquisition module 301, a structured multi-type rich text parsing module 302, a detection demand-oriented information determination module 303, a multimedia content judgment module 304, a detection result information set construction module 305 and a structured text output module 306.

[0124] Rich text historical archive acquisition module 301 is used to obtain text usage requirements; based on the text usage requirements, rich text historical archives are obtained; structured multi-type rich text parsing module 302 is used to parse rich text historical archives and determine structured multi-type rich texts; detection demand-oriented information determination module 303 is used to determine detection demand-oriented information based on text usage requirements and structured multi-type rich texts; multimedia content judgment module 304 is used to determine whether there is multimedia fusion content in the detection demand-oriented information; detection result information set construction module 305 is used to determine the detection result information set based on the multimedia fusion content if it exists; structured text output module 306 is used to output the detection result information set in structured text

[0125] Optionally, the detection result information set construction module 305 is specifically used to: divide the multimedia fusion content into text data and image data; parse the text data to determine the paragraph line information and semantic information of the text data; parse the image data to determine the image coordinates; determine the target image and target image coordinates referring to the corresponding text based on the semantic information, image coordinates and paragraph line information; determine the valid text components represented by the target image based on the target image and semantic information; determine the detection result information set based on the valid text components, semantic information, paragraph line information and target image coordinates.

[0126] The detection result information set construction module 305 is specifically used to calculate the association degree between each text data and each image data based on the semantic information, image coordinates and paragraph line information, and determine the target image according to the association degree, referring to the following formula:

[0127] I sel =I j |I j ∈I,R ij >T;

[0128] Among them, I sel is the target image; I j is the jth data point of the image data; I is the set of image data; R ij is the degree of relevance;

[0129] According to the degree of association, determine the target image coordinates, refer to the following formula:

[0130]

[0131] Among them, C sel is the target image coordinate; for j * The image in the image data pointed to by the index; j * is the index of the target image corresponding to the text; j is the index of any image in the set of image data; R ijFor the degree of relevance.

[0132] The detection result information set construction module 305 is specifically used to: extract a keyword set from a set of text data; extract a keyword set from a set of image data; calculate the intersection size of the keyword set in the set of text data and the keyword set in the set of image data to obtain a keyword matching score, referring to the following formula:

[0133] K rat (K Ti ,K lj )=|K Ti ∩K Ij |;

[0134] Among them, K rat K is the keyword matching score; Ti is the set of keywords in the text data set; K j is a set of keywords in a set of image data;

[0135] Convert the set of text data into text semantic vectors; convert the set of image data into image semantic vectors; calculate the cosine similarity between the text semantic vector and the image semantic vector to obtain the semantic matching score, refer to the following formula;

[0136]

[0137] Where: V rat (V Ti ,V lj ) is the cosine similarity; dot(V Ti ,V Ij ) is the dot product formula of text semantic vector and image semantic vector; ||V Ti || is the length of the text semantic vector; ||V Ij || is the length of the image semantic vector;

[0138] Calculate the proximity between the target image coordinates and the paragraph line information corresponding to the keyword set in any text data to obtain the position relationship score; according to the keyword matching score, semantic matching score and position relationship score, obtain the association between each text data and each image data, refer to the following formula:

[0139]

[0140] Among them, R ij is the correlation degree; ε is the weight coefficient of the position relationship score; K rat (K Ti ,K lj ) is the keyword matching score; is the weight coefficient of semantic matching score; Vrat (V Ti ,V lj ) is the semantic matching score; η is the weight coefficient of the position relationship score; S core Score the positional relationship.

[0141] The detection result information set construction module 305 is specifically used to: analyze the proximity between the target image coordinates and the paragraph line information corresponding to the keywords in the keyword set in any text data, and obtain the difference between the row numbers of the text and the image, the difference between the column numbers of the text and the image, and the surrounding situation between the text and the image; according to the surrounding situation of the image and the text, obtain the surrounding distance of the text on the X axis of the image and the surrounding distance of the text on the Y axis of the image; according to the difference between the row numbers, the difference between the column numbers, the surrounding distance on the X axis and the surrounding distance on the Y axis, calculate the position relationship score, referring to the following formula:

[0142]

[0143] Among them, α is the weight coefficient of the absolute value of the difference between row numbers; D row is the absolute value of the difference in row numbers; β is the weight coefficient of the difference in column numbers; D col is the difference in column numbers; γ is the weight coefficient of the surrounding distance on the X-axis; is the wrapping distance on the X axis; δ is the weight coefficient of the wrapping distance on the Y axis; is the wrap-around distance on the Y axis.

[0144] The detection result information set construction module 305 is specifically used to: parse the target image to determine whether the target image has text content; if so, extract the image text content of the target image; interpret based on the image text content and the semantic information of the image screen context, and extract the valid text components in the image through the image text content, image interpretation information and semantic information; if not, interpret based on the semantic information of the image screen context, and use the image interpretation information as the valid text component.

[0145] The detection result information set construction module 305 is specifically used to: interpret the image based on the context to obtain interpretation information; calculate the similarity between the interpretation information and the semantic information; if the similarity is greater than a preset threshold, the interpretation information is used as a valid text component.

[0146] The detection result information set construction module 305 is specifically used to: determine the distribution status of the effective text components based on the effective text components, according to the target image coordinates, paragraph line information, and semantic information; perform multimodal information fusion on the effective text components and semantic information according to the distribution status to obtain the detection result information set.

[0147] The structured text output module 306 is specifically used to: filter the detection result information set according to the query information to obtain a filtered information set; obtain a preset structured template; fill the structured template with information through the filtered information set to obtain a streamlined information set; and output the streamlined detection result information set in the form of structured text.

[0148] The system of this embodiment can be used to execute the method of any of the above embodiments. The implementation principles and technical effects are similar and will not be described in detail here.

Claims

1. A text detection method, characterized in that: include: Obtaining text usage requirements; obtaining rich text historical archives based on the text usage requirements; Parsing the rich text historical archive to determine structured multi-type rich text; Determining detection demand-oriented information according to the text usage requirements and the structured multi-type rich text; Determining whether there is multimedia fusion content in the detection demand-oriented information; If it exists, determining the detection result information set based on the multimedia fusion content; Output the detection result information set in structured text.

2. According to the method of claim 1, the step of determining the detection result information set based on the multimedia fusion content comprises: Dividing the multimedia fusion content into text data and image data; Parsing the text data to determine paragraph line information and semantic information of the text data; parsing the image data to determine image coordinates; Determine a target image and target image coordinates referring to the corresponding text according to the semantic information, the image coordinates and the paragraph line information; Determining, according to the target image and the semantic information, a valid text component represented by the target image; A detection result information set is determined according to the valid text components, the semantic information, the paragraph line information and the target image coordinates.

3. The method according to claim 2, characterized in that The determining, according to the semantic information, the image coordinates and the paragraph line information, a target image and target image coordinates referring to the corresponding text includes: Calculate the association between each text data and each image data based on the semantic information, the image coordinates and the paragraph line information, According to the association degree, the target image is determined, referring to the following formula: I sel =I j |I j ∈I,R ij >T; Among them, I sel is the target image; I j is the jth data point of the image data; I is the set of image data; R ij is the degree of relevance; According to the correlation degree, the target image coordinates are determined, referring to the following formula: C sel =C j * |j∈J,R ij >T; Among them, C sel is the target image coordinate; C j* for j * The image in the image data pointed to by the index; j * is the index of the target image corresponding to the text; j is the index of any image in the set of image data; R ij For the degree of relevance.

4. The method according to claim 3, characterized in that The calculating the association degree between each text data and each image data based on the semantic information, the image coordinates and the paragraph line information includes: Extracting a set of keywords from the set of text data; extracting a set of keywords from the set of image data; The intersection size of the keyword set in the text data set and the keyword set in the image data set is calculated to obtain a keyword matching score, referring to the following formula: K rat (K Ti ,K lj )=|K Ti ∩K Ij |; Among them, K rat is the keyword matching score; K Ti is the set of keywords in the text data set; K j is a set of keywords in a set of image data; Convert a collection of text data into a text semantic vector; Convert a collection of image data into an image semantic vector; Calculate the cosine similarity between the text semantic vector and the image semantic vector to obtain the semantic matching score, refer to the following formula; Where: V rat (V Ti ,V lj ) is the cosine similarity; dot(V Ti ,V Ij ) is the dot product formula of text semantic vector and image semantic vector; ||V Ti || is the length of the text semantic vector; ||V Ij || is the length of the image semantic vector; Calculating the proximity between the target image coordinates and paragraph line information corresponding to a keyword set in any text data to obtain a position relationship score; According to the keyword matching score, the semantic matching score and the position relationship score, the association degree between each text data and each image data is obtained, referring to the following formula: R ij =ε·K rat (K Ti ,K lj )+ζ·V rat (V Ti ,V lj )+η·S core ; Among them, R ij is the correlation degree; ε is the weight coefficient of the position relationship score; K rat (K Ti ,K lj ) is the keyword matching score; ζ is the weight coefficient of the semantic matching score; V rat (V Ti ,V lj ) is the semantic matching score; η is the weight coefficient of the position relationship score; S core Score the positional relationship.

5. The method according to claim 4, characterized in that The calculating the proximity between the target image coordinates and the paragraph line information corresponding to the keywords in the keyword set in any text data to obtain the position relationship score includes: Analyze the proximity between the target image coordinates and paragraph line information corresponding to a keyword in a keyword set in any text data, and obtain the difference between the row numbers where the text and the image are located, the difference between the column numbers where the text and the image are located, and the surrounding situation between the text and the image; According to the wrapping conditions of the image and the text, obtaining the wrapping distance of the text on the X-axis of the image and the wrapping distance of the text on the Y-axis of the image; The positional relationship score is calculated according to the difference in the row number, the difference in the column number, the surrounding distance on the X-axis, and the surrounding distance on the Y-axis, referring to the following formula: Among them, α is the weight coefficient of the absolute value of the difference between row numbers; D row is the absolute value of the difference in row numbers; β is the weight coefficient of the difference in column numbers; D col is the difference in column numbers; γ is the weight coefficient of the surrounding distance on the X-axis; is the wrapping distance on the X axis; δ is the weight coefficient of the wrapping distance on the Y axis; is the wrap-around distance on the Y axis.

6. The method according to claim 2, characterized in that The determining, based on the target image and the semantic information, the valid text component represented by the target image includes: Parse the target image and determine whether there is text content in the target image; If it exists, extract the image text content of the target image; interpret the image text content and the semantic information of the image screen in the context, and extract the effective text components in the image through the image text content, image interpretation information and the semantic information; If it does not exist, the image interpretation information is interpreted based on the semantic information in the context of the image and the image interpretation information is used as a valid text component.

7. The method according to claim 6, characterized in that The step of interpreting the semantic information based on the context of the image and taking the image interpretation information as a valid text component includes: Interpret the image based on the context to obtain interpretation information; Calculating the similarity between the interpretation information and the semantic information; If the similarity is greater than a preset threshold, the interpretation information is used as a valid text component.

8. The method according to claim 2, characterized in that: The step of determining the detection result information set according to the valid text component, the semantic information, the paragraph line information and the target image coordinates includes: Based on the effective text components, the distribution state of the effective text components is determined according to the target image coordinates, paragraph line information, and semantic information; For the effective text components and the semantic information, multimodal information fusion is performed according to the distribution status to obtain a detection result information set.

9. The method according to claim 8, characterized in that The structured text output of the detection result information set includes: According to the query information, the detection result information set is screened to obtain a screened information set; Obtaining a preset structured template; filling the structured template with information using the filtered information set to obtain a streamlined information set; The streamlined detection result information set is output as structured text.

10. A text detection system, characterized in that: include: A rich text history archive acquisition module is used to acquire text usage requirements; and acquire rich text history archives according to the text usage requirements; A structured multi-type rich text parsing module, used to parse the rich text historical archives and determine the structured multi-type rich text; A detection requirement-oriented information determination module, used to determine detection requirement-oriented information according to the text usage requirements and the structured multi-type rich text; A multimedia content determination module, used to determine whether there is multimedia fusion content in the detection demand-oriented information; A detection result information set construction module, for determining the detection result information set based on the multimedia fusion content, if any; The structured text output module is used to output the detection result information set in the form of structured text.

Images