Display content association method and electronic equipment

Abstract

The embodiment of the invention provides a display content association method and electronic equipment, and the association method comprises the steps: responding to a selection operation of target content in a current display file on a display interface, and determining a target type to which the target content belongs; the current display file comprises various types of display contents; respectively comparing the target content with the target type with a plurality of display contents of different types in the current display file to obtain a comparison result; the plurality of display contents are obtained by dividing different types of display objects in the current display file; based on the comparison result, determining the display content meeting the matching condition in the multiple pieces of display content as the associated content corresponding to the target content; the associated content comprises at least one type of display content; displaying the associated content to a target area on the current display file; different types of display contents in the associated contents correspond to different target areas.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of data processing, and relate to but are not limited to a display content association method and an electronic device. BACKGROUND

[0002] In modern office work and document processing, users often use office software such as Word, WPS, or PDF editor to read text and analyze content. The office documents are usually mixed text and graphics documents, and there is a logical association between the graphics, lists, and text. However, in the process of use, due to the scattered distribution of text and graphics, it is difficult for users to quickly establish the correspondence between the text and the graphics when reading the document. In order to understand the meaning of the article, it is often necessary to frequently page to find the corresponding relationship between the text and the graphics, which affects the efficiency of information acquisition.

[0003] Currently, related technologies often use left thumbnail, split screen display, or click jump to assist users in locating related content. However, the method provided by the related technology still needs to manually page or repeatedly search to confirm the association between different contents in the context when processing multiple types of content files, which is complex and inefficient. SUMMARY

[0004] To solve the problems in the related art, embodiments of the present application provide a display content association method and an electronic device.

[0005] In a first aspect, the present application provides a display content association method, which comprises:

[0006] In response to a selection operation of target content in a currently displayed file on a display interface, determining a target type to which the target content belongs; the currently displayed file contains multiple types of display content;

[0007] Comparing the target content with multiple display contents of different types in the currently displayed file to obtain a comparison result; the multiple display contents are obtained by dividing display objects of different types in the currently displayed file;

[0008] Based on the comparison result, determining, as associated content corresponding to the target content, display content in the multiple display contents that meets a matching condition; the associated content includes at least one type of display content;

[0009] Displaying the associated content to a target area on the currently displayed file; different types of display content in the associated content correspond to different target areas.

[0010] In some embodiments, the target content includes a plurality of target sub-contents, and different target sub-contents are of different types and / or are located at different positions in the current display file; the comparing the target content of the target type with the plurality of display contents of different types in the current display file respectively to obtain a comparison result includes: respectively performing feature extraction on the plurality of target sub-contents to obtain a plurality of sub-feature vectors; performing feature fusion on the plurality of sub-feature vectors to obtain a target feature vector corresponding to the target content; performing feature extraction on the plurality of display contents to obtain a plurality of display feature vectors; and respectively performing similarity comparison on the target feature vector and each display feature vector to obtain the comparison result.

[0011] In some embodiments, based on the comparison result, the display content satisfying a matching condition in the plurality of display contents is determined as the associated content corresponding to the target content, including: based on the comparison result, at least one target display content of each type with a similarity greater than a preset threshold is determined in the plurality of display contents; and based on the similarity corresponding to the at least one target display content of each type, data processing is performed on the at least one target display content of each type to obtain the associated content of different types corresponding to the target content.

[0012] In some embodiments, the data processing on the at least one target display content of each type to obtain the associated content of different types corresponding to the target content includes: based on the similarity, the at least one target display content of each type is sorted to obtain a display content sequence of each type; the display content sequence is spliced to obtain the associated content of different types corresponding to the target content; or the target display content with the highest similarity in the at least one target display content of each type is determined as the associated content of different types corresponding to the target content.

[0013] In some embodiments, after the associated content is displayed to the target area, the association method further includes: in response to a click operation on the target content, based on the similarity of the at least one target display content corresponding to each type, a target display content with a similarity only lower than the associated content is determined as replacement content; and the replacement content is displayed to the target area on the current display file as new associated content.

[0014] In some embodiments, the similarity comparison of the target feature vector and each display feature vector respectively to obtain the comparison result includes: performing feature alignment on the target feature vector and each display feature vector to obtain aligned target feature vectors and aligned display feature vectors with the same dimension; and calculating the cosine similarity between the target feature vector and each aligned display feature vector to obtain the comparison result.

[0015] In some embodiments, the association method further comprises: performing type identification on the content in the current display file to obtain display objects of different types; dividing the display objects of each type at different levels to obtain at least one sub-display object corresponding to each level; and determining the at least one sub-display object corresponding to each level of the display objects of different types as the plurality of display contents.

[0016] In some embodiments, the comparing the target content of the target type with the plurality of display contents of different types in the current display file respectively to obtain comparison results comprises: comparing the target content of the target type with the plurality of display contents of different types in the current display file respectively by using the trained first model to obtain comparison results.

[0017] In some embodiments, the first model comprises a feature extraction network corresponding to different types and a similarity comparison network; and the comparing the target content of the target type with the plurality of display contents of different types in the current display file respectively by using the trained first model to obtain comparison results comprises: performing feature extraction on the target content by using the feature extraction network corresponding to the target type to obtain a target feature vector; performing feature extraction on the plurality of display contents of different types respectively by using feature extraction networks corresponding to different types to obtain a plurality of display feature vectors; and performing similarity comparison between the target feature vector and each display feature vector by using the similarity comparison network to obtain the comparison results.

[0018] In a second aspect, the embodiments of the present application provide an association device of display content, which comprises: a first determination module configured to determine a target type to which a target content belongs in response to a selection operation of the target content in a current display file on a display interface; the current display file comprises display contents of multiple types; a comparison module configured to compare the target content of the target type with a plurality of display contents of different types in the current display file respectively to obtain comparison results; the plurality of display contents are obtained by dividing display objects of different types in the current display file; a second determination module configured to determine, based on the comparison results, display contents that meet matching conditions in the plurality of display contents as association contents corresponding to the target content; the association contents comprise display contents of at least one type; and a display module configured to display the association contents to a target region on the current display file; display contents of different types in the association contents correspond to different target regions.

[0019] In a third aspect, an electronic device is provided. The electronic device includes a memory configured to store a computer program executable on a processor, and the processor is configured to implement the following steps when the computer program is executed: in response to a selection operation of target content in a currently displayed file on a display interface, determining a target type to which the target content belongs; the currently displayed file includes display content of multiple types; comparing the target content of the target type with multiple display contents of different types in the currently displayed file respectively to obtain a comparison result; the multiple display contents are obtained by dividing display objects of different types in the currently displayed file; based on the comparison result, determining display content in the multiple display contents that satisfies a matching condition as associated content corresponding to the target content; the associated content includes display content of at least one type; and displaying the associated content to a target region on the currently displayed file; and display content of different types in the associated content corresponds to different target regions.

[0020] The above description is only a summary of the technical solutions of the present application. In order to enable the technical means of the present application to be more clearly understood, and to be implemented according to the content of the description, and in order to enable the above and other purposes, characteristics and advantages of the present application to be more apparent and easy to understand, the following specific embodiments of the present application are described. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 FIG. 1 is a structural schematic diagram of an electronic device provided by an embodiment of the present application;

[0022] Figure 2 FIG. 2 is a flowchart of an optional display content association method provided by an embodiment of the present application; Figure 1

[0023] Figure 3 FIG. 3 is a flowchart of an optional display content association method provided by an embodiment of the present application; Figure 2

[0024] Figure 4 FIG. 4 is a schematic diagram of a display interface provided by an embodiment of the present application; Figure 1

[0025] Figure 2 FIG. 5 is a schematic diagram of a display interface provided by an embodiment of the present application; Figure 6

[0026] Figure 1 FIG. 6 is a flowchart of a model training provided by an embodiment of the present application. DETAILED DESCRIPTION

[0027] ​​​​In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings, and the described embodiments should not be regarded as limitations to the present application. All other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0028] In the following description, "some embodiments" are related to a subset of all possible embodiments, but it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict. Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meaning as understood by those skilled in the art to which the embodiments of the present application belong. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0029] Currently, in existing office software (such as Word, WPS, PDF editor, etc.), when reading a document, a user often needs to frequently turn pages to find the corresponding relationship between images and texts. For example, a certain page in the document displays a framework diagram, and the specific analysis content of the diagram may be distributed in subsequent pages, resulting in that the user needs to constantly look back at the previous diagram when reading the analysis content, which seriously affects the reading efficiency and experience.

[0030] Although the related technologies try to solve this problem by means of thumbnails, quick previews, split screens or click positioning, these methods still cannot realize dynamic, intelligent and real-time association display between images and texts, and it is difficult to meet the needs of users to efficiently obtain information.

[0031] Based on the problems in the related technologies, the embodiments of the present application can provide an association method for displaying content. In response to a selection operation of target content in a currently displayed file on a display interface, a target type to which the target content belongs is determined. The currently displayed file contains multiple types of display content. The target content with the target type is compared with multiple display contents of different types in the currently displayed file respectively to obtain a comparison result. The multiple display contents are obtained by dividing display objects of different types in the currently displayed file. Based on the comparison result, display content that meets a matching condition is determined as association content corresponding to the target content in the multiple display contents. The association content includes at least one type of display content. The association content is displayed to a target area on the currently displayed file. Display contents of different types in the association content correspond to different target areas.

[0032] The embodiment of the present application determines the target content and the corresponding target type in response to the interaction of the user, and then compares the target content with display content of different types, so as to filter out associated content meeting the matching condition and display the associated content in the corresponding area. In this way, display content of various types highly related to the target content is filtered out according to the matching result, and the display content is displayed in the preset area respectively, realizing real-time and intelligent associated display of graphic and text content. Not only is the matching of cross-modal content in a file realized, but also the time cost required by the user in the reading process for frequently flipping pages to find the corresponding relationship between the graphic and the text is reduced, and the document reading efficiency in the office software and the overall use experience of the user are significantly improved.

[0033] Figure 1 is a structural schematic diagram of an electronic device provided by the embodiment of the present application, Figure 1 The electronic device shown in the figure includes at least one processor 110, a memory 150, at least one network interface 120 and a user interface 130. Various components in the electronic device are coupled together through a bus system 140. It can be understood that the bus system 140 is used to realize the connection and communication between the components. In addition to the data bus, the bus system 140 also includes a power bus, a control bus and a status signal bus. However, for the purpose of clear illustration, all kinds of buses are marked as the bus system 140 in the figure. Figure 2

[0034] The processor 110 can be an integrated circuit chip with signal processing capability, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., wherein the general-purpose processor can be a microprocessor or any conventional processor.

[0035] The user interface 130 includes one or more output devices 131 enabling the presentation of media content, and one or more input devices 132.

[0036] ​The memory 150 can be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, and the like. The memory 150 optionally includes one or more storage devices remotely located from the processor 110. The memory 150 comprises a volatile memory or non-volatile memory, and can also include both volatile and non-volatile memory. Non-volatile memory can be read only memory (ROM), volatile memory can be random access memory (RAM). The memory 150 described in embodiments of the present application is intended to include any suitable type of memory. In some embodiments, the memory 150 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or a subset or superset thereof, which are described below.

[0037] The operating system 151 includes a system program for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, and the like, for implementing various basic services and processing hardware-based tasks;

[0038] The network communication module 152 is used to reach other computing devices via one or more (wired or wireless) network interfaces 120, exemplary network interfaces 120 include Bluetooth, wireless compatibility certification (WiFi), and universal serial bus (USB), and the like;

[0039] The input processing module 153 is used to detect and translate one or more user inputs or interactions from one or more input devices 132.

[0040] In some embodiments, the device provided by the embodiments of the present application can be realized in software, Figure 2 A display content association device 154 stored in the memory 150 is shown, which can be a display content association device in an electronic device, which can be software in the form of programs and plug-ins, etc., including the following software modules: a first determination module 1541, a comparison module 1542, a second determination module 1543, and a display module 1544. These modules are logical, so they can be combined or further split according to the functions implemented. The functions of each module will be described below.

[0041] In some embodiments, the apparatus provided by the embodiments of the present application can be implemented in a hardware manner. For example, the apparatus provided by the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to perform the association method of the display content provided by the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can be implemented by using one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), or other electronic elements.

[0042] In the embodiments of the present application, the first model can also be stored in the memory 150, and the processor 110 can call the first model to compare the target content of the target type with the plurality of display contents of different types in the display file respectively, to obtain a comparison result.

[0043] The association method of the display content provided by the embodiments of the present application can be executed by an electronic device. The electronic device can be a terminal running office software (such as Word, WPS, and PDF editor), or can also be a server. That is, the association method of the display content provided by the embodiments of the present application can be executed by a terminal, can be executed by a server, or can also be executed by interaction between a terminal and a server.

[0044] The technical solutions of the present application will be described in detail below with reference to the accompanying drawings.

[0045] Figure 1 is an optional flowchart of the association method of the display content provided by the embodiments of the present application Figure 2 The execution subject of the association method of the display content provided by the embodiments of the present application is explained by taking a server as an example, as shown in Figure 3 The association method of the display content provided by the embodiments of the present application can be implemented by steps S201 to S204.

[0046] In step S201, in response to a selection operation of target content in a current display file on a display interface, a target type to which the target content belongs is determined. The current display file contains display contents of multiple types.

[0047] In the embodiments of the present application, the currently displayed file refers to a document being browsed by the user, which can be a part of a multi-page document or can only display single-page content. There can be multiple types of display content in the currently displayed file, such as text, pictures, tables, formulas, and the like.

[0048] The selection operation refers to the behavior of the user interacting with a certain content in the currently displayed file through mouse clicking, hovering, dragging, and the like. For example, when the user stops the cursor on a certain sub-content of a picture, the sub-content is considered as the target content. The target content can refer to a content unit selected or triggered by the user, such as a certain text in the currently displayed file or a part of a picture.

[0049] Here, the type of display content can refer to text, image, chart, formula, and the like, and the target type refers to the category to which the target content belongs. If the target content is a text, the target type is text; if the target content is a structural diagram, the target type is image.

[0050] In the embodiments of the present application, the interaction event of the user can be detected, and whether to start the type identification process of the target content can be determined according to the event type (such as clicking or hovering for more than 1 second). Once the target content is identified, the type of the target content can be determined. For example, when the user stops the cursor on a certain sub-module of a framework diagram for more than 1 second, the area is considered as the target content, and the type identification of the target content can be performed at this time to determine that the target type is image.

[0051] In step S202, the target content having the target type is compared with multiple display contents of different types in the currently displayed file respectively to obtain a comparison result, and the multiple display contents are obtained by dividing display objects of different types in the currently displayed file.

[0052] In the embodiments of the present application, the multiple display contents of different types in the currently displayed file can refer to all types of display contents in the currently displayed file, can be display contents of non-target types in the currently displayed file, or can be display contents of different types in the same chapter as the target content.

[0053] In some embodiments, the comparison can refer to comparing based on the similarity between the feature vectors to determine whether there is a semantic or visual association between the target content and each display content. Here, the feature vector can be extracted from the target content, and compared with the feature vector of each display content in the currently displayed file to obtain the matching degree therebetween.

[0054] Here, the similarity comparison can be to extract feature vectors of the target content and each display content using a deep learning model, and map these vectors into the same feature space for cross-modal similarity calculation. For example, if the target content is a certain sub-module in an image, the feature vector of the sub-module can be obtained by identification, and then cosine similarity calculation and comparison are performed with the feature vectors of all text contents in the current display file to obtain the comparison result.

[0055] Here, the display object refers to the basic unit constituting the content of the current display file, and different types of display objects can refer to all text, all pictures and all formulas in the current display file. The division of different types of display objects in the current display file can refer to the division of display objects at different levels, for example, all texts are divided by chapters once to obtain some display objects, divided by paragraphs once to obtain some display objects, divided by periods once to obtain some display objects, and divided by words once to obtain some display objects, and finally all display objects of the text type are obtained.

[0056] In step S203, based on the comparison result, the display content that meets the matching condition in the plurality of display contents is determined as the associated content corresponding to the target content; the associated content includes at least one type of display content.

[0057] In the embodiments of the present application, the matching condition can refer to a standard for judging whether two contents have sufficient strong association. Here, the matching condition can be set as: the cosine similarity between the feature vectors of the two contents is greater than a preset threshold, and is ranked highest in all candidate contents or meets the pre-set combination logic. Here, the similarity threshold can be 0.85. Only when the similarity between a certain text paragraph and the target image part exceeds the similarity threshold, it will be considered as the content associated with the target content.

[0058] Here, the associated content refers to at least one display content that is considered to have a strong association with the target content after screening. Since there can be multiple types of display contents in the current display file, the associated content can also include multiple types, for example, when the target content is a GAM sub-module (i.e. a small box in the figure, and the text content in the box is GAM) in the image, the associated content can be a detailed explanation of GAM in the following text, or another subgraph showing the details of GAM.

[0059] In some embodiments, the top few can be selected as the associated content according to the similarity ranking result, and it is ensured that these contents cover at least two different types. For example, the text description with the highest similarity and the subgraph with the highest similarity can be selected as the associated content.

[0060] In step S204, the associated content is displayed to a target region on the current display file; different types of display content in the associated content correspond to different target regions.

[0061] In some embodiments, the target region can refer to a region in the current display file interface that is preset for displaying the associated content, and is usually located in the left or right blank of the main display area. For example, when reading a PDF document, the left side can be used to display summary content associated with the text, and the right side can be used to display related charts or annotations.

[0062] Here, different types of display content corresponding to different target regions can refer to assigning different types of display content to different preset locations for display according to the type of associated content. For example, text-based associated content can be displayed in the left region, and image-based associated content can be displayed in the right region, which can avoid the mixing of different types of content and improve the user reading experience.

[0063] In some embodiments, the display can adopt a Hyper Text Markup Language (HTML, Hyper Text Markup Language Layout) layout, and different types of associated content are inserted into the left and right sides of the current interface, so that the user can complete the text-image comparison reading without leaving the current page. For example, when the user clicks on a substructure of a figure, the left side can display a context framework diagram of the structure, and the right side can display detailed analysis text.

[0064] In some embodiments, the layout of the target region can be dynamically adjusted according to the type of associated content. For example, when the associated content is mainly text, the left region can be expanded to accommodate more content; when the associated content contains a large number of images, the right region can preferentially display images and limit the length of text to keep the interface neat.

[0065] The embodiments of the present application determine the target content and the corresponding target type in response to user interaction, compare the target content with different types of display content, and thus filter out associated content that meets the matching conditions and display it in the target region. In this way, multiple types of display content that are highly related to the target content are selected according to the matching results and displayed in the preset regions, realizing real-time and intelligent association display of text-image content. Not only does this realize the matching of cross-modal content in the file, but it also reduces the time cost of users in the reading process when frequently flipping pages to find the corresponding relationship between text and images, significantly improving the document reading efficiency in office software and the overall user experience.

[0066] In some embodiments, the target content includes multiple target sub-contents, and the types and / or positions of different target sub-contents in the current display file are different.

[0067] Here, the target sub-content can refer to multiple independent parts that make up the target content, can be simultaneously selected by the mouse, the target sub-content can have different types (such as text, charts, formulas, etc.), and be distributed at different positions in the currently displayed file. For example, in a framework diagram, the GAM module, the attention mechanism module, the loss function module, etc. can be different target sub-contents. Each target sub-content carries specific information and has a clear position identifier in the current display file structure.

[0068] Figure 2 is an optional flowchart of a method for displaying content provided by an embodiment of the present application Figure 2 As shown in Figure 4 Step S202 in the embodiment of the present application can be implemented by steps S301 to S304:

[0069] Step S301, respectively extracting features of the multiple target sub-contents to obtain multiple sub-feature vectors.

[0070] In the embodiment of the present application, feature extraction is a process of extracting key information from original data to form a mathematical representation that can be used for subsequent processing. For text content, the embodiment of the present application can use a word embedding model (such as Word2Vec or Bidirectional Encoder Representations from Transformers (BERT)) to convert text content into a feature vector; for image content, a convolutional neural network (CNN) or a Transformer architecture (such as a visual transformer (ViT)) can be used to extract visual features, and finally obtain multiple sub-feature vectors corresponding to the target content.

[0071] In some embodiments, a dynamic convolution-based image encoder and a Transformer-based text encoder can be used to extract features of images and texts, respectively, based on a Swim Transformer as a basic framework.

[0072] Step S302, fusing the multiple sub-feature vectors to obtain a target feature vector corresponding to the target content.

[0073] In the embodiments of the present application, feature fusion is a process of integrating multiple sub-feature vectors into a comprehensive representation, aiming to capture the overall semantic information of the entire target content and enhance the overall expression capability of the content. Here, feature fusion can adopt a splicing manner, i.e., sequentially connecting each sub-feature vector to form a long feature vector; or a weighted fusion manner, i.e., summing up after giving different weights according to the importance of each sub-feature, the importance can be determined based on similarity, the higher the similarity, the higher the weight.

[0074] In some embodiments, splicing is suitable for the case where all sub-features are equally important, while weighted fusion can highlight the role of certain key sub-features. For example, in a framework diagram, if the importance of the GAM module and the attention mechanism module is different, the contribution difference of different sub-features can be reflected through weighted fusion.

[0075] Step S303, feature extraction is performed on the multiple display contents to obtain multiple display feature vectors.

[0076] Here, the types of the multiple display contents are the same or different, and a corresponding feature extraction method is adopted for display contents of different types to obtain corresponding display feature vectors, and finally multiple display feature vectors are obtained.

[0077] Step S304, similarity comparison is performed on the target feature vector and each display feature vector respectively to obtain the comparison result.

[0078] In the embodiments of the present application, similarity comparison can be calculated in a cosine similarity or inner product point multiplication manner, and the higher the value, the more similar the two contents. Similarity comparison between the target feature vector and each display feature vector can be mapping the feature vectors into the same feature space, calculating by a cosine similarity method, etc., for measuring the closeness between two feature vectors to judge their relevance.

[0079] The embodiments of the present application can find the most relevant display content of the target content according to the similarity ranking, and display it in a suitable position of the current interface, such as the left or right blank area.

[0080] In some embodiments, step S304 can be implemented through step S3041 and step S3042.

[0081] Step S3041, feature alignment is performed on the target feature vector and each display feature vector to obtain an aligned target feature vector and an aligned display feature vector with the same dimension.

[0082] In some embodiments, the feature alignment refers to converting the target feature vector and each display feature vector into a unified feature space through mapping, so that the target feature vector and the display feature vector have the same dimension for subsequent calculation. For example, linear transformation, attention mechanism or other mapping methods can be used to project these feature vectors into a shared high-dimensional vector space, so that the target feature vector and the display feature vector have consistent dimensions.

[0083] Through feature alignment, comparability between the target feature vector and the display feature vector can be ensured, thereby improving the accuracy of subsequent similarity calculation, avoiding information loss or misjudgment due to inconsistent dimensions, and improving the overall matching efficiency of the model.

[0084] Step S3042, calculate the cosine similarity between the target feature vector and each aligned display feature vector to obtain the comparison result.

[0085] In some embodiments, the cosine similarity can be used to measure the difference between two vectors, and the closer the numerical value is to 1, the more similar the two vectors are. Here, the dot product operation is performed on the aligned vectors to obtain the cosine similarity, which is used to evaluate the similarity between the target feature vector (such as the features of a certain image region) and each display feature vector (such as the features of multiple text descriptions).

[0086] Through cosine similarity calculation, the correlation between different display contents can be quantified, and the most relevant matching item can be quickly screened out, thereby realizing efficient image-text association positioning and significantly improving the search efficiency and understanding ability of users when reading complex documents.

[0087] The embodiments of the present application realize efficient and accurate content matching by comparing the similarity between the target content and the plurality of display contents, improve the reading efficiency and interaction experience of users in a complex current display file environment, reduce the time cost of users in the process of using office software to repeatedly page and find the required information, and significantly improve the document reading efficiency in office software and the overall use experience of users.

[0088] In some embodiments, step S203 in the embodiments of the present application can be implemented through step S2031 and step S2032:

[0089] Step S2031, based on the comparison result, determining at least one target display content in each type with a similarity greater than a preset threshold value from the plurality of display contents.

[0090] In some embodiments, the preset threshold value can be a preset numerical value, for example, 85%, for determining whether the target content matches each display content. When the similarity exceeds the preset threshold value, the display content is considered to be highly relevant to the target content, and the display content with a similarity greater than the preset threshold value is determined as the target display content.

[0091] Here, if there is no display content with a similarity greater than the preset threshold value in the multiple display contents, it indicates that the target content selected by the current user has no associated content in the document, and no display is performed; or, the display content with the highest similarity in the multiple display contents is displayed.

[0092] In step S2032, data processing is performed on the at least one target display content of each type based on the similarity corresponding to the at least one target display content of each type, to obtain associated content of different types corresponding to the target content.

[0093] In the embodiments of the present application, the data processing includes, but is not limited to, normalization, sorting, weighted average, clustering and other operations, to ensure that the associated content finally presented to the user is the most relevant and most valuable information. Meanwhile, this step can also be the integration of multiple types of display content, such as combination of text and images, to enhance the richness and hierarchy of the user browsing experience.

[0094] In some embodiments, the data processing can be sorting based on the similarity, splicing in order to obtain the associated content, or the display content with the highest similarity can be used as the associated content.

[0095] The embodiments of the present application can provide more comprehensive and diversified associated content display for users by comprehensively processing high-similarity display contents of different types, improve the understanding and reading efficiency of the users for the display content, realize more natural interaction and connection between display contents of different types, and significantly optimize the information acquisition efficiency of the users.

[0096] In some embodiments, step S2032 can include steps S1 to S3, wherein steps S1 and S2 are parallel to step S3:

[0097] In step S1, the at least one target display content of each type is sorted based on the similarity, to obtain a sequence of display contents of each type.

[0098] In some embodiments, after obtaining the similarity of each target display content, the at least one target display content can be sorted based on the similarity value, to obtain a sequence of display contents of each type.

[0099] Here, the sorting is to sort different types of target display content together, that is, without distinguishing types, to obtain a display content sequence; or to sort each type of target display content respectively, to obtain a display content sequence corresponding to each type respectively.

[0100] Step S2, splicing the display content sequence to obtain the associated content of different types corresponding to the target content.

[0101] In some embodiments, in the case of obtaining a display content sequence by sorting, at least target display content in the sequence can be spliced according to the order in the sequence to obtain the associated content to be displayed; in the case of obtaining a display content sequence corresponding to each type respectively by sorting, the display content sequence of each type is spliced respectively to obtain the associated content corresponding to each type respectively.

[0102] Step S3, determining the target display content with the highest similarity among the at least one target display content of each type as the associated content of different types corresponding to the target content.

[0103] In some embodiments, the target display content with the highest similarity among the at least one target display content of each type can also be determined as the associated content of the type, that is, only one associated content of each type is determined for display.

[0104] The embodiments of the present application can effectively establish the connection between different types of content by sorting and splicing the target display content according to the similarity, or selecting the display content with the highest similarity as the associated content, so as to quickly locate the target content into related information, and further to greatly improve the efficiency of users in reading and understanding the target content in the office software.

[0105] In some embodiments, the associated content displayed in the current interface can not be what the user wants, so a scheme of replacing the associated content is also provided to make the content displayed in the interface meet the user's intention. Therefore, after the associated content is displayed to the target area, the associated method provided by the embodiments of the present application can further include steps S10 and S11:

[0106] Step S10, in response to a click operation on the target content, determining the target display content with a similarity only lower than the associated content as replacement content based on the similarity of the at least one target display content corresponding to each type.

[0107] In some embodiments, the clicking operation refers to the behavior of the user selecting the target content in the interface through the mouse or touch screen, which can include single click, double click, long press, etc. for activating the interactive function related to the target content. Here, after the associated content corresponding to the target content is displayed, the clicking operation can represent that the user is not satisfied with the currently displayed content and needs to replace the currently displayed associated content.

[0108] After receiving the clicking operation of the user, the similarity of at least one target display content corresponding to each type is determined, and the target display content with a similarity lower than that of the associated content is determined as the replacement content.

[0109] Step S11: The replacement content is displayed as new associated content to the target area on the currently displayed file.

[0110] Here, the replacement content is used as new associated content to update the currently displayed information, and this process realizes the dynamic update of the associated content.

[0111] The embodiments of the present application understand the intention of the user through interaction with the user, replace the content on the currently displayed interface, so that the displayed associated content is what the user wants, and improve the reading efficiency and user experience.

[0112] In some embodiments, the display content can be obtained by hierarchically dividing the content in the document, and the method for associating the display content provided by the embodiments of the present application can further include steps S20 to S22:

[0113] Step S20: Type identification is performed on the content in the currently displayed file to obtain display objects of different types.

[0114] In some embodiments, the type identification of the content in the currently displayed file can refer to automatically judging the type of each content according to different forms (such as text, picture, table, formula, etc.) of the content in the currently displayed file to obtain display objects of different types. For example, the display objects of pictures, the display objects of tables, and the display objects of words can be identified.

[0115] Here, the type identification can be realized by combining natural language processing (NLP) technology and image recognition technology, which ensures that the contents of different types can be correctly classified and used as the basis for subsequent processing.

[0116] In some embodiments, when the user opens a PDF or Word currently displayed file, the entire page can be automatically scanned, and a model algorithm can be used to identify the types of various contents. For example, in an academic paper, content modules such as abstract, introduction, experimental part, table description, and reference can be distinguished.

[0117] Through the type identification operation, each type of content form in the current display file can be clearly distinguished, thereby providing a basis for subsequent hierarchical division.

[0118] In step S21, the display objects of each type are divided on different levels to obtain at least one sub-display object corresponding to each level.

[0119] In some embodiments, the content of each type can be divided according to a logical relationship and a hierarchical structure, and information can be structured. For example, a main graph can include multiple sub-modules. At this time, the main graph can be regarded as a first-level sub-display object, and each sub-module can be regarded as a second-level sub-display object. Further, if a sub-module further includes smaller components (such as nodes in a flowchart), the sub-display objects can be further divided into third-level sub-display objects, thereby obtaining at least one sub-display object corresponding to each level.

[0120] In some embodiments, the different levels of text in the document can be multiple levels such as chapters, paragraphs, sentences, and words.

[0121] In specific implementations, the hierarchical division can also be performed according to a reference relationship, a position relationship, and a nesting relationship between contents.

[0122] In step S22, the display objects of different types in at least one sub-display object corresponding to each level are determined as the multiple display contents.

[0123] In some embodiments, the display objects of different types in at least one sub-display object corresponding to each level are determined as the final multiple display contents. When a user reads an article in a current display file and a cursor moves to a target content, the target content and the multiple display contents are matched respectively, and corresponding associated contents are displayed. For example, a total framework diagram of the target content can be displayed on the left side, and the analysis text corresponding to the target content can be displayed on the right side. In this way, the user can obtain complete context information without page turning.

[0124] According to the embodiments of the present application, the content of the current display file is subjected to type identification and hierarchical division, the target content and the multiple display contents of different levels are matched, more accurate associated contents can be matched, and the use efficiency and user experience of the office software are significantly improved.

[0125] In the embodiments of the present application, the comparison between the target content of the target type and the multiple display contents of different types in the display file can be implemented by a pre-trained model. Therefore, the associated method of the display content provided by the embodiments of the present application can further include step S30.

[0126] Step S30, comparing the target content with the plurality of display contents of different types in the current display file respectively through the trained first model to obtain a comparison result.

[0127] In some embodiments, the first model can be a deep learning model trained on a multi-modal neural network, and can include a sub-model for feature extraction of display contents of different types. For example, the first model can include an image encoder and a text encoder, which can respectively extract features of images and texts, and map the extracted image features and text features to the same feature space, thereby realizing cross-modal semantic matching.

[0128] In some embodiments, the image encoder can be a sub-model based on a Swim Transformer architecture and fusing dynamic convolution, and has strong image feature extraction capability. For example, the image encoder can include a Swim Transformer encoding layer, a dynamic convolution layer and a prediction layer (for example, a multi-layer perception head, MLP Head). The dynamic convolution layer learns the complementary attention of the kernel along all four dimensions (spatial size, input channel number, output channel number and kernel number) of the kernel through an attention mechanism, to obtain stronger feature expression capability.

[0129] The text encoder can introduce a triple lightweight attention mechanism on a Transformer model, and use a three-branch structure to capture cross-dimensional interaction to strengthen the interaction capability of channels and spaces, and enhance the long-distance semantic interaction capability, so that the text feature expression is more accurate. For example, the text encoder can include a Transformer encoding layer and a triple lightweight attention mechanism layer.

[0130] In some embodiments, the text encoder can also be a recurrent neural network (RNN, Recurrent Neural Network), and the text encoder can also be a CNN. The neural network types used for display contents of different types in the embodiments of the present application are not limited.

[0131] In the embodiments of the present application, the first model further includes a similarity comparison network. After the target content with the target type and the plurality of display contents of different types in the current display file are respectively extracted by the sub-models corresponding to different types, the corresponding feature vectors are obtained, and the similarity comparison network can be used to compare the similarity between the feature vector corresponding to the target content and the feature vectors corresponding to the display contents, to obtain a comparison result.

[0132] In some embodiments, the plurality of display contents of different types include, but are not limited to, frame diagrams, flowcharts, tables, text paragraphs, etc., each of which has specific structure and semantic information. By inputting these display contents into the trained first model, the similarity between these display contents and the target content can be calculated to determine which contents are related to the target content.

[0133] Through the comparison of the target content and the plurality of display contents of different types in the display file by the trained first model, the application embodiments can quickly identify the graphic and text content related to the semantic of the target content, thereby avoiding frequent page flipping for searching, improving the reading efficiency of the user, realizing a more intelligent document browsing experience, and further improving the user experience and functional practicability of the office software.

[0134] In some embodiments, the first model includes a feature extraction network corresponding to different types and a similarity comparison network.

[0135] Here, the feature extraction network can refer to a neural network structure for feature extraction of display contents of different types (such as images and texts). For example, the image feature extraction network can be an image encoder (PTrans) based on the Swim Transformer architecture, and the text feature extraction network can be a text encoder (WTrans) based on the Transformer architecture.

[0136] The similarity comparison network can be a network for comparing the similarity between two feature vectors, which can calculate the matching degree between vectors in the form of cosine similarity or dot product. The similarity comparison network is responsible for determining whether two features belong to the same semantic category or have high correlation.

[0137] Step S30 can also be implemented by step S40 and step S42:

[0138] Step S40, performing feature extraction on the target content by the feature extraction network corresponding to the target type to obtain a target feature vector.

[0139] In some embodiments, when the target type is an image, the feature extraction network corresponding to the target type can be an image encoder; when the target type is a text, the feature extraction network corresponding to the target type can be a text encoder; and when the target type is a formula, the feature extraction network corresponding to the target type can be a formula encoder, thereby obtaining a target feature vector corresponding to the target content.

[0140] The target feature vector is a vector representation output after processing by the feature extraction network, representing the main semantic information of the target content, which is used for subsequent similarity calculation.

[0141] In step S41, different types of feature extraction networks are used to extract features of different types of display content, and a plurality of display feature vectors are obtained.

[0142] In some embodiments, the feature extraction network corresponding to each type of display content is used to extract features of the display content, and a corresponding display feature vector is obtained.

[0143] In step S42, the target feature vector and each display feature vector are compared by the similarity comparison network to obtain the comparison result.

[0144] In some embodiments, the similarity comparison network can compare the target feature vector with each display feature vector, output the similarity score between the target feature vector and each display feature vector, and obtain the comparison result. Finally, the scores can be sorted to determine which display content is most relevant to the target content.

[0145] In some embodiments, the similarity comparison network can be trained by contrast learning. The model learns to maximize the similarity between positive samples (matching image-text pairs) and minimize the similarity between negative samples (non-matching image-text pairs), so that the first model can better learn the correspondence between images and texts, and obtain the similarity comparison network in the first model.

[0146] In some embodiments, the first model can train the model by using a cross-entropy loss function, considering the bidirectional association between "image and text" and "text and image", and training the model by minimizing the loss function. During the training process, the first model can be evaluated regularly to monitor the performance and convergence of the first model. The accuracy, recall rate, F1 value (harmonic mean of precision and recall), and other indicators of the first model can be evaluated using a validation set or cross-validation method.

[0147] The embodiments of the present application avoid manual page turning to find display content, thereby improving the browsing efficiency of the user.

[0148] The feature extraction network and the similarity comparison network in the first model can efficiently extract the features of the target content and the display content, and establish the association between them through similarity calculation. In this way, the display content related to the target content can be quickly located, thereby reducing the user's manual page turning to find display content, improving the user's browsing efficiency, and significantly improving the efficiency of document reading and understanding.

[0149] Next, an application of a display content association method in an actual scenario is provided.

[0150] This application provides a text-based browsing method for quick retrieval. When the cursor is over a sub-content within an image in a document, the accompanying explanation and illustration of that sub-content will be displayed on the left and right sides of the current screen, respectively. Alternatively, when the cursor is over content, the overall framework diagram mentioned above will also appear on the left or right side of the current screen, facilitating reader comprehension and significantly increasing browsing efficiency. This application's design can be based on existing resources (Word and PDF documents have blank spaces on both sides), without increasing page complexity, and effectively utilizes the blank spaces.

[0151] Figure 1 This is a schematic diagram of the display interface provided in the embodiments of this application. Figure 4 ,like Figure 5 As shown, when the cursor is placed on the sub-content 401 (i.e. target content) of the image, the association method provided in this application embodiment can obtain the framework diagram 402 and detailed explanation 403 (i.e. associated content) associated with the sub-content 401. The framework diagram 402 can be displayed on the left and the detailed explanation 403 can be displayed on the right, which greatly avoids the reader having to flip through pages to find the associated content corresponding to the sub-content 401.

[0152] Figure 2 This is a schematic diagram of the display interface provided in the embodiments of this application. Figure 5 ,like Figure 6 As shown, when the cursor is over the text, such as when it is on "Attention Mechanism", the overall framework diagram 501 will appear on the left. This avoids the problem of having to flip up and down to find the corresponding text explanation when introducing the overall framework diagram below.

[0153] In this embodiment of the application, to obtain features of two different interaction methods—image and text, and text and image—a multimodal neural network TransPW (i.e., the first model) can be used to extract features from images and text. The TransPW model can consist of two parts: an image encoder and a text encoder. The image encoder is responsible for converting the image into a feature vector, named PTrans; the text encoder is responsible for converting the text into a feature vector, which can be a WTrans model. These two encoders can achieve cross-modal information interaction and fusion by sharing a vector space.

[0154] In some embodiments, the image encoder (PTrans) is responsible for converting images into feature vectors. It is based on the SwimTrans ansformer framework and incorporates dynamic convolutions, completely replacing conventional convolutions. Through the attention mechanism in the image encoder, complementary attention of the kernels is learned along all four dimensions of the convolution kernel (spatial size, number of input channels, number of output channels, and number of kernels) to obtain stronger feature representation capabilities.

[0155] The text encoder (WTrans model) utilizes the encoding capability of the Transformer model, but has weak long-distance interaction capability in text feature extraction. Therefore, a triple lightweight attention mechanism is introduced, and a three-branch structure of the mechanism is used to capture the cross-dimensional interaction method to strengthen the interaction capability of the channel and the space.

[0156] Figure 6 is a flowchart of model training provided by an embodiment of the present application, as shown in Figure 1 Text matching can be realized through steps S601 to S605.

[0157] S601, inputting an image and text into a model.

[0158] Here, the image and the text can be samples that have been labeled with similarity.

[0159] S602, outputting image feature vectors and text feature vectors with the same dimension.

[0160] The image encoder can include a Swim Transformer encoding layer, a dynamic convolution layer, and a prediction layer (for example, a multi-layer perception head, MLP Head). The text encoder can include a Transformer encoding layer and a triple lightweight attention mechanism layer.

[0161] The image feature vectors and the text feature vectors are outputted by the image encoder and the text encoder respectively.

[0162] S603, calculating the similarity between the image feature vectors and the text feature vectors.

[0163] The image feature vectors and the text feature vectors are mapped into the same high-dimensional feature space, so that the image feature vectors and the text feature vectors can be directly compared, and the similarity between the image feature vectors and the text feature vectors is further calculated.

[0164] In the feature space, the similarity (such as cosine similarity) between the text feature vectors and the image feature vectors is calculated to measure the similarity between them. The similarity can be measured by inner product (dot product), which reflects the cosine similarity between two vectors, i.e. the included angle between them.

[0165] Through contrastive learning, the model can maximize the similarity between positive samples (matched image-text pairs) and minimize the similarity between negative samples (unmatched image-text pairs), so that the model can better learn the correspondence between images and texts.

[0166] S604, calculating the cross-entropy loss.

[0167] Here, the cross-entropy loss function can be selected to measure the similarity between the image and the text, considering the bidirectional relationship from the image to the text and from the text to the image, to ensure that the model has bidirectional retrieval capability from image to text and from text to image.

[0168] S605, based on the cross-entropy loss iteration training, save the parameters.

[0169] In the embodiments of the present application, the model is trained by minimizing the loss function, and the minimum of the loss function can be a minimum value. For example, after training for 500 rounds, the value of the loss function does not change and the parameters are not optimized, and then the current parameters can be the optimal parameters of the model.

[0170] During the training process, the model performance is evaluated periodically using the validation set or cross-validation method, focusing on accuracy, recall rate, F1 value and other indicators, to monitor the convergence of the model and make necessary parameter adjustments.

[0171] In the embodiments of the present application, when the user uses the Word, WPS and PDF editor and other office software to read the framework diagram, when the mouse moves to the framework diagram, the analysis of this part of the architecture can be placed in the blank on both sides, or the large model can be used to analyze this part of the architecture and placed on both sides or the mouse after simple summary.

[0172] In the embodiments of the present application, the document can be calculated by the model before reading, so that the associated content can appear immediately when the mouse is placed at a certain position, or the calculation can be performed while reading.

[0173] The embodiments of the present application can quickly obtain the association between the text and the graph of the document by using the similarity between the graph and the text, greatly shorten the time of repeated searching of text and graph, effectively solve the problem of low reading efficiency caused by frequent jumping between text and graph, realize the intelligent association between text and graph, and greatly improve the readability and user experience of office documents.

[0174] Please continue to refer to ​The association device 154 for displaying content can include a first determination module 1541, a comparison module 1542, a second determination module 1543, and a display module 1544. The first determination module 1541 is configured to determine a target type of target content in response to a selection operation of the target content in a currently displayed file on a display interface. The currently displayed file includes multiple types of display content. The comparison module 1542 is configured to compare the target content of the target type with multiple display contents of different types in the currently displayed file respectively to obtain comparison results. The multiple display contents are obtained by dividing display objects of different types in the currently displayed file. The second determination module 1543 is configured to determine, based on the comparison results, display content that meets a matching condition from the multiple display contents as associated content corresponding to the target content. The associated content includes at least one type of display content. The display module 1544 is configured to display the associated content to a target region on the currently displayed file. Different types of display content in the associated content correspond to different target regions.

[0175] In some embodiments, the target content includes multiple target sub-contents, and different target sub-contents are of different types and / or are located at different positions in the currently displayed file. The comparison module 1542 is further configured to extract features of the multiple target sub-contents respectively to obtain multiple sub-feature vectors, perform feature fusion on the multiple sub-feature vectors to obtain a target feature vector corresponding to the target content, extract features of the multiple display contents to obtain multiple display feature vectors, and perform similarity comparison on the target feature vector and each display feature vector respectively to obtain the comparison results.

[0176] In some embodiments, the second determination module 1543 is further configured to determine, based on the comparison results, at least one target display content of each type with a similarity greater than a preset threshold from the multiple display contents. The second determination module 1543 is further configured to perform data processing on the at least one target display content of each type based on a similarity corresponding to the at least one target display content to obtain associated content of different types corresponding to the target content.

[0177] In some embodiments, the second determination module 1543 is further configured to sort the at least one target display content of each type based on the similarity to obtain a display content sequence of each type. The second determination module 1543 is further configured to splice the display content sequence to obtain associated content of different types corresponding to the target content. Alternatively, the second determination module 1543 is further configured to determine, as the associated content of different types corresponding to the target content, a target display content with the highest similarity from the at least one target display content of each type.

[0178] In some embodiments, after the associated content is displayed to the target area, the associated device further comprises a third determining module configured to, in response to a click operation on the target content, determine, based on similarity of at least one target display content corresponding to each type, a target display content with a similarity lower than the associated content as a replacement content; and display the replacement content as new associated content to the target area on the current display file.

[0179] In some embodiments, the second determining module 1543 is further configured to perform feature alignment on the target feature vector and each display feature vector to obtain an aligned target feature vector and an aligned display feature vector with the same dimension; and calculate a cosine similarity between the target feature vector and each aligned display feature vector to obtain the comparison result.

[0180] In some embodiments, the associated device further comprises a type identifying module configured to perform type identification on the content in the current display file to obtain display objects of different types; a dividing module configured to divide the display objects of different types at different levels to obtain at least one sub-display object corresponding to each level; and a fourth determining module configured to determine the at least one sub-display object corresponding to each level of the display objects of different types as the plurality of display contents.

[0181] In some embodiments, the comparison module 1542 is further configured to compare the target content with the plurality of display contents of different types in the current display file respectively by using the trained first model to obtain the comparison result.

[0182] In some embodiments, the first model comprises a feature extraction network corresponding to each type and a similarity comparison network; and the comparison module 1542 is further configured to perform feature extraction on the target content by using the feature extraction network corresponding to the target type to obtain a target feature vector; perform feature extraction on the plurality of display contents of different types respectively by using the feature extraction networks corresponding to different types to obtain a plurality of display feature vectors; and perform similarity comparison between the target feature vector and each display feature vector by using the similarity comparison network to obtain the comparison result.

[0183] The description of the associated device of display content in the embodiments of the present application is similar to the description of the associated method of display content, and has similar beneficial effects as the associated method of display content, and thus is not described herein. For technical details not disclosed in the device embodiments, please refer to the description of the associated method of display content for understanding.

[0184] The embodiment of the present application provides a computer program product or computer program, the computer program product or computer program comprises executable instructions, the executable instructions are computer instructions; the executable instructions are stored in a computer readable storage medium. When the processor of the electronic device reads the executable instructions from the computer readable storage medium, the processor executes the executable instructions, so that the electronic device executes the method provided by the embodiment of the present application.

[0185] The embodiment of the present application provides a storage medium storing executable instructions, wherein the executable instructions are stored, and when the executable instructions are executed by a processor, the processor executes the method provided by the embodiment of the present application.

[0186] In some embodiments, the storage medium can be a computer readable storage medium, for example, a ferroelectric memory (FRAM, Ferromagnetic Random Access Memory), a read-only memory (ROM, Read Only Memory), a programmable read-only memory (PROM, Programmable Read Only Memory), an erasable programmable read-only memory (EPROM, Erasable Programmable Read Only Memory), an electrically erasable programmable read-only memory (EEPROM, Electrically Erasable Programmable Read Only Memory), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM, Compact Disk-Read Only Memory) and the like memory; it can also be various devices comprising one or any combination of the above memories.

[0187] In some embodiments, the executable instructions can be in the form of programs, software, software modules, scripts or codes, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and can be deployed in any form, including being deployed as independent programs or being deployed as modules, components, subroutines or other units suitable for use in a computing environment.

[0188] By way of example, an executable instruction can be, but is not limited to, a file, a part of a file, containing high level code (e.g., a script) that can be executed by a machine with or without other code, a high level user interface element, an object, an object component (e.g., a control such as a button or input field of a user interface element), and / or a descriptor, an executable application program that can be run by a machine.

[0189] The storage medium and the computer program product in the embodiments of the present application are similar to the description of the method for associating display content, and have similar beneficial effects, and thus are not described herein. For technical details not disclosed in the embodiments, refer to the description of the method for associating display content.

[0190] The above merely describes the embodiments of the present application, but is not used to limit the protection scope of the present application. Any modification, equivalent replacement, and improvement made within the spirit and scope of the present application shall be included in the protection scope of the present application.

[0191] It should be understood that the "one embodiment" or "an embodiment" mentioned throughout the specification means that the specific features, structures, or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily mean the same embodiment. In addition, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that the size of the serial number of the above processes in various embodiments of the present application does not mean the execution order, and the execution order of the processes should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The serial number of the above embodiments of the present application is only for description, and does not represent the advantages and disadvantages of the embodiments.

[0192] It should be noted that, in the present document, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element preceded by "comprises... a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that comprises the recited element. In the several embodiments provided in the present document, it is to be understood that the disclosed devices and methods can be carried out by other ways of practicing that are either explicitly indicated, related to such implied, specific no matter how many substitutions, permuta tions, additions or modifications can be made thereto without departing from the scope and spirit of the disclosure. The embodiments described above are only illustrative, for example, the division of units is only a logical function division, and in actual implementation, other division manners can be used, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0193] The above description is merely illustrative of the application, and the scope of the application is not limited thereto. Any skilled in the art can easily make changes or replacements within the scope of the present application, and all changes or replacements should be covered within the scope of the present application. Therefore, the scope of the present application should be subject to the scope of protection of the claims.

Claims

1. A method for associating displayed content, the association method comprising: In response to the selection of target content in the currently displayed file on the display interface, determine the target type to which the target content belongs; The currently displayed file contains various types of display content; The target content with the aforementioned target type is compared with multiple display contents of different types in the currently displayed file to obtain comparison results; The multiple display contents are obtained by dividing the current display file into different types of display objects; Based on the comparison results, the display content that meets the matching conditions among the plurality of display contents is determined as the associated content corresponding to the target content; the associated content includes at least one type of display content. The associated content is displayed in the target area of ​​the currently displayed file; different types of the associated content correspond to different target areas.

2. The association method according to claim 1, wherein the target content includes multiple target sub-contents, and the types and / or positions of the different target sub-contents in the currently displayed file are different; The step of comparing target content of the target type with multiple display contents of different types in the currently displayed file to obtain comparison results includes: Feature extraction is performed on the multiple target sub-contents respectively to obtain multiple sub-feature vectors; The multiple sub-feature vectors are fused to obtain the target feature vector corresponding to the target content; Feature extraction is performed on the multiple displayed contents to obtain multiple display feature vectors; The similarity between the target feature vector and each displayed feature vector is compared to obtain the comparison result.

3. The association method according to claim 2, wherein determining the display content that meets the matching conditions among the plurality of display contents as the associated content corresponding to the target content based on the comparison result includes: Based on the comparison results, among the multiple display contents, at least one target display content with a similarity greater than a preset threshold is determined in each type; Based on the similarity of at least one target display content in each type, data processing is performed on at least one target display content in each type to obtain related content of different types corresponding to the target content.

4. The association method according to claim 3, wherein the step of performing data processing on at least one target display content of each type to obtain associated content of different types corresponding to the target content includes: Based on similarity, at least one target display content of each type is sorted to obtain a display content sequence of each type; By concatenating the displayed content sequence, different types of associated content corresponding to the target content can be obtained; or, The target display content with the highest similarity among at least one target display content of each type is determined as the associated content of the different types corresponding to the target content.

5. The association method according to claim 4, wherein after the associated content is displayed in the target area, the association method further includes: In response to a click operation on the target content, based on the similarity of at least one target display content corresponding to each type, the target display content with a similarity only lower than the associated content is determined as the replacement content; The replaced content is displayed as new associated content in the target area of ​​the currently displayed file.

6. The association method according to claim 2, wherein the step of performing similarity comparison between the target feature vector and each displayed feature vector to obtain the comparison result includes: The target feature vector and each display feature vector are aligned to obtain aligned target feature vectors and aligned display feature vectors with the same dimension; The cosine similarity between the target feature vector and each aligned display feature vector is calculated to obtain the comparison result.

7. The association method according to any one of claims 1 to 6, further comprising: The content of the currently displayed file is type-identified to obtain display objects of different types; The display objects of each type are divided into different levels to obtain at least one sub-display object corresponding to each level; At least one sub-display object corresponding to each level of different types of display objects is determined as the plurality of display contents.

8. The association method according to any one of claims 1 to 6, wherein comparing the target content having the target type with multiple display contents of different types in the currently displayed file to obtain a comparison result includes: The trained first model compares the target content of the target type with multiple display contents of different types in the current display file to obtain the comparison results.

9. The association method according to claim 8, wherein the first model includes feature extraction networks and similarity comparison networks corresponding to different types; the step of comparing target content of the target type with multiple display contents of different types in the currently displayed file using the trained first model to obtain comparison results includes: The target content is feature extracted using the feature extraction network corresponding to the target type to obtain the target feature vector; By using different types of feature extraction networks, features are extracted from multiple display contents of different types, resulting in multiple display feature vectors; The similarity comparison network is used to compare the target feature vector with each displayed feature vector to obtain the comparison result.

10. An electronic device, the electronic device comprising: Memory, configured to store computer programs that can run on a processor; When a processor is configured to execute the computer program, it performs the following steps: in response to a selection operation of target content in a currently displayed file on a display interface, it determines the target type to which the target content belongs; the currently displayed file contains multiple types of display content; it compares the target content having the target type with multiple display contents of different types in the currently displayed file to obtain a comparison result; the multiple display contents are obtained by dividing the display objects of different types in the currently displayed file; based on the comparison result, it determines the display content that meets the matching conditions among the multiple display contents as the associated content corresponding to the target content; the associated content includes at least one type of display content; it displays the associated content in a target area on the currently displayed file; the different types of display content in the associated content correspond to different target areas.

Images