Long text vectorization method, device and equipment and readable storage medium

Abstract

The embodiment of the invention discloses a long text vectorization method and device, equipment and a readable storage medium. The method comprises the following steps: acquiring a target problem and a plurality of candidate webpages retrieved based on the target problem, and calculating a correlation score between a webpage text of each candidate webpage and the target problem; arranging the candidate webpages from high to low based on the correlation scores, and rejecting the candidate webpages with the correlation scores smaller than a preset score threshold to obtain a candidate webpage set; calculating the standard time for completing vectorization of each candidate webpage in the candidate webpage set according to a preset vectorization time consumption formula; according to a preset equivalent time formula, calculating an equivalent time length for completing vectorization of each candidate webpage in the candidate webpage set; and executing a task of vectoring the candidate webpages in the candidate webpage set according to the sorting sequence within the maximum processing time length, and stopping the task until the maximum time length. According to the method, efficient utilization of computing load resources is achieved, and the processing speed of key tasks is increased.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of artificial intelligence technology, and in particular to a long text vectorization method, apparatus, device and readable storage medium. Background Technology

[0002] In large-scale natural language processing tasks, text embedding is a crucial step in achieving semantic retrieval, knowledge-based question answering, and intelligent recommendation. With the widespread application of deep learning models and large language models, systems often need to process massive amounts of text data in batches. However, the computation time required for text embedding varies significantly depending on its length, especially for long texts, where model inference time is significantly longer than for short texts. If traditional sequential processing or evenly dividing tasks by quantity is still used, it easily leads to uneven execution time within a single batch, resulting in decreased overall processing efficiency and wasted computing resources.

[0003] Current batch text processing solutions typically group texts based on quantity or length to achieve time balance between batches. However, since the relationship between model inference time and text length is not entirely linear, simple length division cannot accurately predict actual processing time. With the continuous expansion of natural language processing tasks and the increasing demand for real-time computing, traditional static grouping methods are no longer sufficient to meet the requirements of efficient computation. Especially in the scenario of batch long text vectorization, how to simultaneously consider the reasonable estimation of processing time and the dynamic trade-off between task relevance has become a key issue affecting system performance. Therefore, a vectorization method is urgently needed to solve the above-mentioned technical problems. Summary of the Invention

[0004] The embodiments of the present invention provide a method, apparatus, device and readable storage medium for vectorizing long text, aiming to solve the technical problem of poor efficiency in the vectorization processing of long text by traditional technologies.

[0005] In a first aspect, embodiments of the present invention provide a long text vectorization method, which includes:

[0006] Obtain the target question and several candidate web pages retrieved based on the target question, and calculate the relevance score between the web page text of each candidate web page and the target question;

[0007] Based on the relevance score, the candidate web pages are arranged from high to low, and candidate web pages with relevance scores less than a preset score threshold are removed to obtain a set of candidate web pages.

[0008] The standard time for vectorization of the web text corresponding to each candidate webpage in the candidate webpage set is calculated based on the preset vectorization time formula and the text length of the web text corresponding to the candidate webpage.

[0009] Based on the preset equivalent time formula and the relevance score of the webpage text corresponding to each candidate webpage in the candidate webpage set to the target question, the standard time corresponding to the candidate webpages in the candidate webpage set is weighted according to the sorting order to obtain the equivalent time for the vectorization of the webpage text of each candidate webpage in the candidate webpage set.

[0010] The task of vectorizing candidate web pages in the candidate web page set is executed in sorted order within a predetermined maximum processing time, and the task stops when the vectorization processing time reaches the maximum processing time.

[0011] Secondly, embodiments of the present invention provide a long text vectorization apparatus, comprising:

[0012] The acquisition module is used to acquire a target question and several candidate web pages retrieved based on the target question, and to calculate the relevance score between the web page text of each candidate web page and the target question.

[0013] The elimination module is used to sort the candidate web pages from high to low based on the relevance score, and eliminate candidate web pages whose relevance scores are less than a preset score threshold to obtain a set of candidate web pages.

[0014] The standard duration calculation module is used to calculate the standard duration for vectorizing the web text corresponding to each candidate webpage in the candidate webpage set according to the preset vectorization time consumption formula and the text length of the web text corresponding to the candidate webpage, so as to obtain the standard time for vectorizing the web text of each candidate webpage in the candidate webpage set.

[0015] The weighted calculation module is used to perform weighted calculation on the standard time corresponding to the candidate web pages in the candidate web page set according to the preset equivalent time formula and the relevance score of the web page text corresponding to each candidate web page in the candidate web page set to the target question, in order of sorting, so as to obtain the equivalent time for the vectorization of the web page text of each candidate web page in the candidate web page set.

[0016] The vectorization module is used to perform vectorization processing on candidate web pages in the candidate web page set in sorted order within a predetermined maximum processing time, until the vectorization processing time reaches the maximum processing time and the task is stopped.

[0017] Thirdly, embodiments of the present invention provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the long text vectorization method described in the first aspect.

[0018] Fourthly, embodiments of the present invention provide a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, which, when executed by a processor, causes the processor to perform the long text vectorization method described in the first aspect.

[0019] This invention provides a method, apparatus, device, and readable storage medium for long text vectorization. The method acquires a target question and several candidate web pages retrieved based on the target question, and calculates the relevance score between the web page text and the target question for each candidate web page. Based on the relevance score, the candidate web pages are sorted from highest to lowest, and candidate web pages with relevance scores below a preset threshold are removed, resulting in a candidate web page set. A standard time for vectorizing the web page text corresponding to each candidate web page in the candidate web page set is calculated according to a preset vectorization time formula, yielding the standard time for vectorizing the web page text of each candidate web page in the candidate web page set. Based on a preset equivalent time formula and the relevance score between the web page text corresponding to each candidate web page in the candidate web page set and the target question, the standard time corresponding to each candidate web page in the candidate web page set is weighted according to the sorting order, yielding the equivalent time for vectorizing the web page text of each candidate web page in the candidate web page set. The task of vectorizing the candidate web pages in the candidate web page set is executed in sorting order within a predetermined maximum processing time until the vectorization processing time reaches the maximum processing time, at which point the task stops. This method ensures that when processing a number of candidate web pages in batches, it prioritizes vectorizing candidate web pages with high relevance scores within a preset maximum processing time, thereby achieving a balance of computational load and efficient utilization of resources, and improving the processing speed of critical tasks. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating an embodiment of the long text vectorization method provided by the present invention.

[0022] Figure 2 This is a schematic block diagram of a long text vectorization device provided in an embodiment of the present invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0025] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0026] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0027] Please see Figure 1 This is a flowchart illustrating a long text vectorization method provided in an embodiment of the present invention, which includes steps S110 to S150.

[0028] Step S110: Obtain the target question and several candidate web pages retrieved based on the target question, and calculate the relevance score between the web page text of each candidate web page and the target question;

[0029] In this embodiment, to obtain the target question, a user input interface is used. This interface can be a text input box in various intelligent applications, such as the search box of an intelligent search engine or the dialogue input area of ​​an intelligent customer service system. The user inputs the question into these interfaces, and this text information is captured in real time to obtain the target question. Furthermore, a web crawler is deployed to access major search websites and crawl the HTML code of web pages according to preset rules and strategies. To ensure the comprehensiveness and effectiveness of the information obtained, reasonable crawling depth and breadth can be set. For example, for well-known news websites and academic database websites, the frequency and depth of crawling access can be increased to obtain more valuable content. At the same time, to avoid putting excessive load on the target website, the web crawler is configured to follow the website's robots.txt protocol, and the crawler's access time and request frequency are reasonably arranged.

[0030] In one embodiment, after obtaining candidate web pages, the web page text of the candidate web pages is extracted using HTML parsing technology, which is implemented using Python libraries such as BeautifulSoup and Scrapy. These libraries can parse complex HTML structures into easily processed text forms, removing redundant content such as HTML tags, script code, and style information, retaining only pure text information, thus reducing the burden on subsequent processing.

[0031] In one embodiment, calculating relevance scores is one of the core technologies in the information filtering and preparation process. The specific process includes: extracting the webpage title and webpage summary for each candidate webpage to obtain the webpage description text for each candidate webpage; inputting the webpage description text and the target question text into a pre-defined vectorization model for vectorization processing to obtain multiple webpage description vectors and target question vectors; calculating the cosine similarity between each webpage description vector and the target question vector to obtain the relevance score between each candidate webpage and the target question. The vectorization model selected is Bge-M3.

[0032] Step S120: Arrange the candidate web pages from high to low based on the relevance scores, and remove candidate web pages with relevance scores less than a preset score threshold to obtain a set of candidate web pages;

[0033] In this embodiment, all candidate web pages are sorted from highest to lowest relevance score. Setting a preset relevance score threshold is a key decision point for selecting candidate web pages. The setting of this threshold requires comprehensive consideration of multiple factors. Firstly, the scale and quality of the data must be considered. If the number of candidate web pages is large and the overall relevance levels are uneven, the preset score threshold needs to be appropriately increased to effectively reduce the amount of data processed subsequently. For example, when processing a search request for a popular topic, millions of candidate web pages may be obtained. In this case, setting a higher preset score threshold, such as 0.8 (assuming a relevance score range of 0-1), can quickly filter out most low-relevance web pages, retaining only the top few thousand highly relevant web pages for subsequent processing. Secondly, the threshold must be determined in conjunction with the specific application scenario and requirements. In financial risk assessment scenarios where information accuracy is extremely important, the threshold needs to be set more strictly to ensure that the selected web page text can provide a reliable basis for risk assessment. In market research scenarios where the breadth of information is important, the threshold may be appropriately lowered to obtain more relevant information from different perspectives.

[0034] Furthermore, eliminating candidate web pages with relevance scores below a preset threshold has several significant implications for subsequent vectorization processing. From a computational resource perspective, these low-relevance web page texts would consume substantial computation time and memory resources if they entered the vectorization process, while their contribution to solving the target problem would be negligible. By eliminating them in advance, we can avoid performing a large number of computational operations on invalid data, thus concentrating limited computational resources on more valuable, highly relevant web page texts. For example, assuming that when processing a dataset containing 100,000 candidate web pages, setting an appropriate preset score threshold can eliminate 80% of the low-relevance web pages, then the computational load in subsequent vectorization processing will be directly reduced by 80%, significantly improving processing efficiency. In addition, from the perspective of the quality of the processing results, eliminating low-relevance web pages can reduce the interference of noisy data on the vectorization results, allowing the final vectorized representation to more accurately reflect the semantic relationship between the target problem and relevant web page texts, providing a more reliable data foundation for subsequent analysis and applications.

[0035] Step S130: Calculate the standard time for vectorizing the web text corresponding to each candidate webpage in the candidate webpage set according to the preset vectorization time formula and the text length of the web text corresponding to the candidate webpage, and obtain the standard time for vectorizing the web text of each candidate webpage in the candidate webpage set.

[0036] In this embodiment, the core of standard time calculation is based on the linear fitting relationship between text length and processing time. A preset fixed-length time coefficient is used to quickly estimate the estimated processing time (i.e., standard time) for webpage text vectorization. The standard time T for completing webpage text vectorization is calculated as follows: proc :

[0037] T proc =K l ×L;

[0038] Among them, K l The time coefficient is defined as K; L represents the text length of the candidate webpage. In this calculation logic, the text length of the webpage is the only core variable. The longer the text, the more characters it contains, and the more basic units the model needs to process during vectorization, naturally increasing the processing time. Practice shows a significant linear correlation between text length and vectorization processing time; therefore, text length can be used to quickly establish a basis for time estimation. Using the text length L as a metric for text length, it directly reflects the information capacity of the text and is a key input parameter for calculating the standard processing time. Fixed-length time coefficient K l This is a key parameter obtained through fitting a large amount of experimental data. It represents the average time required to vectorize text of a unit character length. This coefficient is dynamically adjusted according to the type of vectorization model used. For example, in lightweight word embedding models and complex pre-trained language models, the inference complexity of the models differs, resulting in different values ​​for K. l The value of K will vary. Through experiments, the correlation between text length and actual time consumption under different models was statistically analyzed, and the optimal K was determined using methods such as linear regression. l This value ensures the accuracy of standard duration calculations. The formula uses simple linear multiplication to quickly obtain the estimated vectorized time for each webpage text, avoiding the time overhead of complex calculations while guaranteeing the reliability of the prediction results.

[0039] In one embodiment, for example, if the fixed-length time coefficient K of a vectorized model l =0.002 seconds / character, and the character length L of a webpage text is 3000 characters, then calculate its standard duration T according to the formula. proc =0.002 × 3000 = 6 seconds. This method enables the rapid generation of accurate standard duration estimates for each candidate webpage text, providing a solid data foundation for subsequent equivalent duration calculations and dynamic binning. The standard duration calculation method based on linear fitting of text length ensures both computational efficiency and meets the time estimation requirements in batch processing scenarios. Compared to traditional multi-factor complex calculation models, it reduces computational costs while achieving K-values ​​through experimental fitting. lThe value ensures the accuracy of the prediction, and can well balance efficiency and accuracy, making it suitable for time optimization scenarios of large-scale long text vectorization.

[0040] Step S140: Based on the preset equivalent time formula and the relevance score of the webpage text corresponding to each candidate webpage in the candidate webpage set to the target question, the standard time corresponding to the candidate webpages in the candidate webpage set is weighted according to the sorting order to obtain the equivalent time for the vectorization of the webpage text of each candidate webpage in the candidate webpage set.

[0041] Step S150: Execute the task of vectorizing the candidate web pages in the candidate web page set in sorted order within the predetermined maximum processing time, until the vectorization processing time reaches the maximum processing time and then stop the task.

[0042] In this embodiment, the core design of the equivalent time formula is to balance text processing time and task priority. It adjusts the standard time by incorporating the relevance score between candidate web pages and the target question, generating an equivalent time that reflects the comprehensive indicator of "computation cost - relevance." The core logic is to correct the standard time using the reciprocal of the relevance score, giving highly relevant candidate web pages a priority advantage during batch text vectorization. In batch text vectorization scenarios, the relevance of different web pages to the target question varies significantly. If processing resources are allocated solely based on the standard time, highly relevant web pages may be delayed due to longer processing times, affecting the system's response speed to core requirements. The equivalent time formula incorporates the relevance score into the calculation, making the equivalent time smaller for web pages with higher relevance, thus making them more likely to be selected first and ensuring that high-value information is processed first. This design considers both actual computation time and the business priority of the task, achieving a balance between technical efficiency and business value. The equivalent time formula is as follows:

[0043]

[0044] Among them, T equiv Indicates equivalent time; S represents the relevance score of the candidate webpage text to the target question (range 0-1); K tThis represents a time-weighted coefficient used to adjust the overall scaling ratio of the equivalent time according to the actual business scenario. The optimal value is usually determined through experimental verification to ensure that the equivalent time accurately reflects the priority ranking requirements of tasks. The core significance of equivalent time calculation lies in constructing a "time consumption-relevance" balance mechanism, breaking away from the traditional single-dimensional model of sorting by time consumption or quantity, and achieving intelligent task scheduling through comprehensive indicators. In scenarios such as information retrieval and intelligent question answering, it ensures that the system prioritizes processing highly relevant web pages that are most valuable to user needs within a limited time budget. This improves the response speed of core tasks while ensuring the business value density of vectorized results, effectively enhancing the overall system performance and user experience.

[0045] For example, suppose the time weighting coefficient K t =1, there are three candidate web pages: standard duration of web page A Seconds, relevance score SA = 0.8; standard duration of webpage B. Seconds, relevance score SB = 0.4; standard duration of webpage C. The time taken is measured in seconds, and the relevance score (SC) is 0.9. Based on the formula, the equivalent duration of webpage A is calculated. seconds; the equivalent duration of webpage B seconds; the equivalent duration of webpage C Seconds. Although webpage C has the longest standard duration, its equivalent duration is relatively short due to its highest relevance. Therefore, it will be prioritized over webpages A and B in binning, thus achieving priority processing of highly relevant webpages.

[0046] Furthermore, executing the vectorization task within the predetermined maximum processing time is a crucial step in the entire time optimization solution. This involves multiple important mechanisms, including task scheduling, progress monitoring, and interrupt handling, to ensure the vectorization task is completed as efficiently as possible within the limited time. Based on the weighted calculation of equivalent duration, web page texts with longer equivalent durations (i.e., higher relevance) are prioritized for vectorization. For example, in a task with a maximum processing time of 60 seconds and multiple candidate web pages, the web page texts with the highest equivalent durations are placed into the processing queue sequentially. Simultaneously, to fully utilize computing resources, we can employ multi-threaded or distributed computing to process multiple web page texts in parallel. Assuming the computing environment supports four threads for parallel processing, four web page texts can be retrieved from the processing queue each time and assigned to the four threads for simultaneous vectorization, significantly accelerating the processing speed.

[0047] Furthermore, a progress monitoring mechanism is a crucial means of understanding task execution. By recording in real time the number of web page texts processed, the processing time consumed, and the processing progress of each web page text, the task's execution status can be intuitively viewed through a visual interface or log records. For example, in a visual task monitoring interface, a progress bar can represent the overall task completion progress, and different colored markers can distinguish the processing status of different web page texts (such as processing, completed, awaiting processing, etc.). Simultaneously, a timed reporting mechanism can be set up to generate a task progress report at regular intervals (e.g., every 5 seconds), detailing relevant information about the processed web page texts, including their standard duration, equivalent duration, and actual processing time, to promptly identify potential problems during task execution.

[0048] When the vectorization processing time is about to reach its maximum, the interruption mechanism kicks in. At this point, the currently ongoing vectorization task is immediately stopped, and the processed results are organized and saved. For unprocessed webpage text, different strategies can be adopted depending on the situation. If time permits, the equivalent processing time of the remaining webpage text can be recalculated and sorted, selecting those with relatively high relevance and that can be processed within the remaining time for further processing. If the remaining time is very limited, making it impossible to complete the vectorization of any complete webpage text, the relevant information of these unprocessed webpage texts can be recorded for reprocessing later when more time is available. For example, in a task with a maximum processing time of 30 seconds, if at the 25th second it is found that the current progress cannot complete the vectorization of all webpage texts, we can evaluate the remaining 5 seconds. If we find a webpage text with a shorter equivalent processing time and high relevance that could potentially be processed within 5 seconds, we can prioritize processing this webpage text; otherwise, we record the information of the remaining webpage texts and wait for a later time when more time is available. Through such task scheduling, progress monitoring, and interruption handling mechanisms, the processing strategy can be dynamically adjusted based on the processed time and remaining tasks within the predetermined maximum processing time, making full use of processing time and achieving high efficiency and flexibility in long text vectorization processing.

[0049] This method acquires the target question and several candidate web pages retrieved based on the target question, and calculates the relevance score between the web page text and the target question for each candidate web page. Based on the relevance score, the candidate web pages are sorted from highest to lowest, and candidate web pages with relevance scores below a preset threshold are removed, resulting in a candidate web page set. The standard time for vectorizing the web page text of each candidate web page in the candidate web page set is calculated according to a preset vectorization time formula, yielding the standard time for vectorization of the web page text of each candidate web page in the candidate web page set. Based on a preset equivalent time formula and the relevance score between the web page text and the target question for each candidate web page in the candidate web page set, the standard times for each candidate web page in the candidate web page set are weighted according to the sorting order, yielding the equivalent time for vectorization of the web page text of each candidate web page in the candidate web page set. The task of vectorizing the candidate web pages in the candidate web page set is executed in sorting order within a predetermined maximum processing time until the vectorization processing time reaches the maximum processing time, at which point the task stops. This method ensures that when processing a number of candidate web pages in batches, it prioritizes vectorizing candidate web pages with high relevance scores within a preset maximum processing time, thereby achieving a balance of computational load and efficient utilization of resources, and improving the processing speed of critical tasks.

[0050] This invention also provides a long text vectorization apparatus, which is used to perform any of the aforementioned long text vectorization methods. Specifically, please refer to... Figure 2 , Figure 2 This is a schematic block diagram of a long text vectorization device provided in an embodiment of the present invention. The long text vectorization device 100 can be configured in a server.

[0051] like Figure 2 As shown, the long text vectorization device 100 includes an acquisition module 110, a rejection module 120, a standard duration calculation module 130, a weighted calculation module 140, and a vectorization module 150.

[0052] The acquisition module 110 is used to acquire a target question and a number of candidate web pages retrieved based on the target question, and to calculate the relevance score between the web page text of each candidate web page and the target question.

[0053] The elimination module 120 is used to sort the candidate web pages from high to low based on the relevance score, and eliminate candidate web pages whose relevance scores are less than a preset score threshold to obtain a candidate web page set.

[0054] The standard duration calculation module 130 is used to calculate the standard duration for vectorizing the web text corresponding to each candidate webpage in the candidate webpage set according to the preset vectorization time consumption formula and the text length of the web text corresponding to the candidate webpage, so as to obtain the standard time for vectorizing the web text of each candidate webpage in the candidate webpage set.

[0055] The weighted calculation module 140 is used to perform weighted calculation on the standard time corresponding to the candidate web pages in the candidate web page set according to the preset equivalent time formula and the relevance score of the web page text corresponding to each candidate web page in the candidate web page set to the target question, in order of sorting, so as to obtain the equivalent time for the web page text of each candidate web page in the candidate web page set to be vectorized.

[0056] The vectorization module 150 is used to perform vectorization processing on candidate web pages in the candidate web page set in sorted order within a predetermined maximum processing time, until the vectorization processing time reaches the maximum processing time and then the task is stopped.

[0057] In one embodiment, the acquisition module 110 includes:

[0058] The vectorization processing unit is used to input the webpage text of each candidate webpage and the question text of the target question into a preset vectorization model for vectorization processing to obtain multiple webpage text vectors and target question vectors.

[0059] The relevance calculation unit is used to calculate the cosine similarity between each webpage text vector and the target question vector, and obtain the relevance score between each webpage text and the target question.

[0060] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the long text vectorization method described above.

[0061] In another embodiment of the invention, a computer-readable storage medium is provided. This computer-readable storage medium may be a non-volatile computer-readable storage medium. The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the long text vectorization method as described above.

[0062] Those skilled in the art will readily understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention.

[0063] In the embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Units with the same function may be grouped into one unit. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or units, or it may be an electrical, mechanical, or other form of connection.

[0064] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of the present invention, depending on actual needs.

[0065] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0066] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks.

[0067] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for vectorizing long text, characterized in that, include: Obtain the target question and several candidate web pages retrieved based on the target question, and calculate the relevance score between the web page text of each candidate web page and the target question; Based on the relevance score, the candidate web pages are arranged from high to low, and candidate web pages with relevance scores less than a preset score threshold are removed to obtain a set of candidate web pages. The standard time for vectorization of the web text corresponding to each candidate webpage in the candidate webpage set is calculated based on the preset vectorization time formula and the text length of the web text corresponding to the candidate webpage. Based on the preset equivalent time formula and the relevance score of the webpage text corresponding to each candidate webpage in the candidate webpage set to the target question, the standard time corresponding to the candidate webpages in the candidate webpage set is weighted according to the sorting order to obtain the equivalent time for the vectorization of the webpage text of each candidate webpage in the candidate webpage set. The task of vectorizing candidate web pages in the candidate web page set is executed in sorted order within a predetermined maximum processing time, and the task stops when the vectorization processing time reaches the maximum processing time.

2. The long text vectorization method as described in claim 1, characterized in that, The vectorization time consumption formula is as follows: T proc =K l ×L, Wherein, the T proc K represents the standard time required for webpage text to be vectorized; l represents a fixed-length time coefficient; L represents the text length of the corresponding webpage text for the candidate webpage.

3. The long text vectorization method as described in claim 2, characterized in that, The equivalent time formula is as follows: Among them, T equiv Indicates equivalent time; S represents the relevance score of the candidate webpage text to the target question; K t This represents the time-weighted coefficient.

4. The long text vectorization method as described in claim 1, characterized in that, The calculation of the relevance score between the webpage text of each candidate webpage and the target question includes: Extract the webpage title and webpage summary of each candidate webpage to obtain the webpage description text of each candidate webpage; The webpage description text and the target question text are input into a preset vectorization model for vectorization processing to obtain multiple webpage description vectors and target question vectors. The cosine similarity between each webpage description vector and the target question vector is calculated to obtain the relevance score between each candidate webpage and the target question.

5. A long text vectorization device, characterized in that, include: The acquisition module is used to acquire a target question and several candidate web pages retrieved based on the target question, and to calculate the relevance score between the web page text of each candidate web page and the target question. The elimination module is used to sort the candidate web pages from high to low based on the relevance score, and eliminate candidate web pages whose relevance scores are less than a preset score threshold to obtain a set of candidate web pages. The standard duration calculation module is used to calculate the standard duration for vectorizing the web text corresponding to each candidate webpage in the candidate webpage set according to the preset vectorization time consumption formula and the text length of the web text corresponding to the candidate webpage, so as to obtain the standard time for vectorizing the web text of each candidate webpage in the candidate webpage set. The weighted calculation module is used to perform weighted calculation on the standard time corresponding to the candidate web pages in the candidate web page set according to the preset equivalent time formula and the relevance score of the web page text corresponding to each candidate web page in the candidate web page set to the target question, in order of sorting, so as to obtain the equivalent time for the vectorization of the web page text of each candidate web page in the candidate web page set. The vectorization module is used to perform vectorization processing on candidate web pages in the candidate web page set in sorted order within a predetermined maximum processing time, until the vectorization processing time reaches the maximum processing time and the task is stopped.

6. The long text vectorization device as described in claim 5, characterized in that, The acquisition module includes: The vectorization processing unit is used to input the webpage text of each candidate webpage and the question text of the target question into a preset vectorization model for vectorization processing to obtain multiple webpage text vectors and target question vectors. The relevance calculation unit is used to calculate the cosine similarity between each webpage text vector and the target question vector, and obtain the relevance score between each webpage text and the target question.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the long text vectorization method as described in any one of claims 1 to 4.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the long text vectorization method as described in any one of claims 1 to 4.

Images