An engineering technical document retrieval method, device and medium

Abstract

The application discloses an engineering technical text retrieval method and device and a medium, relates to the technical field of information retrieval and natural language processing, and comprises the following steps: performing data preprocessing on a query text to obtain a standard text; extracting a standard semantic field of the standard text to obtain a standardized parameter set; selecting a key field value and a standardized expression from the standardized parameter set, and supplementarily screening an anchor point from the standard text according to a word formation and length rule to obtain an anchor point term; splicing the standardized parameter set and the anchor point term into the standard text to form an enhanced text; transforming the enhanced text to obtain a text vector of the query text; and performing retrieval based on the text vector of the query text to obtain a retrieval result. The application improves the accuracy of the retrieval result and can better retrieve engineering technical texts.

Description

Technical Field

[0001] This application relates to the fields of information retrieval and natural language processing technology, and in particular to an engineering technical text retrieval method, device and medium. Background Technology

[0002] In the R&D, procurement, quality inspection, and operation and maintenance processes of enterprises, it is necessary to search for engineering and technical texts such as material specifications, BOM item descriptions, datasheet paragraphs, test report summaries, maintenance and quality inspection records, procurement requirement texts, equipment parameter descriptions, engineering change notices (ECN / ECO), and component packaging and marking information.

[0003] However, engineering and technical texts often mix natural language, numerical values ​​and units, and encapsulation and labeling codes, resulting in verbose and non-standard expressions. The same parameter may have multiple ways of being written (e.g., 10K, 10kΩ, 10KR; mΩ, mohm, mΩ@condition, etc.), and the mixing of Chinese and English / symbols is common. Therefore, in the retrieval scenario of engineering and technical texts, traditional keyword retrieval methods or vectorized retrieval methods without industry feature normalization struggle to reliably determine technical equivalence / approximation relationships, leading to inconsistent accuracy of retrieval results. When using clustering retrieval methods, noise words and length differences in the high-dimensional sparse space affect similarity and cluster boundaries, resulting in unstable retrieval and clustering quality. Therefore, existing technologies cannot effectively retrieve engineering and technical texts. Summary of the Invention

[0004] The purpose of this application is to provide a method, device, and medium for retrieving engineering and technical texts, which can better retrieve engineering and technical texts.

[0005] To achieve the above objectives, this application provides the following solution: Firstly, this application provides a method for retrieving engineering and technical texts, including: The query text is preprocessed to obtain the normalized text; Extract the standard semantic fields from the standard text to obtain the set of normalization parameters; Key field values ​​and their standardized expressions are selected from the set of normalized parameters, and anchor points are supplemented and filtered from the normalized text according to word formation and length rules to obtain anchor point terms; The normalized parameter set and the anchor terms are concatenated into the normalized text to form enhanced text; The enhanced text is transformed to obtain the text vector of the query text; The search is performed based on the text vector of the query text to obtain the search results.

[0006] Optionally, the step of preprocessing the query text to obtain standardized text specifically includes: Merge the key fields of the query text to form the original specification text; wherein the key fields include at least one of title, number, specification and description; For the original standard text, redundant whitespace characters and invisible characters are removed, and the case and half-width characters are standardized to obtain the standard text.

[0007] Optionally, the extraction of standard semantic fields from the standardized text to obtain a set of standardized parameters specifically includes: Based on regular expression templates, industry parameters are identified and unified into standard semantic fields from the normative text to obtain a standardized parameter set.

[0008] Optionally, the step of identifying and unifying industry parameters from the normative text into standard semantic fields based on regular expression templates specifically includes: Based on regular expression templates, resistor, on-resistance, capacitance, inductance, frequency, voltage, current, power, size, package, marking code, certification mark, temperature and / or temperature drift parameters are identified from the specification text and unified into standard semantic fields.

[0009] Optionally, the transformation of the enhanced text to obtain the text vector of the query text specifically includes: The enhanced text is transformed by a fitted vectorizer to obtain a sparse vector matrix; Perform TruncatedSVD on the sparse vector matrix to obtain a dimension-reduced matrix; The reduced-dimensional matrix is ​​L2 normalized to obtain the unit norm matrix, which is used as the text vector of the query text. The retrieval based on the text vector of the query text to obtain the retrieval results specifically includes: Calculate the similarity between the text vector of the query text and each library vector in the constructed library vector set, and sort the library texts in the library text set in descending order based on the similarity, and output the Top-K of the descending sort results as the retrieval results; wherein, for each library text in the library text set, perform the process of obtaining the text vector of the query text from the query text, and obtain the text vector of each library text to form the library vector set.

[0010] Optionally, the transformation of the enhanced text using a fitted vectorizer to obtain a sparse vector matrix specifically includes: The enhanced text is subjected to word segmentation, stop word removal, and dimension truncation operations that are completely consistent with the fitting stage of the vectorizer, resulting in a series of word units; For any word element in the enhanced text, the vectorizer is used to find the word element that was pre-stored during the fitting stage. IDF Value, and based onIDF The value is used to calculate the final weight of each word in the enhanced text; The final weight of any word in the enhanced text is filled into the index position corresponding to any word in the created initial vector to obtain a sparse vector matrix; wherein the dimension of the initial vector is equal to the size of the vocabulary, and all elements of the initial vector are initialized to 0.

[0011] Optionally, the basis IDF The final weight of each word in the enhanced text is calculated according to the following formula: ; in, Score This indicates the final weight of any given word in the enhanced text. TF Indicates the word frequency of any given word element. This represents the word frequency saturation control parameter. b This represents the document length normalization parameter. Indicates the length of the enhanced text. avgdl This represents the average length of all documents in the document collection.

[0012] Optionally, the basis IDF The final weight of each word in the enhanced text is calculated according to the following formula: ; in, Weight This indicates the final weight of any given word in the enhanced text. TF The frequency of any given word element is indicated.

[0013] In a second aspect, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the engineering text retrieval method described in any one of the above.

[0014] Thirdly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the engineering text retrieval method described in any one of the above.

[0015] According to the specific embodiments provided in this application, the following technical effects are disclosed: This application provides a method, device, and medium for retrieving engineering and technical texts. By preprocessing the query text to obtain standardized text, extracting standard semantic fields from the standardized text to obtain a set of standardized parameters, selecting key field values ​​and their standardized expressions from the set of standardized parameters, and supplementing and filtering anchor points from the standardized text according to word formation and length rules to obtain anchor point terms, the set of standardized parameters and anchor point terms are concatenated into the standardized text to form enhanced text. This enhances the ability to perform retrieval based on the enhanced text (by transforming the enhanced text to obtain the text vector of the query text, and performing retrieval based on the text vector of the query text). This significantly improves the ability to distinguish technical equivalence / approximation relationships between texts, reduces errors caused by homonyms and differences in unit conversion, and thus improves the accuracy of retrieval results. Therefore, the embodiments of this application can better retrieve engineering and technical texts. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart illustrating an engineering technology text retrieval method provided in an embodiment of this application; Figure 2 A schematic diagram of the functional modules of an engineering technology text retrieval device provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0020] In one exemplary embodiment, such as Figure 1As shown, an engineering technology text retrieval method is provided. This method is executed by a computer device, specifically by a terminal or server alone, or by both a terminal and a server. It includes the following steps 101 to 106. Wherein: Step 101: Perform data preprocessing on the query text to obtain the standardized text.

[0021] Step 102: Extract the standard semantic fields from the standard text to obtain the set of normalization parameters.

[0022] In this embodiment of the application, standard semantic fields of the standardized text are extracted, that is, different text descriptions of the same object are standardized, thereby converting different text descriptions of the same object into standard semantic fields, which facilitates subsequent retrieval processing.

[0023] Step 103: Select key field values ​​and their standardized expressions from the set of standardized parameters, and supplement and filter anchor points from the standardized text according to word formation and length rules to obtain anchor point terms.

[0024] In this application embodiment, the key field values ​​include at least key parameters such as resistance, capacitance, voltage, current, power, package, and size.

[0025] Anchor points are a supplement to the standardized parameter set. In this application embodiment, terms that are difficult to standardize are recorded as anchor points. Anchor points are supplemented and filtered according to word formation and length rules. This means that terms that are difficult to standardize are filtered according to word formation and length rules, and terms that are too long (such as those exceeding 200 characters) are filtered out. The anchor points in this application embodiment include at least device type, package description, marking code, certification information, performance phrases, etc.

[0026] Step 104: Concatenate the normalized parameter set and anchor terms into the normalized text to form the enhanced text.

[0027] Step 105: Transform the enhanced text to obtain the text vector of the query text.

[0028] Step 106: Perform a search based on the text vector of the query text to obtain the search results.

[0029] The true discriminative information in engineering and technical texts is often not the surface-level wording of natural language itself, but rather the technical features contained within, such as parameters, units, encapsulation, marking codes, and authentication information. If the original text is searched directly, different wordings will be treated as different features. For example, 10K, 10kΩ, and 10000Ω, although technically equivalent, have different literal forms and are easily scattered in the vector space. In this embodiment, domain feature normalization is used to first map these different wordings to standard semantic fields and standardized expressions, so that formal differences converge to the same technical meaning, thereby making technically equivalent texts closer in the representation space. Then, through anchor point enhancement, the most critical parameter values, encapsulation, size, marking codes, authentication, and other highly discriminative information for distinguishing technical objects are explicitly injected into the enhanced text. This makes the model pay more attention to the technical essence rather than surface noise when calculating similarity, thus improving the ability to identify "technically equivalent / approximate" relationships and reducing errors caused by homonyms and differences in unit conversion. Therefore, by normalizing domain features and enhancing anchor points, we can improve the ability to distinguish technical equivalence / approximation relationships between texts and reduce errors caused by homonyms and differences in unit conversion.

[0030] By implementing steps 101 to 106 above, the query text is preprocessed to obtain standardized text. Standard semantic fields are extracted from the standardized text to obtain a set of standardized parameters. Key field values ​​and their standardized expressions are selected from the set of standardized parameters. Anchor points are supplemented and filtered from the standardized text according to word formation and length rules to obtain anchor terms. The set of standardized parameters and anchor terms are concatenated into the standardized text to form enhanced text, so that retrieval can be performed based on the enhanced text (by transforming the enhanced text to obtain the text vector of the query text, retrieval is performed based on the text vector of the query text). This significantly improves the ability to distinguish technical equivalence / approximation relationships between texts, reduces errors caused by homonyms and differences in unit conversion, and thus improves the accuracy of retrieval results. Therefore, the embodiments of this application can better retrieve engineering and technical texts.

[0031] In another exemplary embodiment of this application, step 101 specifically includes steps 201 to 202. Wherein: Step 201: Merge the key fields of the query text to form the original specification text; wherein the key fields include at least one of title, number, specification and description.

[0032] In this application embodiment, the title refers to the material / document name, such as a chip resistor. The number refers to the part number, drawing number, or model, such as RC0603FR-0710K. The specification refers to the parameters, such as 10kΩ, 1% 0603. The description refers to supplementary information, such as thick film, RoHS compliance, and applicability to power boards.

[0033] Step 202: For the original standard text, remove redundant whitespace characters and invisible characters, and unify the capitalization and half-width of the characters to obtain the standard text.

[0034] In this embodiment of the application, invisible characters refer to characters in the text that are not usually displayed directly but affect parsing and matching, such as tabs, newlines, carriage returns, etc.

[0035] The same technical content may be identified as different texts due to scattered fields, inconsistent writing styles, or format noise, thus affecting parameter normalization, similarity calculation, and retrieval accuracy. Therefore, this application first integrates scattered fields into a unified input and eliminates interference from differences in capitalization, full-width / half-width characters, whitespace, and formatting. This ensures that subsequent parameter extraction, anchor point filtering, vectorization encoding, and similarity calculation are based on consistent text, thereby improving feature recognition accuracy, retrieval consistency, and result reproducibility.

[0036] In another exemplary embodiment of this application, step 102 specifically includes: Step 301: Based on regular expression templates, identify and unify industry parameters from the standard text into standard semantic fields to obtain a set of standardized parameters.

[0037] In another exemplary embodiment of this application, step 301 specifically includes: Step 401: Based on regular expression templates, identify resistance, on-resistance, capacitance, inductance, frequency, voltage, current, power, size, package, marking code, certification mark, temperature and / or temperature drift parameters from the standard text and unify them into standard semantic fields to obtain a standardized parameter set.

[0038] In this embodiment of the application, industry parameters are identified and unified into standard semantic fields from the normative text T_norm based on regular expression templates, specifically including: First, the specification text T_norm is matched item by item according to the preset regular expression template to identify the following parameters: resistance ohm (including R, k / M / m, Ω symbols), on-resistance Rds_on, capacitance cap_f (including p / n / u / µ / μ / m dimensions), inductance ind_h, frequency freq_hz, power power_w (including fraction / unit), voltage volt (including k / M / m dimensions), current / peak / continuous current, dimensions L×W×H, package pkg (e.g., 0201 / 0402 / 0603 / …), threshold voltage Vth / Vgs, temperature and temperature drift parameters, and certification marks (UL / TÜV / CSA). After matching parameters such as marking codes, the parameters of resistance, inductance, frequency, power (including fractions / units), voltage (including k / M / m dimensions), current / peak / continuous current, size L×W×H, package (0201 / 0402 / 0603 / …), threshold voltage, temperature and temperature drift parameters, certification marks (UL / TÜV / CSA), and / or marking codes are unified into standard semantic fields (field name-standard value) according to conversion rules, unit rules, and field mapping rules. The field names are ohm, Rds_on, cap_f, ind_h, freq_hz, power_w, volt, pkg, etc. For example, the resistance matching is expressed as 10kΩ, 10K, 10KR, 10000Ω, and unified as ohm=10000.

[0039] The marking code is a character code used to identify the model, batch, or electrical characteristics. Certification marks are certification information indicating that a product conforms to specific safety or industry standards, such as UL, TÜV, and CSA.

[0040] In another exemplary embodiment of this application, step 105 specifically includes steps 501 to 503. Wherein: Step 501: Transform the enhanced text using the fitted vectorizer to obtain a sparse vector matrix.

[0041] Step 502: Perform TruncatedSVD (truncated singular value decomposition) on the sparse vector matrix to reduce the dimensionality of the sparse vector matrix to the target dimension, thereby obtaining the dimensionality-reduced matrix.

[0042] In this embodiment, TruncatedSVD is performed on the sparse vector matrix to reduce the sparse vector to the target dimension (preferably 256 dimensions) to obtain a dimension-reduced matrix, thereby reducing computational and storage overhead.

[0043] Step 503: Perform L2 normalization on the dimensionality reduction matrix to obtain the unit norm matrix, which is used as the text vector of the query text.

[0044] In this embodiment, the dimensionality reduction matrix is ​​L2 normalized to obtain the unit norm matrix, so as to maintain the geometric stability of cosine similarity and ensure the stability of clustering and retrieval.

[0045] In another exemplary embodiment of this application, step 106 specifically includes: The similarity between the text vector of the query text and each library vector in the constructed library vector set is calculated. Based on the similarity, the library texts in the library text set are sorted in descending order, and the Top-K of the descending sort results are output as the search results. The processing method of each library vector in the library vector set is exactly the same as that of the unit norm matrix of the query text.

[0046] The process of obtaining the text vector of the query text from the query text is performed on each text in the library text set in steps 101 to 105, resulting in a text vector for each library text and forming a library vector set. Therefore, there is a one-to-one correspondence between the library vectors in the library vector set and the library texts in the library text set. The library texts in the library text set are then sorted in descending order based on similarity. Specifically, based on the similarity between the unit norm matrix of each library vector in the library vector set and the query text, the library texts corresponding to all library vectors in the library vector set are sorted in descending order, resulting in a descending order ranking. The Top-K represents the top k library texts in the descending order ranking. The processing method for each library vector in the library vector set is completely consistent with that of the unit norm matrix of the query text; that is, the word segmentation rules, vocabulary, stop word list, parameters, IDF statistics, SVD projection matrix, and normalization method used are all consistent, ensuring the reproducibility of the results.

[0047] The embodiments of this application combine sparse representation (sparse vector matrix) + SVD dimensionality reduction + L2 normalization to balance interpretability, performance and memory usage, making it suitable for large-scale processing of short engineering texts.

[0048] In another exemplary embodiment of this application, step 501 specifically includes steps 601 to 603. Wherein: Step 601 involves performing word segmentation, stop word removal, and dimension truncation operations on the enhanced text, which are identical to those performed in the vectorizer fitting stage, to obtain a series of lexical units.

[0049] In this embodiment of the application, word frequency is counted according to the unified word segmentation rules. After the enhanced text is tokenized, the word units obtained by word segmentation are traversed by the stop word removal operation and compared with the preset stop word list to determine whether each word unit obtained by word segmentation exists in the stop word list. If any word unit is in the stop word list, it is removed from the word unit sequence obtained by word segmentation. If any word unit is not in the stop word list, it is retained.

[0050] In this embodiment of the application, the stop word list generally includes words that have no actual distinguishing meaning for retrieval, such as environmental protection labels, brand information, etc.

[0051] During word segmentation, n-gram expansion is performed using configured n-grams. This expansion not only preserves basic lexical features but also captures collocation features of adjacent words, effectively distinguishing homonyms and extracting phrases with specific semantics. For example, in the text "conduction resistance is very low," viewed as a single word, it contains "conduction" and "resistance." However, viewed using n-gram(1,2), in addition to the single word, the phrase "conduction resistance" is also retained. The advantage of this approach is that "conduction resistance" is a more accurate representation of a complete technical concept than when split into two words. Higher n-gram orders result in a larger feature space and significantly increased computational overhead.

[0052] Even after filtering stop words using a stop word list, the unique vocabulary size in a text corpus can still reach tens or even hundreds of thousands. Directly constructing such a high-dimensional matrix would lead to severe "curse of dimensionality" and memory overflow. Therefore, in this embodiment, after stop word filtering, a dimensionality truncation operation is performed using the configured `max_features`. `max_features` limits the number of feature words retained. The system typically sorts all candidate words in descending order according to feature importance or term frequency (TF), retaining only the top N words (e.g., 5000 or 10000). This not only significantly reduces computational complexity but also automatically filters out rare or obscure words that appear only in a very small number of documents (these words are often noise or overfitting sources), making computation faster and results more stable.

[0053] Step 602: For any word element in the enhanced text, use the vectorizer described above to find the word element that was pre-stored during the fitting stage. IDF (Inverse document frequency) value, and based on IDF The value is used to calculate the final weight of any given word in the enhanced text.

[0054] In the embodiments of this application, during the fitting stage IDF The value is calculated using the following formula: IDF =log((N-df+0.5) / (df+0.5)+1); Where N represents the total number of documents participating in the fitting, and df represents the number of documents containing any of the above-mentioned terms.

[0055] Step 603: Fill the final weight of any of the above-mentioned words in the enhanced text into the index position corresponding to the word in the created initial vector to obtain a sparse vector; wherein, the dimension of the initial vector is equal to the size of the vocabulary, and all elements of the initial vector are initialized to 0.

[0056] In another exemplary embodiment of this application, the above-mentioned vectorizer employs a TF-IDF vectorizer, and correspondingly, based on IDF The final weight of any given word in the enhanced text is calculated using the following formula: ; in, Weight This indicates the final weight of any of the above terms in the enhanced text. TF This indicates the word frequency of any given word (the actual number of times that word appears in the currently retrieved document).

[0057] In another exemplary embodiment of this application, to achieve accurate retrieval of short texts, the vector generator described above employs a BM25 vector generator, and correspondingly, based on... IDF The final weight of any given word in the enhanced text is calculated using the following formula: ; in, Score This indicates the final weight of any of the above terms in the enhanced text. TF Indicates the frequency of any given word element. This represents the word frequency saturation control parameter. b This represents the document length normalization parameter. This indicates the length of the currently retrieved document, i.e., the enhanced text. avgdl This represents the average length of all documents in the document set (the set of all enhanced texts used to fit the vector generator during the library vector set building phase).

[0058] In existing text retrieval methods, sparse representation does not combine document length normalization and word frequency saturation control. High-frequency, frequently used words in short texts easily amplify noise, causing texts to be incorrectly classified as similar due to general similarity in description. This weakens truly crucial features such as parameters, encapsulation, and tag codes, leading to false positives and false negatives. The embodiments of this application... and b Normalizing document length and controlling word frequency saturation can effectively avoid the problem of high-frequency commonly used words in short texts amplifying noise, and avoid false detection and missed detection caused by the amplification of noise by high-frequency commonly used words in short texts.

[0059] In another exemplary embodiment of this application, to improve retrieval accuracy, =1.5, b =0.75.

[0060] In another exemplary embodiment of this application, the above-mentioned library vector set is constructed according to the following steps: For each library text in the library text set, perform steps 101 to 105 to obtain the text vector of the query text from the query text, and obtain the text vector of each library text to form a library vector set.

[0061] In another exemplary embodiment of this application, to improve the accuracy of the retrieval results, the text processing parameters are optimized based on steps 701 to 705. Therefore, the above-described engineering text retrieval method further includes, before step 501: Step 701: Based on the current text processing parameters, perform the process of obtaining the text vector of the query text from the query text in steps 101 to 105 for each library text in the library text set.

[0062] In the first execution of step 701, i.e., the initial text processing parameters are used as the current text processing parameters.

[0063] If the vector generator in step 501 uses TF-IDF, then the text processing parameters include at least n-gram, stop words, max_features, and target dimension. If the vector generator in step 501 uses BM25, then the text processing parameters also include word frequency saturation control parameters. Document length normalization parameter b .

[0064] Step 702: Use DBSCAN to cluster all the text vectors obtained in step 701 to obtain the clustering results.

[0065] When using DBSCAN clustering, the parameters are fixed (e.g., eps=0.3, min_samples=8) or candidate parameter groups are generated through grid search.

[0066] The clustering results include the cluster label, cluster number, and noise point distribution for each library text. The cluster label is the group number for each library text. In this embodiment, similar library texts are grouped into the same group through clustering, and each group of library texts has a corresponding group number. The clustering evaluation is used to verify whether steps 101 to 104 and steps 501 to 503 can stably reflect technical similarity.

[0067] Step 703: Check whether the clustering results meet the constraints; the constraints include non-noise coverage ≥ 50%, number of clusters ≥ 2, silhouette coefficient ≥ 0.2, and Calinski-Harabasz exponent recovery.

[0068] In this embodiment of the application, the clustering results are detected to meet the constraints, namely, whether the non-noise coverage, number of clusters, silhouette coefficient, and Calinski-Harabasz index of the clustering results meet the following conditions: non-noise coverage ≥ 50%, number of clusters ≥ 2, silhouette coefficient ≥ 0.2, and Calinski-Harabasz index increases.

[0069] Step 704: If the clustering results meet the constraints, calculate the comprehensive score S of the clustering results according to S = silhouette (representing silhouette coefficient) + 0.05×CH (representing Calinski-Harabasz index) + 0.5×coverage (representing non-noise coverage).

[0070] Step 705: Detect whether the comprehensive score S is not less than the preset comprehensive score threshold. If so, use the current text processing parameters as the optimized text processing parameters. Otherwise, obtain the modified text processing parameters, use the modified text processing parameters as the current text processing parameters, and return to step 701.

[0071] In this embodiment of the application, if the calculated comprehensive score S is less than the preset comprehensive score threshold, the text processing parameters are automatically / manually modified, and the modified text processing parameters are used as the current text processing parameters. Then, the process returns to step 701 to continue cluster evaluation until optimized text processing parameters are obtained.

[0072] Existing text clustering processes do not incorporate business constraints (non-noise coverage, cluster size, silhouette coefficient, etc.), making it difficult to balance quality (intra-cluster consistency, inter-cluster separability) and quantity (non-noise coverage, number of available clusters), resulting in poor performance in practical applications. In contrast, the embodiments in this application perform clustering and parameter selection under constraints such as non-noise coverage, cluster size, and silhouette coefficient, achieving a balance between quality and quantity, and better meeting business application needs (such as specification archiving, material grouping, and knowledge base organization).

[0073] The offline retrieval and clustering evaluation in this application embodiment share a common pipeline (i.e., steps 101 to 104 and steps 501 to 503), and the results are reproducible, monitorable, and easy to migrate and audit in different systems.

[0074] The retrieval method of this application embodiment is applicable to all engineering technical texts (such as BOM, datasheet fragments, test / quality inspection, maintenance records, procurement requirements, equipment parameters, ECN / ECO, packaging / marking information, etc.). By simply expanding the feature templates and rules, the method can maintain consistency and stability in different engineering fields. The feature templates include parameter recognition templates, i.e., regular expression templates used to identify resistors, capacitors, voltages, currents, power, dimensions, packages, marking codes, certification marks, etc. The rules include unit normalization rules, field mapping rules, and text normalization rules. The unit normalization rules are used to unify different expressions (such as 10K, 10kΩ, 10000Ω) to the same standard expression. The field mapping rules are used to unify the identified industry parameter results to standard semantic fields such as ohm, cap_f, volt, pkg, etc. The text normalization rules are used to unify capitalization, full-width / half-width characters, and clean up whitespace and invisible characters.

[0075] This application addresses the problem that existing technologies cannot effectively retrieve engineering and technical texts through a multi-layered approach. First, it performs parameter normalization and anchor point enhancement to reduce invalid expression differences and increase the proportion of key technical features. Second, in the sparse representation stage, it uses BM25 or TF-IDF to control word frequency saturation, preventing the unlimited amplification of high-frequency word weights, and normalizes document length to mitigate the natural dominance of long texts. Third, it further reduces noise dimensions and stabilizes similarity metrics, thereby making cluster boundaries more stable and retrieval results more reliable.

[0076] Based on the same inventive concept, this application also provides an engineering technical text retrieval device for implementing the above-mentioned engineering technical text retrieval method. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more engineering technical text retrieval device embodiments provided below can be found in the limitations of the engineering technical text retrieval method described above, and will not be repeated here.

[0077] In one exemplary embodiment, such as Figure 2 As shown, an engineering technology text retrieval device 80 is provided, comprising: Input module 801 is used to obtain the query text; Processing module 802 is used for: The query text is preprocessed to obtain the normalized text; Extract the standard semantic fields from the standard text to obtain the set of normalization parameters; Key field values ​​and their standardized expressions are selected from the set of normalized parameters, and anchor points are supplemented and filtered from the normalized text according to word formation and length rules to obtain anchor point terms; The normalized parameter set and anchor terms are concatenated into the normalized text to form the enhanced text; The enhanced text is transformed to obtain the text vector of the query text; The search is performed based on the text vector of the query text to obtain the search results.

[0078] In this embodiment of the application, the input module 801 can obtain the query text input by the user through the human-computer interaction interface.

[0079] In another exemplary embodiment of this application, regarding the aspect of preprocessing query text to obtain standardized text, processing module 802 is specifically used for: Merge the key fields of the query text to form the original specification text; the key fields include at least one of the following: title, number, specification, and description; For the original standard text, remove redundant whitespace and invisible characters, and unify the capitalization and half-width of the characters to obtain the standard text.

[0080] In another exemplary embodiment of this application, regarding the aspect of extracting standard semantic fields from canonical text to obtain a set of normalized parameters, processing module 802 is specifically used for: Based on regular expression templates, industry parameters are identified and unified into standard semantic fields from standard text to obtain a standardized parameter set.

[0081] In another exemplary embodiment of this application, regarding the aspect of identifying and unifying industry parameters from canonical text into standard semantic fields based on regular expression templates to obtain a normalized parameter set, processing module 802 is specifically used for: Based on regular expression templates, resistor, on-resistance, capacitance, inductance, frequency, voltage, current, power, size, package, marking code, certification mark, temperature and / or temperature drift parameters are identified from the standard text and unified into standard semantic fields to obtain a standardized parameter set.

[0082] In another exemplary embodiment of this application, regarding the aspect of transforming the enhanced text to obtain the text vector of the query text, the processing module 802 is specifically used for: The enhanced text is transformed by a fitted vectorizer to obtain a sparse vector matrix; Perform TruncatedSVD on the sparse vector matrix to obtain the dimension-reduced matrix; The reduced-dimensional matrix is ​​L2 normalized to obtain the unit norm matrix, which is used as the text vector of the query text.

[0083] In another exemplary embodiment of this application, regarding the aspect of retrieving search results based on text vectors of query text, the processing module 802 is specifically used for: The similarity between the text vector of the query text and each library vector in the constructed library vector set is calculated. Based on the similarity, the library texts in the library text set are sorted in descending order, and the Top-K of the descending sort results are output as the search results.

[0084] The process of obtaining the text vector of the query text from the query text is performed on each text in the library text set in steps 101 to 105, resulting in a text vector for each library text and forming a library vector set. Therefore, there is a one-to-one correspondence between the library vectors in the library vector set and the library texts in the library text set. The library texts in the library text set are then sorted in descending order based on similarity. Specifically, based on the similarity between the unit norm matrix of each library vector in the library vector set and the query text, the library texts corresponding to all library vectors in the library vector set are sorted in descending order, resulting in a descending order ranking. The Top-K represents the top k library texts in the descending order ranking. The processing method for each library vector in the library vector set is completely consistent with that of the unit norm matrix of the query text; that is, the word segmentation rules, vocabulary, stop word list, parameters, IDF statistics, SVD projection matrix, and normalization method used are all consistent, ensuring the reproducibility of the results.

[0085] In another exemplary embodiment of this application, regarding the aspect of transforming the enhanced text through a fitted vectorizer to obtain a sparse vector matrix, the processing module 802 is specifically used for: The enhanced text undergoes word segmentation, stop word removal, and dimension truncation operations that are completely consistent with the fitting stage of the vectorizer, resulting in a series of word units; For any word in the enhanced text, the vectorizer described above is used to find the word that was pre-stored during the fitting phase. IDF (Inverse document frequency) value, and based on IDF The value is used to calculate the final weight of any given word in the enhanced text; The final weight of any of the above-mentioned words in the enhanced text is filled into the index position corresponding to that word in the created initial vector to obtain a sparse vector matrix; where the dimension of the initial vector is equal to the size of the vocabulary, and all elements of the initial vector are initialized to 0.

[0086] In this embodiment, word frequencies are statistically analyzed according to a unified word segmentation rule. After tokenization of the enhanced text, the segmented lexical units are traversed through a stop word removal operation and compared with a preset stop word list. It is determined whether each segmented lexical unit exists in the stop word list. If any lexical unit is in the stop word list, it is removed from the segmented lexical unit sequence; otherwise, it is retained. The stop word list generally includes words without actual distinguishing meaning for retrieval, such as environmental labels and brand information. During the word segmentation process, n-gram expansion is performed using configured n-grams.

[0087] In another exemplary embodiment of this application, for use based on IDF In terms of calculating the final weight of any given word in the enhanced text, processing module 802 is specifically used for: The final weight of any word in the enhanced text is calculated using the following formula: ; in, Weight This indicates the final weight of any of the above terms in the enhanced text. TF This indicates the word frequency of any given word (the actual number of times that word appears in the currently retrieved document).

[0088] In another exemplary embodiment of this application, for use based on IDF In terms of calculating the final weight of any given word in the enhanced text, processing module 802 is specifically used for: The final weight of any word in the enhanced text is calculated using the following formula: ; in, Score This indicates the final weight of any of the above-mentioned terms in the enhanced text. TF Indicates the frequency of any given word element. This represents the word frequency saturation control parameter. b This represents the document length normalization parameter. This indicates the length of the currently retrieved document. avgdl This represents the average length of all documents in the document collection.

[0089] In another exemplary embodiment of this application, the processing module 802 described above is further configured to: Perform the above process of obtaining the text vector of the query text from the query text on each text in the library text set to obtain the text vector of each text, forming a library vector set.

[0090] In another exemplary embodiment of this application, before transforming the enhanced text using a fitted vectorizer to obtain a sparse vector matrix, the processing module 802 is further configured to: Based on the current text processing parameters, the process of obtaining the text vector of the query text from the query text is performed for each library text in the library text set; where, initially, the initial text processing parameters are used as the current text processing parameters. DBSCAN was used to cluster all the obtained text vectors to obtain the clustering results. The clustering results are checked to see if they meet the constraints, which include non-noise coverage ≥ 50%, number of clusters ≥ 2, silhouette coefficient ≥ 0.2, and Calinski-Harabasz exponent recovery. If the clustering results meet the constraints, the overall score S of the clustering results is calculated as S = silhouette (representing the silhouette coefficient) + 0.05 × CH (representing the Calinski-Harabasz index) + 0.5 × coverage (representing the non-noise coverage). If the overall score S is not less than the preset overall score threshold, the current text processing parameters are used as the optimized text processing parameters. Otherwise, the modified text processing parameters are obtained, the modified text processing parameters are used as the current text processing parameters, and the process of obtaining the text vector of the query text from the query text is returned for each library text in the library text set based on the current text processing parameters.

[0091] In another exemplary embodiment of this application, the above-described engineering text retrieval device 80 further includes: Storage module 803 is used to store the library vector set constructed by processing module 802.

[0092] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 3 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores engineering and technical text retrieval data. The I / O interfaces allow the processor to exchange information with external devices. The communication interface allows communication with external terminals via a network connection. When executed by the processor, the computer program implements an engineering and technical text retrieval method.

[0093] Those skilled in the art will understand that Figure 3The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0094] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0095] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0096] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0097] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0098] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0099] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0100] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for retrieving engineering and technical texts, characterized in that, The engineering and technical text retrieval method includes: The query text is preprocessed to obtain the normalized text; Extract the standard semantic fields from the standard text to obtain the set of normalization parameters; Key field values ​​and their standardized expressions are selected from the set of normalized parameters, and anchor points are supplemented and filtered from the normalized text according to word formation and length rules to obtain anchor point terms; The normalized parameter set and the anchor terms are concatenated into the normalized text to form enhanced text; The enhanced text is transformed to obtain the text vector of the query text; The search is performed based on the text vector of the query text to obtain the search results.

2. The engineering and technical text retrieval method according to claim 1, characterized in that, The step of preprocessing the query text to obtain standardized text specifically includes: Merge the key fields of the query text to form the original specification text; wherein the key fields include at least one of title, number, specification and description; For the original standard text, redundant whitespace characters and invisible characters are removed, and the case and half-width characters are standardized to obtain the standard text.

3. The engineering and technical text retrieval method according to claim 1, characterized in that, The extraction of standard semantic fields from the standardized text yields a set of standardized parameters, specifically including: Based on regular expression templates, industry parameters are identified and unified into standard semantic fields from the normative text to obtain a standardized parameter set.

4. The engineering and technical text retrieval method according to claim 3, characterized in that, The process of identifying and unifying industry parameters into standard semantic fields from the normative text based on regular expression templates specifically includes: Based on regular expression templates, resistor, on-resistance, capacitance, inductance, frequency, voltage, current, power, size, package, marking code, certification mark, temperature and / or temperature drift parameters are identified from the specification text and unified into standard semantic fields.

5. The engineering and technical text retrieval method according to claim 1, characterized in that, The transformation of the enhanced text to obtain the text vector of the query text specifically includes: The enhanced text is transformed by a fitted vectorizer to obtain a sparse vector matrix; Perform TruncatedSVD on the sparse vector matrix to reduce its dimensionality to the target dimension, thus obtaining a dimensionality-reduced matrix; The reduced-dimensional matrix is ​​L2 normalized to obtain the unit norm matrix, which is used as the text vector of the query text. The retrieval based on the text vector of the query text to obtain the retrieval results specifically includes: Calculate the similarity between the text vector of the query text and each library vector in the constructed library vector set, and sort the library texts in the library text set in descending order based on the similarity, and output the Top-K of the descending sort results as the retrieval results; wherein, for each library text in the library text set, perform the process of obtaining the text vector of the query text from the query text, and obtain the text vector of each library text to form the library vector set.

6. The engineering and technical text retrieval method according to claim 5, characterized in that, The transformation of the enhanced text using a fitted vectorizer to obtain a sparse vector matrix specifically includes: The enhanced text is subjected to word segmentation, stop word removal, and dimension truncation operations that are completely consistent with the fitting stage of the vectorizer, resulting in a series of word units; For any word element in the enhanced text, the vectorizer is used to find the word element that was pre-stored during the fitting stage. IDF Value, and based on IDF The value is used to calculate the final weight of each word in the enhanced text; The final weight of any word in the enhanced text is filled into the index position corresponding to any word in the created initial vector to obtain a sparse vector matrix; wherein the dimension of the initial vector is equal to the size of the vocabulary, and all elements of the initial vector are initialized to 0.

7. The engineering and technical text retrieval method according to claim 6, characterized in that, The basis IDF The final weight of each word in the enhanced text is calculated according to the following formula: ； in, Score This indicates the final weight of any given word in the enhanced text. TF Indicates the word frequency of any given word element. This represents the word frequency saturation control parameter. b This represents the document length normalization parameter. Indicates the length of the enhanced text. avgdl This represents the average length of all documents in the document collection.

8. The engineering and technical text retrieval method according to claim 5, characterized in that, The basis IDF The final weight of each word in the enhanced text is calculated according to the following formula: ； in, Weight This indicates the final weight of any given word in the enhanced text. TF The frequency of any given word element is indicated.

9. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the steps of the engineering text retrieval method according to any one of claims 1-8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the engineering and technical text retrieval method according to any one of claims 1-8.

Images