Data storage method supporting retrieval, indexing method and storage system
By determining the continuous feature values and occurrence times of feature words in the data storage, calculating the representation ability parameters, and generating an index information table, the problem of taking into account both the data retrieval efficiency and the effect in the prior art is solved, and the effective index of massive data is achieved. Search.
Patent Information
- Application Number
- CN202411076066.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-07
- Publication Date
- 2024-12-06
AI Technical Summary
The prior art cannot effectively avoid the occurrence of data invalid search while ensuring the efficiency of data retrieval, especially in the case of massive data operation.
By comparing the content of the storage subject with the stop word list, and calculating the characterization ability parameters based on the continuous feature values and occurrence times of the feature word, an index information table is generated to support data retrieval, ensuring the relevance and continuity of the feature word to optimize data storage and retrieval.
It effectively avoids the occurrence of invalid data retrieval, improves the efficiency of data storage and retrieval, and ensures effective index of the retrieval content in the database, especially when searching for multiple file types, the index information displayed is accurate and reliable.
Smart Images

Figure CN119088801A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing technology, and in particular to a data storage method, an indexing method and a storage system supporting retrieval. Background Art
[0002] In the Internet era, business applications have grown rapidly, and the resulting big data operations have become the norm. The operation of massive data and the complex data content have also brought tremendous pressure to the retrieval of data in the database.
[0003] The prior art discloses a data storage and retrieval device, including: an index file management module, which formulates index information for data and records the index information of the data in an index file; a positioning file management module, which formulates storage location information for the data and records the storage location information of the data in a positioning file; and a data file management module, which stores the data as a data file for retrieval according to the storage file information. Through this technical solution, in data applications, especially in situations where massive data is used, it is possible to avoid loading excessive data at one time, reduce system pressure, and at the same time improve the efficiency of data storage, retrieval and recovery.
[0004] However, the above technical solution cannot ensure the retrieval effect while ensuring the retrieval efficiency of data. Summary of the invention
[0005] The purpose of the present invention is to provide a data storage method, indexing method and storage system that support retrieval, which can store data while taking into account the later retrieval of data, and effectively avoid the occurrence of invalid data retrieval.
[0006] To this end, in a first aspect, the present invention provides a data storage method supporting retrieval, the data storage method supporting retrieval comprising:
[0007] Compare the content of a single storage subject with a stop word list to determine a number of characteristic words in the single storage subject;
[0008] Determine the representation capability parameters of each feature word for the storage subject;
[0009] Generate the characteristic words of the storage subject and the characterization capability parameters of the characteristic words into an index information table and store it together with the storage subject;
[0010] The characteristic words are words that appear in the storage body in the top preset number of rankings, excluding the words included in the stop word list;
[0011] The characterization capability parameter is determined according to the continuous feature value and the number of occurrences of the feature word, and the characterization capability parameter is positively correlated with the continuous feature value and the number of occurrences respectively;
[0012] Among them, for the storage body of text type, the continuous feature value is the ratio of the maximum number of continuous fields where the feature word appears continuously in a number of consecutive fields with the same interval to the total number of fields contained in the storage body;
[0013] For a video or audio storage body, the continuous feature value is the ratio of the maximum number of consecutive time periods in which the feature word appears continuously in a number of consecutive time periods with the same interval to the total number of time periods contained in the storage body.
[0014] As a preferred technical solution of the data storage method supporting retrieval, in the determination of the characterization capability parameter of each feature word for the storage subject, the product of the continuous feature value of the feature word and the number of occurrences of the feature word is determined as the characterization capability parameter of the feature word for the storage subject.
[0015] As a preferred technical solution of the data storage method supporting retrieval, in the case of excluding the words included in the stop word list, for determining the characteristic words in the storage subject, in response to the storage subject category being text, the following steps are performed:
[0016] Count and store words that appear more than once in the subject;
[0017] The number of occurrences of each word is compared and counted, and the words ranked before a preset number of positions are determined as the characteristic words of the storage subject.
[0018] As a preferred technical solution of the data storage method supporting retrieval, in the case of excluding the words included in the stop word list, for determining the characteristic words in the storage subject, in response to the storage subject category being audio, the following steps are performed:
[0019] Perform audio and text recognition on the storage subject;
[0020] Count and store words that appear more than once in the subject;
[0021] The number of occurrences of each word is compared and counted, and the words ranked before a preset number of positions are determined as the characteristic words of the storage subject.
[0022] As a preferred technical solution of the data storage method supporting retrieval, in the case of excluding the words included in the stop word list, for determining the characteristic words in the storage subject, in response to the storage subject category being a video, the following steps are performed;
[0023] Performing audio text recognition and image text recognition on the storage subject respectively;
[0024] Count and store words that appear more than once in the subject;
[0025] The number of occurrences of each word is compared and counted, and the words ranked before a preset number of positions are determined as the characteristic words of the storage subject.
[0026] As a preferred technical solution of the data storage method supporting retrieval, after determining the characterization capability parameter of the feature word for the storage subject, the method further includes:
[0027] Semantic analysis is performed on the feature words of the storage subject, and the representation ability parameters of several feature words with the same semantics are all determined as the one with the highest value among the representation ability parameters of the feature words with the same semantics.
[0028] In a second aspect, the present invention provides an indexing method for searching a database obtained by the data storage method supporting retrieval in the above scheme, comprising:
[0029] Determine the storage subject whose characteristic words match the search content;
[0030] The corresponding index information table is read, and the display order of the index information for the storage subject is determined according to the characterization capability parameters of the feature words for the storage subject.
[0031] As a preferred technical solution of the indexing method, determining the display order of the index information for the storage subject according to the characterization capability parameter of the feature word for the storage subject includes:
[0032] Determine the feature words that match each storage entity with the search content;
[0033] The order of displaying the index information for each storage entity is determined by the sum of the representation capability parameters of the feature words matching the search content from large to small.
[0034] In a third aspect, the present invention provides a data storage system, which applies the above-mentioned data storage method supporting retrieval to perform data storage, including:
[0035] A data storage module, used for storing the storage of the subject;
[0036] An extraction module, connected to the data storage module, for extracting the characteristic words of the storage subject and counting the number of occurrences and occurrence nodes of the characteristic words;
[0037] A calculation module, connected to the extraction module, for calculating the continuous feature value and the characterization capability parameter of the feature word;
[0038] The index support module is connected to the extraction module and the calculation module respectively, and is used to generate and store an index data table including the characteristic words of the storage subject and the characterization capability parameters of the characteristic words of the storage subject.
[0039] As a preferred technical solution of the data storage system, the computing module is provided with a semantic analysis unit, and the semantic analysis unit is used to determine semantically identical feature words and refresh the representation capability parameters of the semantically identical feature words.
[0040] The beneficial effects of the present invention are:
[0041] The data storage method supporting retrieval provided by the present invention takes into account the correlation between the continuity of words in the storage subject and the representation ability of the words for the storage subject by determining the representation ability parameters. In actual applications, there are situations where feature words appear in a certain time period. In this case, most feature words can only represent the content of the corresponding time period, which makes the representation of the number of occurrences of feature words weaker. In contrast, feature words that appear continuously in several adjacent time periods are more representative, indicating that the feature words run through the storage subject to a large extent. In this way, the representation ability parameters determined by the number of occurrences and continuous feature values can better reflect the correlation between the feature words and the storage subject. In summary, the data storage method of the present application can take into account the later retrieval of data while storing data, ensuring the effective indexing of the retrieval content in the database, and thus effectively avoiding the occurrence of invalid data retrieval.
[0042] Furthermore, the present invention determines the continuous characteristic values of different types of storage entities by using the ratio of fields to total fields and the ratio of time periods to total time periods, respectively. Such a setting enables storage entities of audio, video and text types to have good uniformity when responding to retrieval, makes the characterization capability parameters comparable for different types of storage entities, and makes the index information displayed in the case of retrieval of multiple file types accurate and reliable, further ensuring the effective indexing of the retrieved content in the database. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] Figure 1 A flowchart of a data storage method supporting retrieval in an embodiment of the present invention;
[0044] Figure 2 is a flowchart of an indexing method in an embodiment of the present invention;
[0045] Figure 3 4 is a structural block diagram of a data storage system in an embodiment of the present invention. DETAILED DESCRIPTION
[0046] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0047] In the description of the present invention, it should be noted that the orientations or positional relationships indicated by the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside", etc. are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. Among them, the terms "first position" and "second position" are two different positions, and the first feature "above", "above" and "above" the second feature include the first feature being directly above and obliquely above the second feature, or simply indicates that the first feature is higher in level than the second feature. The first feature "below", "below" and "below" the second feature include the first feature being directly below and obliquely below the second feature, or simply indicates that the first feature is lower in level than the second feature.
[0048] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
[0049] Embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.
[0050] For ease of understanding, the following terms in this application are defined:
[0051] Stop words: In information retrieval, in order to save storage space and improve search efficiency, certain characters or words are automatically filtered out before or after processing data. These characters or words are called stop words.
[0052] See also Figure 1 As shown, this embodiment provides a data storage method supporting retrieval, and the data storage method supporting retrieval includes:
[0053] Step S1, comparing the content of a single storage subject with a stop word list to determine a number of characteristic words in the single storage subject. It can be understood that the storage subject is data to be stored in the database.
[0054] Step S2, determining the characterization capability parameter of each feature word for the storage subject;
[0055] Step S3, generating the characteristic words of the storage subject and the characterization capability parameters of the characteristic words into an index information table and storing it together with the storage subject;
[0056] Among them, the feature words are the words that appear in the storage body in the top preset number of rankings, excluding the words included in the stop word list; in detail, the stop word list needs to be determined in combination with actual scenarios, such as language, field, etc. The establishment of the stop word list is an existing technology and will not be repeated here.
[0057] The representation ability parameter is determined according to the continuous feature value and the number of occurrences of the feature word, and the representation ability parameter is positively correlated with the continuous feature value and the number of occurrences respectively;
[0058] Among them, for the storage body of text type, the continuous feature value is the ratio of the maximum number of continuous fields where the feature word appears continuously in a number of consecutive fields with the same interval to the total number of fields contained in the storage body;
[0059] For a video or audio storage body, the continuous feature value is the ratio of the maximum number of consecutive time periods in which the feature word appears continuously in a number of consecutive time periods with the same interval to the total number of time periods contained in the storage body.
[0060] In the above embodiment, by determining the representation ability parameters, the correlation between the continuity of words in the storage subject and the representation ability of words for the storage subject is taken into account. In actual applications, there are situations where feature words appear in a certain period of time. In this case, most feature words can only represent the content of the corresponding period of time, which makes the representation of the number of occurrences of feature words weaker. The continuous appearance of feature words in adjacent time periods is more representative, indicating that the feature words run through the storage subject to a large extent. In this way, the representation ability parameters determined by the number of occurrences and continuous feature values can better reflect the relevance of feature words to the storage subject. In summary, the data storage method of this embodiment can store data while taking into account the later retrieval of data, ensuring the effective indexing of the retrieval content in the database, thereby effectively avoiding the occurrence of invalid data retrieval.
[0061] Furthermore, in this embodiment, the representation capability parameters adopted for different types of storage entities are determined by the ratio of fields to total fields and the ratio of time periods to total time periods, respectively. Such a setting can make the storage entities of audio, video and text types have good uniformity when responding to retrieval, make the representation capability parameters comparable for different types of storage entities, and the index information displayed in the case of retrieval of multiple file types is accurate and reliable, further ensuring the effective indexing of the search content in the database.
[0062] In detail, before determining the continuous characteristic values, the storage body is also divided into a number of equally spaced fields or time periods. To ensure that fields or time periods of different lengths are comparable, the number of field or time period divisions must be directly proportional to the length of the field or time period. Exemplarily, the number of field or time period divisions is greater than 5 segments.
[0063] Specifically, in determining the representation capability parameter of each feature word for the storage subject, the product of the continuous feature value of the feature word and the number of occurrences of the feature word is determined as the representation capability parameter of the feature word for the storage subject.
[0064] For example, a storage entity is divided into 100 fields, each field is 200 words long, the feature word "data set" appears in the 10th to 19th fields, and the feature word "data set" appears in the 15th to 18th fields. The maximum number of consecutive fields in which the feature word appears is determined to be 10, and the continuous feature value is 0.1. The feature word "data set" appears 98 times in the entire storage entity, and the characterization capability parameter of the feature word for the storage entity is determined to be 9.8.
[0065] Specifically, in the case of excluding words included in the stop word list, for determining the characteristic words in the storage body, in response to the storage body category being text, the following steps are performed:
[0066] Step S101, counting the words that appear more than once in the storage body;
[0067] Step S102, comparing the number of occurrences of each word counted, and determining the words ranked before a preset number of positions as the characteristic words of the storage subject.
[0068] Specifically, when the words included in the stop word list are excluded, for determining the characteristic words in the storage body, the following steps are performed in response to the storage body category being audio:
[0069] Step S111, performing audio and text recognition on the storage subject;
[0070] Step S112, counting the words that appear more than once in the storage body;
[0071] Step S113, comparing the number of occurrences of each word that has been counted, and determining the words that rank before a preset number of positions as the characteristic words of the storage subject.
[0072] Specifically, excluding words included in the stop word list, for determining the characteristic words in the storage subject, the following steps are performed in response to the storage subject category being a video;
[0073] Step S121, performing audio text recognition and image text recognition on the storage subject respectively;
[0074] Step S122, counting the words that appear more than once in the storage body;
[0075] Step S123, comparing the number of occurrences of each word counted, and determining the words ranked before a preset number of positions as the characteristic words of the storage subject.
[0076] Of course, in implementation, the preset number needs to be adaptively set in combination with the amount of data contained in the storage body. The preset number cannot be set too small, so that the preset number of feature words can effectively represent the content of the storage body. At the same time, the preset number cannot be set too large to ensure that there are fewer words that are irrelevant to the content of the storage body among the preset number of feature words, thereby reducing the waste of storage space and computing power.
[0077] In the above embodiment, it is taken into consideration that in actual applications, the original text description of the storage subject of audio, video or text type may not accurately or effectively represent the actual content, so it is necessary to extract and determine to eliminate invalid retrieval or poor retrieval caused by this problem. By recognizing text in the storage process, the impact caused by this phenomenon is avoided, and the searchability of the stored database is optimized.
[0078] Specifically, after determining the characterization capability parameters of the feature words for the storage subject, the following steps are also included:
[0079] Semantic analysis is performed on the characteristic words of the storage subject, and the representation ability parameters of several characteristic words with the same semantics are determined as the one with the highest value among the representation ability parameters of the characteristic words with the same semantics. Optionally, NLP (natural language processing) technology is used for semantic analysis. Semantic analysis and redetermination of the representation ability parameters further improve the retrieval effect.
[0080] See also Figure 2 As shown, this embodiment further provides an indexing method for searching a database obtained by the data storage method supporting retrieval in the above solution, including:
[0081] Step S01, determining a storage entity whose characteristic words match the search content;
[0082] Step S02, reading the corresponding index information table, and determining the display order of the index information for the storage subject according to the representation capability parameters of the feature words for the storage subject.
[0083] Specifically, determining the display order of the index information for the storage subject according to the representation capability parameter of the feature word for the storage subject includes:
[0084] Determine the feature words that match each storage entity with the search content;
[0085] The order of displaying the index information for each storage subject is determined by the sum of the representation capability parameters of the feature words matching the search content from large to small.
[0086] For example, the search content is "Internet data".
[0087] Storage entity A: feature word “Internet”, characterization capability parameter: 0.18.
[0088] Storage entity B: feature word “data”, characterization capability parameter: 0.22.
[0089] Storage entity C: feature word “Internet”, characterization capability parameter: 0.18; feature word “data”, characterization capability parameter: 0.22.
[0090] Storage entity A: The sum of the characterization capability parameters = 0.18.
[0091] Storage entity B: The sum of the characterization capability parameters = 0.22.
[0092] Storage entity C: the sum of the characterization capability parameters = 0.18 + 0.22 = 0.40.
[0093] In the index information it is shown as:
[0094] 1. Storage entity C (0.40);
[0095] 2. Storage entity B (0.22);
[0096] 3. Storage entity A (0.18).
[0097] See also Figure 3 As shown, this embodiment further provides a data storage system, which applies the above-mentioned data storage method supporting retrieval to perform data storage, including:
[0098] A data storage module, used for storing the storage of the subject;
[0099] An extraction module, connected to the data storage module, is used to extract the characteristic words of the storage subject and count the number of occurrences and occurrence nodes of the characteristic words;
[0100] A calculation module, connected to the extraction module, for calculating the continuous feature value and representation capability parameter of the feature word;
[0101] The index support module is connected to the extraction module and the calculation module respectively, and is used to generate and store an index data table including the characteristic words of the storage subject and the characterization capability parameters of the characteristic words of the storage subject.
[0102] As a preferred technical solution for the data storage system, the computing module is provided with a semantic analysis unit, which is used to determine semantically identical feature words and refresh the representation capability parameters of the semantically identical feature words, that is, to determine the representation capability parameters of several semantically identical feature words as the one with the highest value among the representation capability parameters of the semantically identical feature words.
[0103] The modules involved in the embodiments described in the present application may be implemented by software or hardware. The modules described may also be set in a processor, for example, it may be described as: a processor includes a data storage module, an extraction module, a calculation module, and an index support module. The names of these modules do not, in some cases, constitute limitations on the modules themselves.
[0104] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the device, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based device that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0105] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.
Claims
1. A method for storing data that supports retrieval, It is characterized in that include: Compare the content of a single storage subject with a stop word list to determine a number of characteristic words in the single storage subject; Determine the representation capability parameters of each feature word for the storage subject; Generate the characteristic words of the storage subject and the characterization capability parameters of the characteristic words into an index information table and store it together with the storage subject; The characteristic words are words that appear in the storage body in the top preset number of rankings, excluding the words included in the stop word list; The characterization capability parameter is determined according to the continuous feature value and the number of occurrences of the feature word, and the characterization capability parameter is positively correlated with the continuous feature value and the number of occurrences respectively; Among them, for the storage body of text type, the continuous feature value is the ratio of the maximum number of continuous fields where the feature word appears continuously in a number of consecutive fields with the same interval to the total number of fields contained in the storage body; For a video or audio storage body, the continuous feature value is the ratio of the maximum number of consecutive time periods in which the feature word appears continuously in a number of consecutive time periods with the same interval to the total number of time periods contained in the storage body.
2. The data storage method supporting retrieval according to claim 1, It is characterized in that In the step of determining the characterization capability parameter of each feature word for the storage subject, the product of the continuous feature value of the feature word and the number of occurrences of the feature word is determined as the characterization capability parameter of the feature word for the storage subject.
3. The data storage method supporting retrieval according to claim 2, It is characterized in that After determining the characterization capability parameter of the feature word for the storage subject, the method further includes: Semantic analysis is performed on the feature words of the storage subject, and the representation ability parameters of several feature words with the same semantics are all determined as the one with the highest value among the representation ability parameters of the feature words with the same semantics.
4. The data storage method supporting retrieval according to claim 3, It is characterized in that In the case of excluding the words included in the stop word list, for determining the characteristic words in the storage body, in response to the storage body category being text, the following steps are performed: Count and store words that appear more than once in the subject; The number of occurrences of each word is compared and counted, and the words ranked before a preset number of positions are determined as the characteristic words of the storage subject.
5. The data storage method supporting retrieval according to claim 4, It is characterized in that In the case of excluding the words included in the stop word list, for determining the characteristic words in the storage body, in response to the storage body category being audio, the following steps are performed: Perform audio and text recognition on the storage subject; Count and store words that appear more than once in the subject; The number of occurrences of each word is compared and counted, and the words ranked before a preset number of positions are determined as the characteristic words of the storage subject.
6. The data storage method supporting retrieval according to claim 5, It is characterized in that In the case of excluding the words included in the stop word list, for determining the characteristic words in the storage body, the following steps are performed in response to the storage body category being a video; Performing audio text recognition and image text recognition on the storage subject respectively; Count and store words that appear more than once in the subject; The number of occurrences of each word is compared and counted, and the words ranked before a preset number of positions are determined as the characteristic words of the storage subject.
7. An indexing method for searching a database obtained by the search-supported data storage method according to any one of claims 1 to 6, It is characterized in that include: Determine the storage subject whose characteristic words match the search content; The corresponding index information table is read, and the display order of the index information for the storage subject is determined according to the characterization capability parameters of the feature words for the storage subject.
8. The indexing method according to claim 7, It is characterized in that Determining the display order of index information for the storage subject according to the representation capability parameter of the feature word for the storage subject includes: Determine the feature words that match each storage entity with the search content; The order of displaying the index information for each storage entity is determined by the sum of the representation capability parameters of the feature words matching the search content from large to small.
9. A data storage system, using the data storage method supporting retrieval according to any one of claims 1 to 6 to perform data storage, It is characterized in that include: A data storage module, used for storing the storage of the subject; An extraction module, connected to the data storage module, for extracting the characteristic words of the storage subject and counting the number of occurrences and occurrence nodes of the characteristic words; A calculation module, connected to the extraction module, for calculating the continuous feature value and the characterization capability parameter of the feature word; The index support module is connected to the extraction module and the calculation module respectively, and is used to generate and store an index data table including the characteristic words of the storage subject and the characterization capability parameters of the characteristic words of the storage subject.
10. The data storage system according to claim 9, It is characterized in that The calculation module is provided with a semantic analysis unit, and the semantic analysis unit is used to determine feature words with the same semantics and to update the representation capability parameters of the feature words with the same semantics.
Citation Information
Patent Citations
Keyword based evaluation expert intelligent search and recommendation method
CN103605665A
Data index constructing method, retrieving method and device, medium and electronic equipment
CN108255985A
News video content keyword extraction method, terminal device and medium
CN111814770A
Knowledge graph construction method, knowledge graph construction device, medium and electronic equipment
CN116304104A
Information classification system and program
JP2005085112A
