A data classification storage method and device and a computer storage medium

Abstract

The application provides a data classified storage method and device and a computer storage medium. At least one piece of data is obtained by performing crawling processing on a preset information source. The at least one piece of data is classified and processed to determine the respective category identifiers of the at least one piece of data. The respective storage locations of the at least one piece of data are determined based on the respective category identifiers of the at least one piece of data. The at least one piece of data is stored in the respective storage locations. In this way, the crawled data is classified and stored according to the respective categories, so that data storage disorder can be avoided, and the subsequent data query efficiency can be effectively improved.

Description

Technical Field

[0001] This application relates to the field of computer software technology, and in particular to a data classification and storage method, apparatus and computer storage medium. Background Technology

[0002] Data crawling is a common method for acquiring information. For example, in public opinion analysis platforms, web crawlers are often used to acquire data, and then the big data is analyzed to obtain the trend and development of public opinion. However, during the data crawling process, the data is stored in a disordered manner, making subsequent searching and analysis cumbersome and resulting in unnecessary waste of resources and performance. Summary of the Invention

[0003] This application provides a data classification and storage method, apparatus, and computer storage medium, which can classify and store crawled data according to their respective categories, thereby avoiding data storage chaos.

[0004] The technical solution of this application is implemented as follows:

[0005] Firstly, this application provides a data classification and storage method, which includes:

[0006] The preset information source is crawled and processed to obtain at least one piece of data;

[0007] The at least one piece of data is classified to determine the category identifier of each of the at least one piece of data.

[0008] Based on the category identifier of each of the at least one piece of data, the storage location of each of the at least one piece of data is determined;

[0009] The at least one data item is stored in its respective storage location.

[0010] Secondly, this application provides a data classification and storage device, which includes an acquisition unit, a classification unit, a determination unit, and a storage unit, wherein...

[0011] The acquisition unit is configured to crawl and process a preset information source to obtain at least one piece of data;

[0012] A classification unit is configured to classify the at least one piece of data and determine the category identifier of each of the at least one piece of data.

[0013] The determining unit is configured to determine the storage location of each of the at least one piece of data based on the category identifier of each of the at least one piece of data.

[0014] The storage unit is configured to store the at least one data item in its respective storage location.

[0015] Thirdly, this application provides a data classification storage device, which includes a memory and a processor; wherein,

[0016] The memory is used to store computer programs that can run on the processor;

[0017] The processor is configured to perform the steps of the method as described in the first aspect when running the computer program.

[0018] Fourthly, this application provides a computer storage medium storing a data classification storage program that, when executed by at least one processor, implements the steps of the method described in the first aspect.

[0019] This application provides a data classification and storage method, apparatus, and computer storage medium. The method involves crawling data from a preset information source to obtain at least one piece of data; classifying the at least one piece of data to determine its respective category identifier; determining the storage location of each piece of data based on its category identifier; and storing the at least one piece of data in its respective storage location. By classifying and storing the crawled data according to its categories, data storage chaos can be avoided, and subsequent data retrieval efficiency can be effectively improved. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating a data classification and storage method provided in an embodiment of this application.

[0021] Figure 2 This is a flowchart illustrating another data classification and storage method provided in an embodiment of this application.

[0022] Figure 3 This is a schematic diagram illustrating the principle of another data classification and storage method provided in this application embodiment.

[0023] Figure 4 This is a schematic diagram of the composition structure of a data classification and storage device provided in an embodiment of this application.

[0024] Figure 5 This is a schematic diagram of the specific hardware structure of a data classification and storage device provided in an embodiment of this application.

[0025] Figure 6 This is a schematic diagram of the composition structure of a data classification and storage system provided in an embodiment of this application. Detailed Implementation

[0026] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It is understood that the specific embodiments described herein are merely for explaining the relevant application and not for limiting the application. Furthermore, it should be noted that, for ease of description, only the parts relevant to the application are shown in the accompanying drawings.

[0027] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. The terminology used herein is for the purpose of describing embodiments of this disclosure only and is not intended to be limiting of this disclosure.

[0028] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0029] It should be noted that the terms "first, second, third" used in the embodiments of this disclosure are only used to distinguish similar objects and do not represent a specific order of objects. It is understood that "first, second, third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of this disclosure described herein can be implemented in an order other than that illustrated or described herein.

[0030] Data crawling is a common method for acquiring information. During the crawling process, data is typically stored haphazardly across various distributed storage systems; or, it may be stored according to the time it was crawled, such as storing data crawled on a specific day in a particular storage location. This means that if different types of data are needed for processing later, all data must be traversed to find the correct information, making it impossible to accurately locate and retrieve the data, resulting in unnecessary waste of resources and performance.

[0031] Based on this, embodiments of this application provide a data classification and storage method. This method involves crawling data from a preset information source to obtain at least one piece of data; classifying the at least one piece of data to determine its respective category identifier; determining the storage location of each piece of data based on its category identifier; and storing the at least one piece of data in its respective storage location. By classifying and storing the crawled data according to its categories, data storage chaos can be avoided, and subsequent data retrieval efficiency can be effectively improved.

[0032] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0033] In one embodiment of this application, see Figure 1It illustrates a flowchart of a data classification and storage method provided in an embodiment of this application, such as... Figure 1 As shown, the method may include:

[0034] S101: Crawling and processing the preset information source to obtain at least one piece of data.

[0035] It should be noted that the data classification and storage method provided in this application embodiment can be applied to various computing devices, such as personal computers, laptops, cloud computing systems, etc.

[0036] Here, the crawling process can be performed using a web crawling tool such as Scripy.

[0037] S102: Classify at least one data point and determine the category identifier for each data point.

[0038] It should be noted that since the crawled data may contain various types of data, it can be categorized for easier searching and analysis later.

[0039] When performing classification processing, the specific classification criteria can be determined based on the actual application scenario. For example, data can be divided into categories such as images, text, and sound; or data can be classified according to topic relevance, such as strong correlation, moderate correlation, weak correlation, etc.

[0040] S103: Determine the storage location of at least one piece of data based on the category identifier of each piece of data.

[0041] S104: Store at least one data record to its respective storage location.

[0042] It's important to note that data is allocated to different storage locations based on its category identifier. This categorization and storage of the crawled data prevents data clutter and effectively improves subsequent data retrieval efficiency.

[0043] It should be noted that, in order to further subdivide the data, more detailed attribute analysis can be performed after classifying the data. The following example uses any one of the at least one data point to illustrate this.

[0044] In some embodiments, see Figure 2 The step of determining the storage location of each of the at least one data item based on its respective category identifier may include:

[0045] S201: Determine the storage range of the first data based on the category identifier of the first data.

[0046] Here, the first data point is any one of at least one data point;

[0047] S202: Perform attribute analysis on the first data to obtain the attribute classification results.

[0048] S203: Based on the attribute classification results, determine the storage location of the first data from the storage range of the first data.

[0049] The following detailed explanation uses the example of category identifiers including images, text, and sound. Adaptation is also possible if other category identifier systems are used.

[0050] In some embodiments, such as Figure 3 As shown, determining the storage range of the first data based on the category identifier of the first data includes:

[0051] When the category identifier of the first data is image, the storage range of the first data is determined to be a first preset storage range, which can be represented as [A:B]; when the category identifier of the first data is text, the storage range of the first data is determined to be a second preset storage range, which can be represented as [C:D]; when the category identifier of the first data is sound, the storage range of the first data is determined to be a third preset storage range, which can be represented as [E:F].

[0052] Based on this, the content of attribute analysis differs for different categories of data, and can be determined according to the actual application scenario. For example, if the first data is an image, the attribute analysis includes: determining whether it is a pure image or an image containing text, and / or determining whether it is a blurred image or a clear image; if the first data is text, the attribute analysis includes: determining its temperature (hot or cold) value; if the first data is sound, the attribute analysis includes: determining its duration.

[0053] In one specific embodiment, when the category of the first data is identified as an image, the attribute classification result is used to indicate whether the first data is a pure image and / or whether the first data is a clear image. Therefore, the attribute analysis of the first data to obtain the attribute classification result may include:

[0054] The first data is subjected to image recognition processing to obtain a first recognition result; wherein the first recognition result indicates that the first data is a pure image or an image containing text; and / or, the first data is subjected to clarity judgment to obtain a second recognition result; wherein the second recognition result indicates that the image corresponding to the first data is a clear image or a blurry image.

[0055] The following are two specific scenarios.

[0056] In the first scenario, the attribute classification results can include: (1) pure images, and (2) images containing text. In this case, only image recognition processing is needed for the first data. Here, image recognition processing can be performed using existing image recognition algorithms, which will not be elaborated upon here.

[0057] Thus, if the first data is a pure image, then the storage location of the first data is sk1 in [A:B]. 图 [per], if the first data is an image containing text, then the storage location of the first data is sk1 in [A:B]. 图 [wd].

[0058] The second scenario, such as Figure 3 As shown, the attribute classification results can include: (1) a pure image and a clear image, (2) a pure image and a blurry image, (3) an image containing text and a clear image, and (4) an image containing text and a blurry image. At this time, image recognition processing and clarity judgment need to be performed on the first data respectively.

[0059] Here, the methods for judging sharpness can be different for plain images and images containing text.

[0060] When the first data is pure image data, the step of determining the sharpness of the first data to obtain the second recognition result may include:

[0061] The image corresponding to the first data is processed based on the spatial domain image quality assessment algorithm MSCN to obtain an MSCN coefficient map. In the MSCN coefficient map, the sum of the MSCN coefficients that meet the first preset condition is calculated to obtain the target statistical value. The target statistical value is compared with the first preset value to obtain the second recognition result based on the comparison result.

[0062] In other words, when the first data is a pure image, the specific steps for sharpness determination may include: preprocessing the image to be identified (corresponding to the first data) to obtain an MSCN (Mean Subtracted Contrast Normalized) image; then statistically analyzing the MSCN coefficients {mscn} in the corresponding bar chart of the MSCN image within the range of 0.1-0.5; summing the above coefficients to obtain a first measurement value sum{mscn}; comparing the first measurement value sum{mscn} with a first preset threshold; if the first measurement value sum{mscn} is greater than the first preset threshold, then the first data is a blurry image; if the first measurement value sum{mscn} is less than (or equal to) the first preset threshold, then the first data is a sharp image.

[0063] When the first data is image data containing text, the step of determining the clarity of the first data to obtain the second recognition result may include:

[0064] The image corresponding to the first data is calculated, normalized, and quantized to obtain a gradient magnitude map. In the gradient magnitude map, the number of gradient magnitude values ​​that meet the second preset condition is counted to obtain the target number. The target number is compared with the second preset value to obtain the second recognition result based on the comparison result.

[0065] When the first data is an image containing text, the specific steps for determining sharpness may include: calculating the gradient magnitude of the image to be identified (corresponding to the first data), normalizing the gradient magnitude to a range of 0-1; quantizing the gradient magnitude and controlling it within the range of 0-0.5; counting the number of non-zero points in the range of [0~rate (ratio constant)] / gap in the bar chart of the quantized gradient magnitude to obtain a second measurement value, where gap refers to the width of the bars in the bar chart; comparing the second measurement value with a second preset threshold, if the second measurement value is greater than the second preset threshold, then the first data is a blurry image; if the second measurement value is less than (or equal to) the second preset threshold, then the first data is a sharp image.

[0066] Thus, as Figure 3 As shown, if the first data is a pure image and a blurred image, then the storage location of the first data is sk1 in [A:B]. 图 [per, blurry]; If the first data is a pure image and a clear image, then the storage location of the first data is sk1 in [A:B]. 图 [per, clear]; If the first data is a blurry image containing text, then the storage location of the first data is sk1 in [A:B]. 图 [wd, blurry]; If the first data is a clear image containing text, then the storage location of the first data is sk1 in [A:B]. 图 [wd, clear].

[0067] In some embodiments, when the category identifier of the first data is text, the attribute classification result includes at least general public opinion and trending public opinion; the attribute analysis of the first data to obtain the attribute classification result may include:

[0068] Sentiment analysis is performed on the first data to obtain its confidence value; the exposure value and popularity value of the first data are determined; the confidence value, exposure value, and popularity value are calculated to obtain the hotness / coldness value of the first data; if the hotness / coldness value is less than a third preset value, the first data is determined to be ordinary public opinion; if the hotness / coldness value is greater than or equal to the third preset value, the first data is determined to be hot public opinion.

[0069] It should be noted that, for text, the temperature rating of the first data needs to be further determined based on the temperature rating of the first data in SK2. 字 The specific locations for cold and hot storage.

[0070] In this embodiment, the confidence value, exposure value, and heat value of the first data are used to calculate the hotness / coldness value L(ψ) of the first data.

[0071] For example, equation (1) can be used to calculate the temperature value L(ψ) of the first data.

[0072]

[0073] Where EM(ψ) is the confidence value of the first data ψ. Let ψ be the exposure value of the first data ψ at time t+k, where t is the time when the first data ψ was captured, and k is the time taken to classify and label the first data ψ. This represents the popularity value of the first data point.

[0074] Here, the confidence value EM(ψ) of the first data can be obtained using a text sentiment analysis model in existing Natural Language Processing (NLP) techniques.

[0075] For the exposure value of the first data The exposure value can be calculated based on the number of posts and exposure of similar public opinion (hereinafter referred to as identical public opinion). For example, obtain the identical public opinion that has existed in the last hour and obtain its publisher information and posting website information. Where t is the time when the first data is captured (assuming that the time from the publication of a public opinion to its capture is fixed). Here, equation (2) can be used to calculate the exposure value of the first data.

[0076]

[0077] in, Used to calculate the exposure value of the first data point on self-media websites. This is used to calculate the exposure value of the first data point on a regular website. Here, the calculations are performed separately for self-media websites and regular websites, and then summed to obtain the exposure value of the first data point.

[0078] Specifically, self-media websites calculate the exposure value of primary data based on the publisher's follower count, where μ i For the exposure of the i-th self-media website, fan(u) ijLet ν be the number of followers of the j-th publisher on the i-th self-media website. i∈n, j∈m, where n is the total number of self-media websites and m is the total number of publishers on the corresponding self-media website. For ordinary websites, the exposure value of the first data point is calculated based on daily active users, where ν... i For the exposure of the i-th ordinary website, du i Let be the total number of daily active users of the i-th ordinary website.

[0079] Here, the exposure intensity of self-media websites and ordinary websites is taken into account through the above calculations, which improves the accuracy of the exposure value calculation of the first data.

[0080] The popularity value of the first data It is represented as the popularity value of the first data point at time t+k. It can be calculated based on the number of comments, reposts, likes, and the influence of the corresponding individuals.

[0081] For example, equation (3) can be used to calculate the heat value of the first data.

[0082]

[0083] Where cmt, fwd, and thumb represent the total number of comments, total number of reposts, and total number of likes for the first data point, respectively. v fwd v and thumb v The numbers represent the number of comments, reposts, and likes for users with more than N followers, respectively, with w1, w2, w3, w1', w2', and w3' being the corresponding weights.

[0084] It should be understood that the influence of the corresponding personnel is reflected in the fact that only users with more than N followers are included in the calculation, that is, only the data (number of comments, number of reposts, number of likes) of users with greater influence are included in the calculation.

[0085] Thus, based on equations (1) to (3), the popularity / coldness value of the first data can be calculated. If the popularity / coldness value is less than the third preset value, the first data is determined to be ordinary public opinion; if the popularity / coldness value is greater than or equal to the third preset value, the first data is determined to be hot public opinion. Furthermore, as... Figure 3 As shown, if the first data is general public opinion, then the storage location of the first data is sk2 in [C:D]. 字 [Cold]; If the first data is a hot topic, then the first data is stored in sk2 in [C:D]. 字 [hot].

[0086] In some embodiments, when the category identifier of the first data is sound, the attribute classification result includes at least short-term data, medium-term data, and long-term data; the attribute analysis of the first data to obtain the attribute classification result includes:

[0087] Determine the duration of the first data; if the duration of the first data falls within a first preset time range, determine that the first data is invalid data and discard the invalid data; if the duration of the first data falls within a second preset time range, determine that the first data is short-term data; if the duration of the first data falls within a third preset time range, determine that the first data is medium-term data; if the duration of the first data falls within a fourth preset time range, determine that the first data is long-term data.

[0088] It should be noted that if the category identifier of the first data is sound, then the storage location of the first data is sk3 in [E:F]. 音 .

[0089] In one scenario, if the duration of the first data is less than or equal to *a* seconds, falling within a first preset time range, the first data is determined to be invalid and discarded. If the duration of the first data is greater than *a* seconds but less than or equal to *b* minutes, falling within a second preset time range, the first data is determined to be short-term data, and its storage location is sk3 in [E:F]. 音 [Short]; If the duration of the first data is greater than b minutes and less than or equal to c minutes, which falls within the third preset time range, then the first data is determined to be medium-time data, and its storage location is sk3 in [E:F]. 音 [Middle]; If the duration of the first data is greater than c minutes, that is, it falls within the fourth preset time range, then the first data is determined to be long-term data, and the storage location of the first data is sk3 in [E:F]. 音 [Length]. In another scenario, if the duration of the first data is less than *a* seconds, falling within the first preset time range, the first data is determined to be invalid and discarded. If the duration of the first data is greater than or equal to *a* seconds but less than *b* minutes, falling within the second preset time range, the first data is determined to be short-term data. In this case, the storage location of the first data is sk3 in [E:F]. 音 [Short]; If the duration of the first data is greater than or equal to b minutes but less than c minutes, which falls within the third preset time range, then the first data is determined to be medium-time data, and its storage location is sk3 in [E:F]. 音 [Middle]; If the duration of the first data is greater than or equal to c minutes, that is, it falls within the fourth preset time range, then the first data is determined to be long-term data, and the storage location of the first data is sk3 in [E:F]. 音 [long].

[0090] Here, a < b < c, and a, b, and c need to be determined according to the actual application scenario. This disclosed embodiment does not impose specific limitations.

[0091] In summary, in this embodiment, for the crawled first data, the first data is first classified to determine its storage range. Then, attribute analysis is performed on the first data under different categories, and different storage locations are further allocated within the corresponding storage range. If the first data is categorized as an image, existing image recognition technology is used to determine whether the first data is a pure image or an image containing text. Based on this, it is further determined whether the first data is a blurred image, thus determining the specific storage location of the first data. If the first data is categorized as text, a text sentiment analysis model in NLP technology is used to obtain the confidence value of the first data. The exposure value of the first data is calculated based on the number of posts and exposure of the same sentiment. The popularity value of the first data is calculated based on the number of comments, reposts, likes, and the influence of the corresponding individuals. The confidence value, exposure value, and popularity value of the first data are calculated to obtain the "hotness / coldness" value of the first data, thus determining the specific storage location of the first data. If the first data is categorized as audio, the duration of the first data is used to determine whether it is short-term, medium-term, or long-term data, thus determining the specific storage location of the first data.

[0092] In some embodiments, after determining the storage location of at least one piece of data, the method may further include:

[0093] Based on the storage range, attribute classification results, and storage location, at least one data storage label is generated for each data item.

[0094] It should be noted that if the category identifier of the first data is image, then the storage location label of the first data is {[A:B], sk1 图 ∈[per / wd, clear / blurred]}; if the category identifier of the first data is text, then the storage location label of the first data is {[C:D], sk2 字 ∈[Cold / Hot]}; If the category identifier of the first data is sound, then the storage location label of the first data is {[E:F], sk3 音 ∈[short / medium / long]}.

[0095] In some embodiments, the method further includes:

[0096] Statistical analysis is performed on the storage locations corresponding to at least one piece of data to obtain data storage information; the data storage information is then visualized.

[0097] In this way, by visually presenting the data storage information to technical personnel, they can more clearly and intuitively understand the storage location of each data item, facilitating subsequent retrieval and searching.

[0098] As can be seen from the above, in this embodiment of the application, images, sounds, and text are stored according to certain storage allocation rules. Among them, the images take into account the factors of the image content itself and divide the images according to whether they contain text and their clarity. The text information takes into account the popularity of the data, storing more popular text information in one location and less popular text information in another location, which is convenient for subsequent public opinion analysis.

[0099] This application provides a data classification and storage method. By storing images, sounds, and text according to certain storage allocation rules, it can effectively solve the problem of confusion in subsequent data extraction and analysis caused by non-standard data storage, and can also effectively improve the query efficiency of public opinion product information.

[0100] In another embodiment of this application, see [link to application]. Figure 4 This illustrates a schematic diagram of the composition of a data classification and storage device 40 provided in an embodiment of this application. For example... Figure 4 As shown, the data classification and storage device 40 includes an acquisition unit 401, a classification unit 402, a determination unit 403, and a storage unit 404, wherein...

[0101] The acquisition unit 401 is configured to crawl and process data from a preset information source to obtain at least one piece of data.

[0102] Classification unit 402 is configured to classify at least one piece of data and determine the category identifier of each of the at least one piece of data.

[0103] The determination unit 403 is configured to determine the storage location of at least one piece of data based on the category identifier of each piece of data.

[0104] Storage unit 404 is configured to store at least one piece of data to its respective storage location.

[0105] In some embodiments, the determining unit 403 is specifically configured to determine the storage range of the first data based on the category identifier of the first data; wherein the first data is any one of at least one data; perform attribute analysis on the first data to obtain attribute classification results; and determine the storage location of the first data from the storage range of the first data based on the attribute classification results.

[0106] In some embodiments, the determining unit 403 is further configured to determine the storage range of the first data as a first preset storage range when the category identifier of the first data is an image; to determine the storage range of the first data as a second preset storage range when the category identifier of the first data is text; and to determine the storage range of the first data as a third preset storage range when the category identifier of the first data is sound.

[0107] In some embodiments, the determining unit 403 is further configured to perform attribute analysis on the first data to obtain attribute classification results, including: performing image recognition processing on the first data to obtain a first recognition result; wherein the first recognition result indicates that the first data is a pure image or an image containing text; and / or, performing clarity judgment on the first data to obtain a second recognition result; wherein the second recognition result indicates that the image corresponding to the first data is a clear image or a blurry image.

[0108] In some embodiments, the determining unit 403 is further configured to, when the first data is pure image data, process the image corresponding to the first data based on the Spatial Domain Image Quality Assessment Algorithm (MSCN) to obtain an MSCN coefficient map; in the MSCN coefficient map, count the sum of the MSCN coefficients that satisfy the first preset condition to obtain a target statistical value; compare the target statistical value with the first preset value to obtain a second recognition result based on the comparison result; or, when the first data is image data containing text, perform calculation, normalization, and quantization processing on the image corresponding to the first data to obtain a gradient magnitude map; in the gradient magnitude map, count the number of gradient magnitude values ​​that satisfy the second preset condition to obtain a target number; compare the target number with the second preset value to obtain a second recognition result based on the comparison result.

[0109] In some embodiments, the determining unit 403 is further configured to perform attribute analysis on the first data to obtain attribute classification results, including: performing sentiment analysis on the first data to obtain a confidence value of the first data; determining an exposure value and a popularity value of the first data; calculating the confidence value, exposure value, and popularity value of the first data to obtain a hot / cold value of the first data; determining the first data as ordinary public opinion when the hot / cold value is less than a third preset value; and determining the first data as hot / cold public opinion when the hot / cold value is greater than or equal to the third preset value.

[0110] In some embodiments, the determining unit 403 is further configured to perform attribute analysis on the first data to obtain attribute classification results, including: determining the duration of the first data; if the duration of the first data falls within a first preset time range, determining the first data as invalid data and discarding the invalid data; if the duration of the first data falls within a second preset time range, determining the first data as short-term data; if the duration of the first data falls within a third time range, determining the first data as medium-term data; and if the duration of the first data falls within a fourth preset time range, determining the first data as long-term data.

[0111] Understandably, in this embodiment, a "unit" can be a portion of a circuit, a portion of a processor, a portion of a program or software, etc., and can also be a module or a non-modular component. Furthermore, the components in this embodiment can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional module.

[0112] If the integrated unit is implemented as a software functional module and not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the method described in this embodiment. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0113] Therefore, this embodiment provides a computer storage medium storing a data classification storage program, which, when executed by at least one processor, implements the steps of the method described in any of the foregoing embodiments.

[0114] Based on the above-described composition of a data classification storage device 40 and a computer storage medium, see [link to relevant documentation]. Figure 5 It illustrates a specific hardware structure example of a data classification and storage device 40 provided in an embodiment of this application, such as... Figure 5As shown, the data classification storage device 40 may include: a communication interface 601, a memory 602, and a processor 603; the various components are coupled together via a bus device 604. It is understood that the bus device 604 is used to implement communication between these components. In addition to a data bus, the bus device 604 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 5 The general labels all buses as bus devices 604.

[0115] The communication interface 601 is used for receiving and sending signals during the process of sending and receiving information with other external network elements;

[0116] Memory 602 is used to store computer programs that can run on processor 603;

[0117] Processor 603, when running the computer program, performs the following:

[0118] The preset information source is crawled and processed to obtain at least one piece of data;

[0119] Classify at least one piece of data and determine the category identifier for each of the at least one piece of data.

[0120] Based on the category identifier of each of the at least one piece of data, determine the storage location of each of the at least one piece of data.

[0121] Store at least one piece of data in its respective storage location.

[0122] It is understood that the memory 602 in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The memory 602 of the apparatus and method described in this application is intended to include, but is not limited to, these and any other suitable types of memory.

[0123] The processor 603 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 603 or by instructions in software form. The processor 603 can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 602. Processor 603 reads the information in memory 602 and, in conjunction with its hardware, completes the steps of the above method.

[0124] It is understood that the embodiments described in this application can be implemented using hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in this application, or combinations thereof.

[0125] For software implementation, the techniques described in this application can be implemented through modules (e.g., procedures, functions, etc.) that perform the functions described in this application. The software code can be stored in memory and executed by a processor. The memory can be implemented in the processor or externally.

[0126] Alternatively, as another embodiment, the processor 603 is further configured to perform the steps of the method described in any of the foregoing embodiments when running the computer program.

[0127] Based on the above-described composition and hardware structure diagram of the data classification storage device 40, see [link to diagram]. Figure 6 This illustrates a schematic diagram of the composition structure of a data classification and storage system 70 provided in an embodiment of this application. For example... Figure 6 As shown, the data classification storage system 70 includes at least the data classification storage device 40 described in any of the foregoing embodiments.

[0128] For the data classification and storage system 70, by classifying and storing the crawled data according to their respective categories, data storage chaos can be avoided, and subsequent data query efficiency can be effectively improved.

[0129] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application.

[0130] It should be noted that, in this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0131] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0132] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0133] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0134] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0135] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data classification and storage method, characterized in that, The method includes: The preset information source is crawled and processed to obtain at least one piece of data; The at least one piece of data is classified to determine the category identifier for each of the at least one piece of data; the category identifier includes text. Based on the category identifier of the first data, the storage range of the first data is determined; wherein, the first data is any one of the at least one data item; Perform attribute analysis on the first data to obtain attribute classification results; Based on the attribute classification results, the storage location of the first data is determined from the storage range of the first data; Store the at least one piece of data to its respective storage location; Wherein, when the category identifier of the first data is text, the attribute classification result includes at least general public opinion and hot public opinion; The attribute analysis of the first data to obtain attribute classification results includes: Sentiment analysis is performed on the first data to obtain the confidence value of the first data; the confidence value of the first data is obtained using a text sentiment analysis model; Determine the exposure value and popularity value of the first data; the popularity value of the first data is based on Obtain; among them, , and These represent the total number of comments, total number of reposts, and total number of likes for the first data, respectively. , and These represent the number of comments, reposts, and likes for users with more than N followers, in that order. , , , , , These are the corresponding weights; based on The temperature value of the first data was calculated; where, Indicates the first data The confidence level value, Indicates the first data The exposure value at time t+k, where t represents the capture of the first data. The time, k represents the first data The time required to classify and label categories This represents the popularity value of the first data; If the temperature value is less than a third preset value, the first data is determined to be ordinary public opinion. If the hotness / coldness value is greater than or equal to a third preset value, the first data is determined to be a hot topic of public opinion.

2. A data classification and storage device, characterized in that, The data classification and storage device includes an acquisition unit, a classification unit, a determination unit, and a storage unit, wherein, The acquisition unit is configured to crawl and process a preset information source to obtain at least one piece of data; The classification unit is configured to classify the at least one piece of data and determine the category identifier for each of the at least one piece of data; the category identifier includes text. The determining unit is configured to determine the storage range of the first data based on the category identifier of the first data; wherein the first data is any one of the at least one data; perform attribute analysis on the first data to obtain attribute classification results; determine the storage location of the first data from the storage range of the first data based on the attribute classification results; when the category identifier of the first data is text, the attribute classification results include at least ordinary public opinion and hot public opinion. The storage unit is configured to store the at least one piece of data to its respective storage location; The determining unit is further configured to perform attribute analysis on the first data to obtain attribute classification results, including: performing sentiment analysis on the first data to obtain a confidence value of the first data; obtaining the confidence value of the first data using a text sentiment analysis model; determining the exposure value and popularity value of the first data; the popularity value of the first data is based on... Obtain; among them, , and These represent the total number of comments, total number of reposts, and total number of likes for the first data, respectively. , and These represent the number of comments, reposts, and likes for users with more than N followers, in that order. , , , , , These are the corresponding weights; based on The temperature value of the first data was calculated; where, Indicates the first data The confidence level value, Indicates the first data The exposure value at time t+k, where t represents the capture of the first data. The time, k represents the first data The time required to classify and label categories The first data represents the popularity value; if the popularity value is less than a third preset value, the first data is determined to be ordinary public opinion; if the popularity value is greater than or equal to the third preset value, the first data is determined to be hot public opinion.

3. A data classification and storage device, characterized in that, The data classification storage device includes a memory and a processor; wherein the memory is used to store a computer program that can run on the processor; and the processor is used to execute the steps of the method as described in claim 1 when running the computer program.

4. A computer storage medium, characterized in that, The computer storage medium stores a data classification storage program, which, when executed by at least one processor, implements the steps of the method as described in claim 1.

Images