Data processing method, data processing device and storage medium

Abstract

The present application discloses a data processing method, a data processing device and a storage medium, and relates to the field of data processing technology. The data processing method includes: constructing a resource library, and crawling data to be updated based on the resource category or resource information of the resource library; when the update process of the resource library is triggered, obtaining the vocabulary information to be updated of the resource library; training the pre-processed vocabulary information to be updated based on a deep learning model to obtain a training result; updating and storing the training result in a graph database based on preset semantic logic rules, wherein the graph database includes a graph structure of all data in the resource library. When the data of the constructed resource library changes, the changed data is updated and stored in the graph database in real time through the deep learning model, thereby avoiding the occurrence of processing results based on outdated information when the artificial intelligence model performs tasks based on the information of the graph database, and improving the reliability of the system and the accuracy of task processing.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, a data processing device and a storage medium. Background Art

[0002] In existing systems, artificial intelligence algorithm models usually rely on regular manual research and updates to adapt to changes in laws and regulations. Due to the frequent and complex changes in laws and regulations, manual updates often cannot be processed in real time or in a timely manner, resulting in artificial intelligence systems relying on outdated information when handling legal-related matters or conducting data analysis on legal-related matters, resulting in low system reliability.

[0003] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0004] The main purpose of this application is to provide a data processing method, a data processing device and a storage medium, aiming to solve the technical problem that an artificial intelligence system performs data analysis based on outdated information, resulting in low system reliability.

[0005] To achieve the above objectives, the present application proposes a data processing method, which includes:

[0006] Building a resource library, and crawling the data to be updated based on the resource category or resource information of the resource library;

[0007] When the update process of the resource library is triggered, the word library information to be updated of the resource library is obtained;

[0008] Performing training processing on the preprocessed vocabulary information to be updated based on a deep learning model to obtain a training result;

[0009] The training results are updated and stored in a graph database based on preset semantic logic rules, wherein the graph database contains a graph structure of all data in the resource library.

[0010] In one embodiment, after the step of updating and storing the training results in the graph database based on the preset semantic logic rules, the method further includes:

[0011] When the received data processing task is associated with a keyword of the vocabulary information to be updated, parsing the graph database based on the keyword;

[0012] Outputting the task result corresponding to the data processing task according to the analysis result; or

[0013] When a data structure query instruction of the graph database is received, the data structure and data relationship of the target data associated with the query instruction are output.

[0014] In one embodiment, the step of updating and storing the training results in the graph database based on preset semantic logic rules includes:

[0015] Determining the semantic definition relationship associated with the preset semantic logic rule;

[0016] Determining the logical relationship of the training result based on the semantic definition relationship, wherein the logical relationship includes at least one of a synonymous relationship, an antonymous relationship, a causal relationship, and a subordinate relationship;

[0017] The training result is updated and stored in the graph database based on the logical relationship, and the graph database includes a graph structure of nodes and edges.

[0018] In one embodiment, before the step of training the preprocessed vocabulary information to be updated based on the deep learning model to obtain the training result, the step further includes:

[0019] Performing data cleaning processing on the vocabulary information to be updated;

[0020] Based on the pre-trained BERT model and the domain vocabulary corresponding to the vocabulary information to be updated, the pre-processed vocabulary information to be updated is segmented and POS tagged to obtain key entity data;

[0021] The key entity data is semantically encoded to obtain the pre-processed vocabulary information to be updated.

[0022] In one embodiment, before the step of performing word segmentation and part-of-speech tagging processing on the pre-processed vocabulary information to be updated based on the pre-trained BERT model and the domain vocabulary corresponding to the vocabulary information to be updated to obtain key entity data, the step further includes:

[0023] Determine the information category of the vocabulary information to be updated, and obtain the corpus information corresponding to the information category;

[0024] Perform word segmentation, duplicate removal and word screening on the corpus information, and determine target words greater than a preset number among the processed words;

[0025] Writing the target word into the domain vocabulary; or

[0026] In response to a word screening or word supplementation operation, the words associated with the word screening or word supplementation operation are added to the domain vocabulary.

[0027] In one embodiment, the step of constructing a resource library and crawling the data to be updated based on the resource category or resource information of the resource library includes:

[0028] Arriving at a time point associated with a data crawling task, crawling the to-be-updated data corresponding to the resource category or resource information in the resource library;

[0029] Based on the current data of the resource library, the data to be updated is deduplicated, and the deduplicated data to be updated is added to the resource library.

[0030] In one embodiment, the update process of the resource library is triggered when at least one of the following conditions is met:

[0031] Receive resource update task;

[0032] The preset resource library update time is reached;

[0033] The total amount of new data in the resource library exceeds a preset threshold;

[0034] The resource library crawls updated information of the target data.

[0035] In one embodiment, when the update process of the resource library is triggered, the step of obtaining the to-be-updated vocabulary information of the resource library includes:

[0036] When the update process of the resource library is triggered, the data update time and data update strategy of the resource library are obtained;

[0037] Determine new data and modified data based on the data update time and the data update strategy;

[0038] The newly added data and the modified data are used as the vocabulary information to be updated.

[0039] In addition, to achieve the above objectives, the present application also proposes a data processing device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data processing method described above.

[0040] In addition, to achieve the above objectives, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the data processing method described above are implemented.

[0041] One or more technical solutions proposed in this application have at least the following technical effects:

[0042] By building a resource library, the latest data information is crawled in real time based on the resource library, and after the update process of the resource library is triggered, the latest vocabulary information to be updated in the resource library is updated to the system's graph database based on the deep learning model and preset semantic logic rules. This allows the system to perceive changes in the external environment in real time and automatically adjust the model values ​​to cope with these changes. At the same time, it can ensure that when processing related tasks, the results that meet the requirements are output based on the latest data, thereby improving the reliability of the system and the accuracy of data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0044] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0045] Figure 1 A schematic diagram of a first embodiment of the data processing method of the present application;

[0046] Figure 2 A flowchart of a second embodiment of the data processing method of the present application;

[0047] Figure 3 Schematic diagram of the device structure of the hardware operating environment involved in the data processing method in the embodiment of the present application.

[0048] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0049] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0050] Current AI algorithm models often rely on manual research and updates when dealing with issues involving laws and regulations. When external laws and regulations change, the system cannot automatically perceive and adjust its internal model, which may lead to deviations or errors when dealing with related issues. This defect not only increases the maintenance cost of the system, but may also cause legal risks and disputes. At the same time, in addition to the legal and regulatory database, case libraries and academic research papers are also important learning resources for AI algorithm models. However, existing systems often lack the ability to automatically connect these resources to the model, resulting in the model's inability to fully utilize these resources for learning and optimization. This limits the model's application capabilities and accuracy in complex scenarios.

[0051] Although user feedback is an important basis for model optimization, this method has obvious lags and subjectivity. User feedback is often generated after a problem occurs and may be affected by the user's personal bias and subjective feelings. In other words, manual updates are often not processed in real time or in a timely manner, resulting in the artificial intelligence system relying on outdated information when handling legal-related matters or conducting data analysis on legal-related matters. Therefore, relying solely on user feedback for model optimization may not be able to detect and correct potential problems in a timely manner, nor can it make full use of valuable information in external resources.

[0052] As existing systems lack the ability to automatically connect to external resources, their model optimization process is often slow and rigid. When the external environment changes (such as updates to laws and regulations, changes in market trends, etc.), the system cannot perceive and adjust its internal model in a timely manner, resulting in possible inadaptability or errors when dealing with related issues.

[0053] Based on this, the main solution of the embodiment of the present application is: construct a resource library, and crawl the data to be updated based on the resource category or resource information of the resource library;

[0054] When the update process of the resource library is triggered, the word library information to be updated of the resource library is obtained;

[0055] Performing training processing on the preprocessed vocabulary information to be updated based on a deep learning model to obtain a training result;

[0056] The training results are updated and stored in a graph database based on preset semantic logic rules, wherein the graph database contains a graph structure of all data in the resource library.

[0057] The present application provides a solution by constructing a resource library and crawling the latest data information such as laws and regulations or academic paper information in real time based on the resource library. Then, after triggering the update process of the resource library, based on the deep learning model and related semantic logic rules, the latest data in the Jia Ang resource library is updated to the system's graph database, realizing automatic docking with external data, so that the artificial intelligence system can perceive and learn the latest data such as the latest laws and regulations or academic information in real time, thereby automatically adjusting and optimizing the model values, ensuring that the system always complies with the latest regulatory requirements or academic requirements when dealing with related issues, and improving the reliability of the system and the accuracy of data processing.

[0058] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a data processing device, etc. The following takes a data processing device as an example to illustrate this embodiment and the following embodiments.

[0059] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0060] The present application embodiment provides a data processing method, referring to Figure 1 , Figure 1 This is a flowchart of the first embodiment of the data processing method of the present application.

[0061] In this embodiment, the data processing method includes steps S10 to S40:

[0062] Step S10, building a resource library, and crawling the data to be updated based on the resource category or resource information of the resource library;

[0063] It should be noted that the resource library refers to the use of crawler technology or API (Application Programming Interface) to regularly obtain the latest data from official channels, academic websites, etc. Resource category or resource information refers to the type of data that needs to be crawled. For example, if the resource library stores legal and regulatory information, it is necessary to crawl legal and regulatory information. If it stores academic paper information, it is necessary to crawl the latest academic paper information. Data to be updated refers to the latest data crawled.

[0064] In this embodiment, taking laws and regulations and academic papers as examples, the resource library sets a scheduled data crawling task. When the time point associated with the data crawling task is reached, the data to be updated corresponding to the resource category or resource information in the resource library is crawled, and then the data to be updated is deduplicated based on the current data of the resource library, and the deduplicated data to be updated is added to the resource library.

[0065] Specifically, taking laws and regulations and academic papers as examples, when the data crawling time point is reached, the latest laws and regulations, judicial cases, academic research papers and other content are obtained from various official channels and academic websites. The obtained content is then analyzed and compared with the existing content to remove duplicate information. At the same time, the latest legal and regulatory information and academic paper information are updated to the resource library.

[0066] In this embodiment, the latest data is crawled regularly and updated to the resource library so that the processing system of the data processing device, such as the artificial intelligence system, can extract the latest and valuable data information from the resource library and update it to the graph database, thereby completing the optimization and improvement of the system and improving the accuracy and reliability of the system in processing tasks.

[0067] Step S20, when the update process of the resource library is triggered, the word library information to be updated of the resource library is obtained;

[0068] In this embodiment, the word library information to be updated refers to the latest data crawled by the resource library, or the data that has changed in the original data, which has not been stored or updated in the graph data of the data processing device. Therefore, it is necessary to update the data of the resource library to the graph database of the data processing device so that the processing system of the data processing device can process related tasks based on the latest data or the latest data definition information.

[0069] When a resource update task is received from the user, the update process of the resource library will be triggered, or when the preset update time of the resource library is reached, the process will be triggered; optionally, when the total amount of new data in the resource library exceeds the preset threshold, such as the latest published legal and regulatory information, the amount of data changes involved is far greater than the preset threshold, for example, the set data amount is 20, and the data change amount is 30, then the update process of the resource library needs to be triggered.

[0070] Optionally, the resource library can also mark the data of focus, also known as target data. When the updated information of the target data is crawled, the update process of the resource library can be triggered. For example, a legal service company or law firm may set specific legal provisions (such as tax law, commercial law, environmental law, etc.) as target data. Once the relevant legal provisions are modified or updated, they need to know immediately in order to provide the latest legal advice to customers. At this time, these legal provisions are the target data. If new target data is crawled, the data stored in the graph database needs to be updated based on the new data to improve system reliability and the effectiveness of task processing.

[0071] After the update process of the resource library is triggered, the newly added data and modified data in the resource library can be determined according to the data update time of the resource library and the data update strategy of the resource library, and then the newly added data and modified data are used as the vocabulary information to be updated.

[0072] Exemplarily, when the preset time for triggering the update process of the resource library is reached, the newly added data within this time period can be determined based on the current time and the previous data update time. At the same time, the data that has changed within this time period can be determined based on the data modification information in the data update strategy. Finally, the changed data and the newly added data are used as the vocabulary information to be updated.

[0073] Step S30, training the pre-processed vocabulary information to be updated based on a deep learning model to obtain a training result;

[0074] In this embodiment, before updating the graph database with the latest data crawled from the resource library, the data needs to be pre-processed such as cleaning, verification, and standardization to ensure that only accurate and consistent data is added to the graph database, thereby improving system reliability during subsequent processing tasks.

[0075] When adding the data of the resource library to the graph database, the original data needs to be converted into a form that can be understood by the deep learning model to improve data quality and training efficiency. Therefore, it is necessary to train the pre-processed vocabulary information to be updated through the deep learning model to obtain the training results, which are the data form that can be understood by the model in the subsequent data classification processing.

[0076] Specifically, the process of data conversion can include data cleaning, named entity recognition and semantic coding. Data cleaning ensures data accuracy and consistency by removing duplicate, erroneous and irrelevant data. Named entity recognition uses natural language processing technology to identify key entities in the text, such as names of people, places, and legal clauses. Semantic coding encodes the cleaned data based on custom semantic logic specifications to form a standardized data format suitable for graph database training. When performing named entity recognition, it is also necessary to build a vocabulary information that conforms to the named entity recognition rules. Exemplarily, the semantic coding process includes standardized naming, data type consistency, clear logical relationships, unique identifier allocation and attribute coding. Standardized naming is to unify the names of named entities and relationships; data type consistency is to ensure that the same type of data uses a consistent data format, and at the same time, it is necessary to clarify the logical relationships of entities to avoid entity isolation; in the unique identifier allocation process, including node identifiers and relationship identifiers, a unique identifier (ID) is assigned to each entity through the node identifier to ensure that each node is unique in the graph database, and a unique identifier is assigned to each relationship through the relationship identifier to ensure the uniqueness and traceability of the relationship; attribute coding is to perform numerical coding and categorical coding. For numerical attributes, standardization or normalization is required, and for categorical attributes, integer coding methods are used.

[0077] As an optional implementation, the to-be-updated vocabulary information may be subjected to data cleaning processing, and then based on a pre-trained BERT (Bidirectional Encoder Representations from Transformers) model and a domain vocabulary corresponding to the to-be-updated vocabulary information, the pre-processed vocabulary information to be updated may be subjected to word segmentation and part-of-speech tagging processing to obtain key entity data, and finally the key entity data may be subjected to semantic encoding processing to obtain the pre-processed vocabulary information to be updated.

[0078] For example, the pre-processed text of the vocabulary information to be updated can be segmented and POS tagged by using the trained BERT model and the vocabulary information of the domain vocabulary, so as to obtain key entity data, where segmentation refers to dividing the text into independent words or phrases; POS tagging refers to marking each word with its part of speech (such as noun, verb, adjective, etc.). Optionally, the pre-processed vocabulary information to be updated can also be annotated with domain information based on the knowledge graph constructed in the graph database and combined with the Wikipedia graph information, that is, the self-built knowledge graph can be improved by training the model on the existing historical data, including the annotation of domain information.

[0079] Optionally, during the preprocessing process, after data cleaning, it is necessary to extract the features of each word, and based on the BERT technology, convert the word into a vector representation to capture its semantic information, and then input the vector feature into the trained BERT model.

[0080] Optionally, during the model training process, the labeled training data needs to be used to train the BERT model, and then the trained model is applied to the vocabulary information to be updated, and word segmentation and part-of-speech tagging are performed on it, and then the model is used to identify entities in the text.

[0081] Furthermore, when constructing a domain vocabulary, the information category of the vocabulary information to be updated can be determined first, and the corpus information corresponding to the information category can be obtained, and then the corpus information can be processed by word segmentation, deduplication and word screening, and finally the target words greater than the preset number of words after processing are determined, and finally the target words are written into the domain vocabulary. In addition, the words associated with the word screening or word supplement operation can be added to the domain vocabulary in response to the word screening or word supplement operation input by the user. For example, based on the key information such as the published legal and regulatory database, case database, academic research papers, etc., the text data is then divided into independent words or phrases, and duplicate words are removed at the same time to ensure that the entries in the vocabulary are unique. Finally, the frequency of occurrence of each word in the corpus is counted, and high-frequency words are usually given priority to enter the vocabulary. In addition, the results of machine screening can be manually reviewed to ensure the accuracy and coverage of the words, or the domain words in the knowledge graph and the named entity recognition model can be combined to obtain new entities in the corpus to improve the vocabulary.

[0082] In a specific implementation scenario, the system of the data processing device is a legal information retrieval system, which helps users find regulations, cases, and interpretations related to specific legal issues. As laws are constantly updated and revised, the system needs to regularly update its legal vocabulary to ensure the accuracy and timeliness of the relevant information provided. Among them, the system's graph database can be updated with the latest legal documents and revisions to ensure that the system can understand and process the latest legal terms and concepts.

[0083] In this process, the latest legal text data can be collected from official legal notices, legislative websites, and newly released legal opinions based on the constructed resource library. Then, when updating these latest data to the legal information retrieval system, it is necessary to clean these data first, standardize the format of legal terms, remove irrelevant characters, correct spelling errors, etc., and then use the pre-trained BERT model for word segmentation and part-of-speech tagging. For example, "tort" and "liability law" in "tort liability law" are identified as noun phrases. At the same time, the model is used to identify key legal entities in the text, such as legal names, clause numbers, related cases, etc. Finally, the processed key entity data is deeply semantically encoded based on the pre-trained BERT model to capture the association and contextual information between entities. Based on this, the pre-trained BERT model is used to preprocess and analyze text data in professional fields (such as laws and regulations) in practical applications to achieve dynamic updating of the vocabulary of specific systems (such as legal information retrieval systems) and improve the effectiveness of system processing.

[0084] Optionally, in other implementation scenarios such as an online restaurant review system, the latest restaurant review text data can be collected from the online platform based on the constructed resource library, and the latest review text can be processed accordingly, and the processed data can be updated to the graph database.

[0085] Step S40: updating and storing the training results in a graph database based on preset semantic logic rules.

[0086] In this embodiment, after the updated vocabulary information is processed accordingly, it is necessary to be based on a deep learning model that can handle complex semantic relationships. The model has the ability to understand the preprocessed text and generate agreed rule data, and can accurately capture the legal logic and rules in the text, so as to determine the definition of nodes and edges in the graph data of the preprocessed vocabulary information to be updated. Among them, the architecture of the deep learning model is the Transformer architecture, and the model can be a BERT model, a convolutional neural network or a recurrent neural network, etc. It can be understood that as an optional implementation method, the processing of the preprocessed vocabulary information to be updated and the processing based on preset semantic logic rules can be processed by the BERT model, and the two processing processes are not the same.

[0087] Specifically, the semantic definition relationship associated with the preset semantic logic rule can be determined first, and then the logical relationship of the training result can be determined based on the semantic definition relationship, wherein the logical relationship includes at least one of a synonymous relationship, antonym relationship, causal relationship and subordinate relationship. Finally, based on the logical relationship, the training result is updated and stored in the graph database, and the graph database includes a graph structure of nodes and edges. For example, the trained data is imported into the graph database according to the preset semantic logic specification to form a graph structure containing nodes (such as legal clauses, cases, papers, etc.) and edges (such as citation relationships, similarity relationships, etc.). It should be noted that the preset semantic logic specification is used to ensure the interoperability of the model results to the graph database knowledge graph and ensure the semantic consistency of different knowledge nodes and edges, helping the system to understand and process the semantics and logical relationships of the model. Specifically, it includes semantic definition, which is used to define key words and terms used in the system, and give clear explanations and definitions; logical relationships, which are used to define the relationship types between concepts, such as synonymous relationships, antonymous relationships, causal relationships, subordinate relationships, etc., and clarify the application rules and constraints of the relationships, such as "if A is the parent of B, then B must be the child of A", and the result is expressed in the node relationship network formed by the graph database. Optionally, the preset semantic logic specifications can be set according to actual development needs. Among them, after the data processing is completed, the graph database contains the graph structure of all data in the resource library, indicating that the data in the graph database at this time are all the latest data.

[0088] This embodiment provides a data processing method, which uses the resource library as a medium to update the vocabulary information to be updated into the graph database. In this process, the data is preprocessed accordingly to avoid duplicate data entries in the graph database and keep the data neat and efficient. At the same time, the resource library is used as a buffer to perform security checks on the data entering the system to prevent malicious data or potential privacy information from directly entering the graph database, thereby enhancing the security of the system. Furthermore, based on the resource library, the source of the data, the crawling time, and any processing history can be recorded to provide a complete traceability chain for data updates. Based on the resource library, data is automatically crawled in real time to ensure that the graph database can reflect the latest information in a timely manner without manual intervention, thereby improving efficiency and response speed. Finally, through the corresponding model and processing rules, the data is added to the graph database, thereby ensuring that the artificial intelligence system processes the latest data when processing legal-related matters or performing data analysis on legal-related matters, thereby improving the reliability of the system.

[0089] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the first embodiment can refer to the above introduction, and will not be repeated in the following. Figure 2After step S40, the data processing method further includes steps S50 to S60:

[0090] Step S50, when the received data processing task is associated with a keyword of the vocabulary information to be updated, parsing the graph database based on the keyword;

[0091] Step S60: outputting the task result corresponding to the data processing task according to the analysis result.

[0092] In this embodiment, as an optional implementation method, after adding the latest data of the resource library to the graph database, when receiving the processing tasks of the newly added data, the content of the graph database can be parsed based on the keywords associated with these data processing tasks, and then the corresponding results can be output based on the parsed results. That is, based on the parsed results, the corresponding results are generated and output, thereby improving the effectiveness of the system in data processing.

[0093] For example, there is a company that provides online legal consulting services. They use a legal consulting system based on a graph database to automatically answer users' legal questions. This system has a built-in regulatory database that contains various legal keywords and their corresponding legal interpretations or related cases. With the revision of laws and the release of new regulations, the regulatory database needs to be updated regularly to keep the information up-to-date and accurate.

[0094] After a certain law is issued and the legal consulting system updates the latest keywords and related content to the graph database, it receives a data processing task, which is "What is the return and exchange policy of the new consumer rights protection regulations?" The task is associated with the keywords corresponding to the updated data, such as "Article 12 of the Consumer Rights Protection Regulations" and "Return and Exchange Policy". The system then searches for data nodes and relationship edges associated with these keywords, identifies the keywords "Article 12 of the Consumer Rights Protection Regulations" and "Return and Exchange Policy", and searches for corresponding nodes and relationships in the graph database. Then, based on the parsing results, the system outputs the corresponding task results, that is, provides a specific explanation of the return and exchange policy in Article 12 of the Consumer Rights Protection Regulations as an answer. Based on this, the efficient execution of data processing tasks and the consistency of consulting services are ensured, and the effectiveness of the system is improved. It is understandable that the above processing method can still be used when processing nodes corresponding to existing relevant data, and no further elaboration is given here.

[0095] In addition, users can also query the data structure and related relationships of all data in the graph database. That is, when receiving a data structure query instruction of the graph database, the data structure and data relationship of the target data associated with the query instruction are output to facilitate data verification and comparison.

[0096] The present application provides a data processing device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the data processing method in the above-mentioned first embodiment.

[0097] Reference below Figure 3 , which shows a schematic diagram of the structure of a data processing device suitable for implementing an embodiment of the present application. The data processing device in the embodiment of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 3 The data processing device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0098] like Figure 3 As shown, the data processing device may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the data processing device are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the data processing device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a data processing device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0099] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0100] The data processing device provided by the present application adopts the data processing method in the above embodiment, which can solve the technical problem that the artificial intelligence system performs data analysis based on outdated information, resulting in low system reliability. Compared with the prior art, the beneficial effects of the data processing device provided by the present application are the same as the beneficial effects of the data processing method provided by the above embodiment, and the other technical features in the data processing device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0101] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0102] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0103] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the data processing method in the above-mentioned embodiment.

[0104] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0105] The computer-readable storage medium may be included in a data processing device, or may exist independently without being incorporated into a data processing device.

[0106] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by a data processing device, the data processing device:

[0107] Building a resource library, and crawling the data to be updated based on the resource category or resource information of the resource library;

[0108] When the update process of the resource library is triggered, the word library information to be updated of the resource library is obtained;

[0109] Performing training processing on the preprocessed vocabulary information to be updated based on a deep learning model to obtain a training result;

[0110] The training results are updated and stored in a graph database based on preset semantic logic rules, wherein the graph database contains a graph structure of all data in the resource library.

[0111] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0112] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0113] The modules involved in the embodiments of the present application may be implemented by software or hardware, wherein the name of the module does not limit the unit itself in some cases.

[0114] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned data processing method, and can solve the technical problem that the artificial intelligence system performs data analysis based on outdated information, resulting in low reliability of the system. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the data processing method provided in the above-mentioned embodiment, and will not be elaborated here.

[0115] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned data processing method when executed by a processor.

[0116] The computer program product provided by this application can solve the technical problem that the artificial intelligence system performs data analysis based on outdated information, resulting in low system reliability. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the data processing method provided by the above embodiment, and will not be repeated here.

[0117] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A data processing method, characterized in that: The method comprises: Building a resource library, and crawling the data to be updated based on the resource category or resource information of the resource library; When the update process of the resource library is triggered, the data update time and data update strategy of the resource library are obtained; based on the data update time and the data update strategy, the newly added data and the modified data are determined; the newly added data and the modified data are used as the vocabulary information to be updated, wherein the update process of the resource library is triggered when the resource library crawls the update information of the target data and the total amount of the newly added data of the resource library exceeds the preset threshold, and the target data is the legal provisions selected by the user; Performing data cleaning processing on the vocabulary information to be updated; Determine the information category of the vocabulary information to be updated, and obtain the corpus information corresponding to the information category; The corpus information is processed by word segmentation, deduplication and word screening, and target words greater than a preset number are determined among the processed words; the target words are written into a domain word library; or in response to a word screening or word supplement operation, the words associated with the word screening or word supplement operation are added to the domain word library; Based on the pre-trained BERT model and the domain vocabulary corresponding to the vocabulary information to be updated, the pre-processed vocabulary information to be updated is segmented and POS tagged to obtain key entity data; Performing semantic coding processing on the key entity data to obtain pre-processed vocabulary information to be updated, wherein the semantic coding processing process includes standardized naming processing, data type consistency processing, clear logical relationship, unique identifier allocation and attribute coding processing; Performing training processing on the preprocessed vocabulary information to be updated based on a deep learning model to obtain a training result; Determine the semantic definition relationship associated with the preset semantic logic rules; Determining the logical relationship of the training result based on the semantic definition relationship, wherein the logical relationship includes at least one of a synonymous relationship, an antonymous relationship, a causal relationship, and a subordinate relationship; Based on the logical relationship, the training results are updated and stored in a graph database, the graph database includes a graph structure of nodes and edges, the graph database contains the graph structure of all data in the resource library, and the preset semantic logic rules are used to ensure the interoperability of the model results to the graph database knowledge graph and the semantic consistency of different knowledge nodes and edges; When the received data processing task is associated with a keyword of the vocabulary information to be updated, the graph database is parsed based on the keyword, and the data processing task is a query task corresponding to the crawled data; The task result corresponding to the data processing task is output according to the analysis result.

2. The method according to claim 1, characterized in that After the step of updating and storing the training results in the graph database based on the preset semantic logic rules, the method further includes: When a data structure query instruction of the graph database is received, the data structure and data relationship of the target data associated with the query instruction are output.

3. The method according to claim 1, characterized in that The step of constructing a resource library and crawling the data to be updated based on the resource category or resource information of the resource library includes: Arriving at a time point associated with a data crawling task, crawling the to-be-updated data corresponding to the resource category or resource information in the resource library; Based on the current data of the resource library, the data to be updated is deduplicated, and the deduplicated data to be updated is added to the resource library.

4. The method according to claim 1, characterized in that The update process of the resource library is triggered when at least one of the following conditions is met: Receive resource update task; The preset resource library update time has arrived.

5. A data processing device, characterized in that: The data processing device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data processing method according to any one of claims 1 to 4.

6. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 4 are implemented.

Images