A data processing method, device and readable storage medium

Abstract

Embodiments of the present application disclose a data processing method, device and readable storage medium. The data processing method comprises: acquiring source service data, and acquiring a data format type of the source service data; performing annotation information extraction in the source service data according to the data format type to obtain annotation information corresponding to the source service data; acquiring multimedia sub-data with a target media type in the source service data, performing format standardization processing on the multimedia sub-data to obtain updated source service data; and storing the updated source service data and the annotation information in association to a database. By using the embodiments of the present application, the cost of manual annotation can be minimized, the efficiency of data annotation can be improved, and the compatibility of data can be improved.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a data processing method, apparatus, and readable storage medium. Background Technology

[0002] With the development of data informatization, the amount of data is growing rapidly. The labeled information of data plays an important role in recommendation systems, supervised learning and other fields. Therefore, how to quickly and efficiently extract effective information from open domain data has become an important problem.

[0003] Existing technologies rely heavily on human resources. Some fields require a high level of expertise from team members, and information is mainly labeled manually. This results in a large amount of data to be labeled, which is time-consuming and leads to low data processing efficiency. Furthermore, existing technologies cannot handle data compatibility issues, making it difficult to guarantee data universality. Summary of the Invention

[0004] This invention provides a data processing method, apparatus, and readable storage medium that can minimize the cost of manual annotation, improve the efficiency of data annotation, and enhance data compatibility.

[0005] One embodiment of the present invention provides a data processing method, the method comprising:

[0006] Obtain source business data, and obtain the data format type of the source business data;

[0007] According to the data format type, annotation information is extracted from the source business data to obtain the annotation information corresponding to the source business data;

[0008] Obtain multimedia sub-data with the target media type from the source service data, perform format standardization processing on the multimedia sub-data, and obtain updated source service data;

[0009] The updated source business data and the annotation information are associated and stored in the database.

[0010] One embodiment of the present invention provides a data processing apparatus, the apparatus comprising:

[0011] The acquisition module is used to acquire source business data and obtain the data format type of the source business data;

[0012] The extraction module is used to extract annotation information from the source business data according to the data format type, so as to obtain the annotation information corresponding to the source business data.

[0013] The standardization module is used to obtain multimedia sub-data with target media type from the source service data, perform format standardization processing on the multimedia sub-data, and obtain updated source service data.

[0014] The storage module is used to associate and store the updated source business data and the annotation information in the database.

[0015] The extraction module includes:

[0016] The parsing unit is used to parse the source business data if the data format type is a web page format type, and obtain the valid fields contained in the source business data;

[0017] The webpage extraction unit is used to perform field matching in a standard tag set in the database based on the valid fields; obtain field annotation information based on the valid fields that match the standard tag set; if there is an identifier field in the field annotation information, perform identifier mapping on the identifier field in the field annotation information according to the identifier mapping relationship table in the database to obtain the mapping value corresponding to the identifier field; and determine the field annotation information and the mapping value corresponding to the identifier field as the annotation information corresponding to the source business data.

[0018] The extraction module includes:

[0019] The filename acquisition unit is used to acquire the filename of the source business data if the data format type is a file format type.

[0020] The first file extraction unit is configured to extract the annotation information corresponding to the source business data from the file name based on the regular expression if the naming format of the file name conforms to the filtering logic of the regular expression; if the naming format of the file name does not conform to the filtering logic of the regular expression, it traverses the standard tags in the standard tag set in the database and determines the data in the file name that matches the value of the standard tag as the annotation information corresponding to the source business data.

[0021] The extraction module includes:

[0022] The filename acquisition unit is used to acquire the filename of the source business data if the data format type is a file format type.

[0023] The second file extraction unit is used to perform word segmentation on the file name to obtain N words; N is a positive integer greater than or equal to 1; perform part-of-speech tagging on the N words to obtain the part-of-speech tags corresponding to the N words; perform named entity recognition on the N words based on the part-of-speech tags to obtain the entity types corresponding to the N words respectively; and determine the word whose entity type is a geographic organization entity type as the annotation information corresponding to the source business data.

[0024] The standardization module includes:

[0025] Sub-data acquisition unit, used to acquire multimedia sub-data with target media type from the source service data;

[0026] The image processing unit is configured to, if the target media type is an image type and the multimedia sub-data is webpage description data of the image type, extract image tags and image resource addresses from the webpage description data, convert the image tags and image resource addresses into standard image data according to the standard media resource protocol, filter out invalid tags in the webpage description data, and determine the filtered webpage description data containing the standard image data as the updated source business data.

[0027] The table processing unit is configured to, if the target media type is a table type and the multimedia sub-data is webpage description data of the table type, extract table tags and table content from the webpage description data, convert the table tags and table content into standard table data according to the standard media resource protocol, filter out invalid tags in the webpage description data, and determine the filtered webpage description data containing the standard table data as the updated source business data.

[0028] The standardization module includes:

[0029] Sub-data acquisition unit, used to acquire multimedia sub-data with target media type from the source service data;

[0030] The formula processing unit is used to: if the target media type is a formula type, identify the formula format of the multimedia sub-data; if the formula format is a general image format, call the formula recognition interface to convert the multimedia sub-data into standard text data to obtain updated source business data; if the formula format is a metafile format, convert the multimedia sub-data into general image format data, call the formula recognition interface to convert the multimedia sub-data with general image format into standard text data to obtain updated source business data.

[0031] The standardization module includes:

[0032] Sub-data acquisition unit, used to acquire multimedia sub-data with target media type from the source service data;

[0033] The audio and video processing unit is configured to, if the target media type is an audio and video type, obtain the audio and video encoding format corresponding to the multimedia sub-data; if the audio and video encoding format is not a standard audio and video encoding format, convert the multimedia sub-data into standard audio and video data to obtain updated source service data.

[0034] The storage module includes:

[0035] The standardized data generation unit is used to assemble the updated source business data and the annotation information according to the standard data protocol to generate standardized data.

[0036] The storage association unit is used to store the standardized data in a database and establish an association between the standardized data and the set of standard labels in the database.

[0037] One embodiment of the present invention provides a data processing apparatus, including: a processor and a memory;

[0038] The processor is connected to a memory, wherein the memory is used to store a computer program, and the processor is used to invoke the computer program to execute a method as described in one aspect of the embodiments of the present invention.

[0039] One aspect of the present invention provides a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a processor, perform the method as described in one aspect of the present invention.

[0040] This invention can acquire source business data and identify its data format type. Then, based on the data format type, appropriate methods can be used to extract annotation information from the source business data to obtain corresponding annotation information. Furthermore, various types of multimedia sub-data can be standardized to obtain updated source business data. The updated source business data and annotation information can then be associated and stored in a database, enabling smooth interaction and sharing of the same data between different business systems, thus improving data compatibility. Because the extraction of annotation information and data storage can be automated, manual costs can be minimized, and the efficiency of data annotation can be improved. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a schematic diagram of a system architecture provided by an embodiment of the present invention;

[0043] Figure 2 This is a schematic diagram of a data processing method provided in an embodiment of the present invention;

[0044] Figure 3 This is a flowchart illustrating a data processing method provided in an embodiment of the present invention;

[0045] Figure 4 This is a flowchart illustrating a data processing method provided in an embodiment of the present invention;

[0046] Figure 5 This is a flowchart illustrating another data processing method provided in an embodiment of the present invention;

[0047] Figure 6 This is a schematic diagram of a scenario for pre-labeled data provided in an embodiment of the present invention;

[0048] Figure 7 This is a flowchart illustrating another data processing method provided in an embodiment of the present invention;

[0049] Figure 8 This is a schematic diagram of the structure of a data processing device provided in an embodiment of the present invention;

[0050] Figure 9 This is a schematic diagram of another data processing device provided in an embodiment of the present invention. Detailed Implementation

[0051] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0052] Please see Figure 1 This is a schematic diagram of a system architecture provided in an embodiment of the present invention. Figure 1As shown, the system architecture may include server 100, terminal devices 200a, 200b, and 200c. Server 100 can transmit data with each terminal device via a network. Each terminal device can collect source business data to be processed. Server 100 can extract annotation information and perform format standardization processing on the source business data collected by each terminal device. Here, the data transmission between terminal device 200a and server 100 is used as an example to describe this embodiment of the invention. Terminal device 200a can send the collected source business data to server 100. Server 100 can obtain the source business data and, according to the data format type of the source business data, extract annotation information and perform format standardization processing on the source business data, and then return the processing result to terminal device 200a. The source business data can be resources such as documents, images, audio and video, and courseware. The annotation information can represent data tags and classifications in different fields. For example, in the field of education, the annotation information can be tags such as textbook version, grade, subject, and chapter.

[0053] It is understood that the method provided in this embodiment of the invention can be executed by a computer device, which includes, but is not limited to, terminal devices or servers. The server 100 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud databases, cloud services, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Terminal devices can include, but are not limited to, mobile phones, tablets, laptops, PDAs, mobile internet devices (MIDs), and wearable devices (such as smartwatches and smart bracelets).

[0054] Please see also Figure 2 This is a schematic diagram of a data processing method provided in an embodiment of the present invention. Figure 2 As shown, this scene diagram is... Figure 1Taking server 100, terminal device 200a, and terminal device 200c as examples in the corresponding embodiments, terminal device 200c can upload test questions (also known as source business data) provided by teacher users through the open platform. Specifically, terminal device 200c responds to a trigger operation for uploading test questions (such as a teacher user clicking the upload control in terminal device 200c), sending the test questions to server 100. Server 100 receives the test questions and can obtain the data format of the test questions. Based on the data format, it can extract usable annotation information from the test questions, such as the subject, grade, school, and knowledge points corresponding to the test questions. Furthermore, server 100 can obtain multimedia sub-data with target media types from the test questions and perform format standardization processing on the multimedia sub-data. For example, it can convert all formulas in the test questions into predefined standard text data to obtain updated test questions. The updated test questions and the extracted annotation information can then be associated and stored in the test question database. The target media type can be image type, table type, formula type, audio and video type, etc. The format standardization processing of various multimedia sub-data in the test questions is carried out, that is, multimedia sub-data with the same media type is converted into a unified format, which can improve data compatibility and facilitate data sharing between different business systems.

[0055] After opening the application interface of the learning application C1, the terminal device 200a first displays the default homepage of the learning application. This homepage displays at least six function buttons: "Topic," "Announcement," "Practice," "Test," "Materials," and "Review Package." When a student selects the "Practice" button, the terminal device 200a jumps to the corresponding selection page. This selection page displays a list of test question tags, such as "Elementary School," "Middle School," "High School," "Chinese," "Mathematics," "Region," "Knowledge Point," and "Grade." The tag list can be generated based on annotation information, facilitating the establishment of a correspondence between the tag list and the user's selection. In response to the student's selection of the tag list, the terminal device 200a sends a test question database access request to the server 100 to retrieve matching test questions. Furthermore, server 100 can determine the tag selection information corresponding to the requesting terminal device 200a based on the test question database access request, and can filter out test questions that match the tag selection information from the test question database as target test questions, and then send the target test questions to terminal device 200a for display. Optionally, student users can enter keywords in the search box on the selection page, and terminal device 200a can send the entered keywords to server 100. Server 100 can search the test question database for tag information that matches the above keywords, and then use the test questions corresponding to the matched tag information as target test questions. After the student user completes the test questions, terminal device 200a can send the student user's answer feedback to server 100. Server 100 can judge the answer feedback and send the judgment result to terminal device 200a and terminal device 200c. Teacher users can judge the student user's mastery of a certain knowledge point based on the accuracy of the answers, and then decide whether to intervene. It can be understood that server 100 can also automatically push relevant test questions to terminal device 200a based on the student user's historical tag selection information or historical search records.

[0056] Therefore, server 100 can extract annotation information from the source business data according to the data format of the source business data, obtain the corresponding annotation information, and perform format standardization processing on various types of multimedia sub-data to obtain updated source business data. By associating and storing the updated source business data and annotation information, different business systems can smoothly interact and share the same data, improving data compatibility. Since the extraction of annotation information and data storage can be automated, manual costs can be minimized, the efficiency of data annotation can be improved, and relevant source business data can be quickly retrieved from the database, improving retrieval efficiency.

[0057] Please see Figure 3 This is a flowchart illustrating a data processing method provided in an embodiment of the present invention. The method may include the following steps:

[0058] Step S101: Obtain source service data and obtain the data format type of the source service data;

[0059] Specifically, the server can obtain source business data, which can be resources such as documents, images, audio and video from various fields. For example, in the education field, source business data can include, but is not limited to, questions, test papers, teaching courseware, teaching videos, audio materials, and image materials. Furthermore, the server can identify the data format type of the source business data. Depending on the source of the source business data, the data format type can include web page format type and file format type. For example, source business data provided by content providers through API interfaces (Application Programming Interfaces) or cloud storage sharing corresponds to a web page format type, which includes the resource download address. Meanwhile, resources such as documents, courseware, images, and audio and video correspond to a file format type.

[0060] Step S102: According to the data format type, extract annotation information from the source business data to obtain the annotation information corresponding to the source business data;

[0061] Specifically, the server can obtain a set of standard tags from the database, and based on the data format type identified in step S101 above, extract the required annotation information from the source business data. For example, when the data format type of the source business data is a webpage format type, the source business data is structured data, that is, data arranged in a certain structure to describe the content of a webpage. The server can parse the source business data to obtain the valid fields contained in the source business data, and then match the valid fields with the set of standard tags to obtain field annotation information. It can also perform identifier mapping on the identifier fields in the field annotation information to obtain the mapping value corresponding to the identifier field. Then the server can determine the field annotation information and the mapping value corresponding to the identifier field as the annotation information corresponding to the source business data. For example, after parsing the source business data, the server obtains the valid field "Grade 9" and can identify "Grade 9" as an identifier field. Then "Grade 9" can be mapped to the corresponding mapping value "9" through identifier mapping. Optionally, when the source business data is in file format, the server can extract the annotation information corresponding to the source business data from the file name. For example, if the file name of a certain test paper is "2006 Guangdong Province Shenzhen Experimental School Direct Admission Examination Mathematics Test Paper", the year, subject, region and other annotation information can be extracted from it.

[0062] The standard tag set can be predefined and stored in a database. When the server extracts annotation information, it can read the standard tags in the standard tag set to match the information extracted from the source business data. For example, before extracting annotation information from educational resources, a standard tag set for the education field can be created. This set can include standard tags such as grade level, subject, textbook version, question type, year, region, school, chapter, and knowledge point, as well as the corresponding values ​​of these standard tags. For example, the value corresponding to the [Grade] tag can include "Grade 1", "Grade 2", "Grade 3", etc. When the test paper contains "Chinese", and a matching value "Chinese" is found in the [Subject] tag of the standard tag set, then the [Subject] Chinese can be extracted as the annotation information corresponding to the test paper.

[0063] Step S103: Obtain multimedia sub-data with target media type from the source service data, perform format standardization processing on the multimedia sub-data, and obtain updated source service data;

[0064] Specifically, the source business data may include at least one or more multimedia sub-data such as text, images, tables, formulas, audio, and video. Correspondingly, the target media type may include image type, table type, formula type, and audio / video type. Among them, audio / video type is a general term for audio type and video type. The server can use different methods to perform corresponding format standardization processing on multimedia sub-data according to different target media types. That is, multimedia sub-data with the same media type can be converted into a unified format to obtain updated source business data. Therefore, it can improve data compatibility and facilitate the use of different businesses.

[0065] For example, when the source business data contains multimedia sub-data of video type, the server can obtain the video encoding format corresponding to the multimedia sub-data. If the video encoding format is not a standard video encoding format, the server can convert the multimedia sub-data into standard video data.

[0066] It should be noted that the order of steps S102 and S103 can be adjusted according to actual application needs, and these two steps can also be executed simultaneously. In this embodiment of the invention, no specific limitation is made on the order of execution of the two steps.

[0067] Step S104: The updated source business data and the annotation information are associated and stored in the database.

[0068] Specifically, the server can assemble the updated source business data and extracted annotation information according to a predefined standard data protocol to generate standardized data. For example, below is a fragment of standardized data generated from an essay according to the annotation data protocol:

[0069] As can be seen from the above excerpt, the standard data protocol defines the overall framework of standardized data. It can include the attributes of the source business data (such as "author_name", "guider", etc.) and the specific arrangement of the data. By assembling and arranging the updated source business data and the extracted annotation information with the corresponding attributes, standardized data can be obtained. The standard data protocol can be defined according to actual needs, and this embodiment of the invention does not impose specific limitations here.

[0070] Furthermore, the server can store standardized data in a database, and establish an association between the standardized data and the set of standard labels in the database based on the mapping value corresponding to the identifier field obtained in step S102 above. Specifically, standardized data with the same mapping value can be associated with the standard label corresponding to that mapping value. Subsequently, standardized data with a specific mapping value can be found in the database through this association, and then the corresponding source business data can be accessed based on the resource address in the standardized data.

[0071] This invention can acquire source business data and identify its data format type. Then, based on the data format type, appropriate methods can be used to extract annotation information from the source business data to obtain corresponding annotation information. Furthermore, it can standardize the formats of various types of multimedia sub-data to obtain updated source business data. The updated source business data and annotation information can then be associated and stored in a database, enabling smooth interaction and sharing of the same data between different business systems, thus improving data compatibility. Because the extraction of annotation information and the storage of data can be automated, manual costs can be minimized, and the efficiency of data annotation can be improved.

[0072] Please see Figure 4 This is a flowchart illustrating a data processing method provided in an embodiment of the present invention. The method may include the following steps:

[0073] Step S201: Obtain source service data and obtain the data format type of the source service data;

[0074] Specifically, the server can obtain source business data and identify its data format type. When the data format type is identified as a webpage format, subsequent steps S202-S208 can be executed. The webpage format source business data can originate from an API interface provided by the content provider or from a cloud storage link provided by the content provider. This webpage format source business data may contain annotation information and resource download addresses. Specifically, the webpage format source business data can be in JSON (JavaScript Object Notation) format. JSON is a lightweight data exchange format that is easy for users to read and write, and also easy for machines to parse and generate, effectively improving network transmission efficiency. JSON is built on two structures: a collection of name / value pairs and an ordered list of values. In different languages, name / value pairs can be understood as objects, records, structs, dictionaries, hash tables, keyed lists, or associative arrays; while the ordered list of values ​​is understood as an array in most languages. These are common data structures, and in fact, most modern computer languages ​​support them in some form, making it possible for JSON to be exchanged between programming languages ​​that also rely on these data structures.

[0075] Step S202: Perform text cleaning on the source business data to obtain cleaned source business data;

[0076] Specifically, the source business data may contain some data that cannot be directly displayed or used for algorithm training. The server can perform text cleaning on this type of data to obtain the cleaned source business data.

[0077] Taking HTML (Hyper Text Markup Language) text cleaning as an example, the following processing steps can be included:

[0078] The server can convert some HTML elements into standard text. Specifically, the standard text can be LaTeX text. Here, HTML elements refer to all the code from the start tag to the end tag, such as... 、 . LaTeX is a TeX-based typesetting system. Using this format, even users without knowledge of typesetting and programming can fully utilize the powerful features provided by TeX and can generate many printed materials of book quality within a few days or even a few hours. This is particularly prominent when generating complex tables and mathematical formulas. For example, for the superscript in HTML elements, "x 2 ²" can be converted to the LaTeX text "$x^{2}$"; for the subscript in HTML elements, "x2" can be converted to the LaTeX text "$x_{2}$"; for the underline in HTML elements, "闻鸡 rise 舞" can be converted to the LaTeX text "闻鸡$\underline{起}$舞".

[0079] The server can extract information according to some special data protocols defined by the content provider. For example, the formula of an open interface is placed in the "data-latex" attribute field of the tag. The server can remove the

[0080]

[0081] tag and only retain the value of the "data-latex" attribute:

[0082] ·; =>.

[0083] ±; => ±

[0084] The server can also replace some HTML entities with text. For example: 、 , Remove tags from HTML elements, retaining only the text information within them, for example:

[0085] Guess what, the sixth line has <span data-ph＝\″1\″>_ an apple

[0086] => Guess how many apples are in the sixth row?

[0087] Therefore, text cleaning of source business data can effectively improve data readability and compatibility, reduce the amount of source business data, and improve the efficiency of extracting annotation information.

[0088] Furthermore, in webpage-formatted source business data, images, files, audio, and video data are typically represented as URLs (Uniform Resource Locators). The server can perform availability checks on these URLs, filtering out any invalid ones. Adding this availability check step improves data accuracy. It's understandable that the server can also perform availability checks on other types of source business data that contain URLs.

[0089] Step S203: Parse the cleaned source business data to obtain the valid fields contained in the source business data;

[0090] Specifically, source business data in webpage format is structured data, meaning data arranged according to a certain structure to describe webpage content. For example, JSON-formatted source business data may contain many name / value pairs and arrays. The server can parse it to obtain multiple fields (i.e., the "names" in the aforementioned name / value pairs) and their corresponding values. Then, based on a standard tag set, the server can determine the valid fields from among these fields. Valid fields can represent the basic attributes of the source business data, such as... Figure 4 The test paper data in the education field shown can be parsed to obtain valid fields such as "title" and "analysis". Defining valid fields can filter out some irrelevant information in the source business data, thereby reducing the amount of calculation in subsequent steps.

[0091] Step S204: Based on the valid fields, extract annotation information from the cleaned source business data to obtain the annotation information corresponding to the source business data;

[0092] Specifically, the server can perform field matching in the standard tag set in the database based on valid fields, and then obtain field annotation information based on valid fields that match the standard tag set. For example, a retrieval logic can be built based on the valid field "title" in the test paper data. Through this retrieval logic, the standard tag "source" can be retrieved from the standard tag set. Then, the value corresponding to the standard tag "source" and the valid field "title" can be assembled, i.e., "

source

[0093] The server can further determine whether an identifier field exists in the above field annotation information. If an identifier field exists, it can perform identifier mapping based on the identifier mapping table in the database to obtain the corresponding mapping value. The identifier field represents the identifier information of the source business data, such as grade, academic stage, chapter, question type, etc. The identifier mapping table can include identifier fields and their corresponding mapping values, and can be predefined and stored in the database. For example, when the question type field annotation information contains the string "answer question", the string "answer question" can be used to search in the identifier mapping table to obtain the corresponding question type ID (Identity document), also known as the question type mapping value. Finally, the server can determine the above field annotation information and the corresponding mapping values ​​of the identifier fields as the annotation information corresponding to the source business data. Partial code for extracting annotation information is as follows:

[0094] def parse_resource_info(result):

[0095] …

[0096] # Extracting information from different educational stages

[0097] period=result.get("period","")

[0098] study_period_name=_cp_period_mappings.get(period, "")

[0099] study_period_id=_study_period_info_mappings.get(study_period_name, 0)

[0100] …

[0101] # Extract year information

[0102] year=result.get("year", 0)

[0103] …

[0104] year = int(year)

[0105] In the above logic code, when extracting the learning period information, the valid field `period` can be obtained first from the source business data `result`. Based on the valid field `period`, the standard label `study_period_name` in the standard label set can be matched. Then, the identifier field corresponding to the valid field `period` is mapped to obtain the corresponding mapped value `study_period_id`. The logic code for extracting other labeled information is similar to the logic code for extracting learning period information.

[0106] The mapping of the identifier field to the corresponding mapping value can be used to establish an association with the standard tag set when the source business data is subsequently stored in the database. It is understood that the valid fields and identifier fields can be defined according to actual application needs, and this embodiment of the invention does not impose any limitations on them.

[0107] Step S205: Obtain multimedia sub-data with the target media type from the cleaned source service data;

[0108] Specifically, the source business data may contain various multimedia sub-data, such as images, tables, audio, and video. The server can identify the types of these multimedia sub-data. When the identified type matches the target media type, the server can acquire the multimedia sub-data of the target type. In this embodiment of the invention, the target media type may include image type, table type, formula type, and audio / video type.

[0109] Step S206: Perform format standardization processing on the multimedia sub-data to obtain updated source service data;

[0110] Specifically, when the target media type is an image and the multimedia sub-data is image-type webpage description data, the server can perform format standardization processing on the aforementioned webpage description data. For example, the multimedia sub-data can be image-type HTML data (corresponding to the aforementioned webpage description data). The server can extract the image's HTML tags and image resource addresses from the aforementioned HTML data. Then, according to the Standard Media Resource Protocol (SMRP), the image's HTML tags can be converted into custom image placeholders, such as $[img:0]. An image array can be defined in the SMRP to store the image resource addresses, thus obtaining standard image data. The server can further filter out invalid tags and invalid resource addresses in the HTML data. The specific filtering process can be referred to in step S202 above. Subsequently, the filtered webpage description data containing standard image data can be determined as the updated source business data.

[0111] Optionally, when the target media type is a table type and the multimedia sub-data is a table type webpage description data, the server can perform format standardization processing on the aforementioned webpage description data. For example, the multimedia sub-data can be table type HTML data (corresponding to the aforementioned webpage description data). The server can extract the HTML tags and table content of the table from the aforementioned HTML data. Then, according to the standard media resource protocol, the table HTML tags can be converted into custom table placeholders, such as [table:0]. A table array is defined in the standard media resource protocol, the table content is stored in the table array, and then JSON serialization is used to obtain standard table data. Here, serialization refers to the process of converting the object state into a format that can be maintained or transmitted. The server can further filter out invalid tags in the HTML data. The specific filtering process can refer to step S202 above. Then, the filtered webpage description data containing standard table data can be determined as the updated source business data.

[0112] Optionally, when the target media type is a formula type, especially when the formula is represented using an inline image, it is difficult to unify with the text interface and is not easy to edit and index. Therefore, it is necessary to standardize the formula format. The server can first identify the formula format of the multimedia sub-data. If the formula format is a common image format, such as png (Portable Network Graphics) or jpeg (Joint Photographic Expert Group), the server can call the formula recognition interface, input the parameters required by the formula recognition interface, and convert the multimedia sub-data with formula format into standard text data to obtain the updated source business data. If the formula format is a metafile format, specifically the wmf (Wireless MulticastForwarding) format commonly used in Word documents, which is a graphics file format defined by Microsoft for the Windows platform, the server can first convert the multimedia sub-data with formula format into common image format data, and then call the formula recognition interface to convert the multimedia sub-data with common image format into standard text data to obtain the updated source business data. The aforementioned standard text data can specifically be LaTeX text data.

[0113] Optionally, when the target media type is audio / video, the server can obtain the audio / video encoding format corresponding to the multimedia sub-data. If it detects that the aforementioned audio / video encoding format does not belong to the standard audio / video encoding format, the server can convert the multimedia sub-data into standard audio / video data to obtain the updated source service data. Audio / video types include audio and video types. The standard audio encoding format can be AAC (Advanced Audio Coding), and the standard video encoding format can be H.264. H.264 is a digital video coding standard developed by the Joint Video Team (JVT) of ITU-T's VCEG (Video Coding Experts Group) and ISO / IEC's MPEG (Moving Picture Coding Experts Group). For example, if a webpage only supports playing MP4 videos encoded in H.264 and AAC formats, the server can convert MP4 videos in other encoding formats into MP4 videos encoded in H.264 and AAC formats, thereby improving the universality of audio / video data.

[0114] Step S207: According to the standard data protocol, the updated source business data and the annotation information are assembled to generate standardized data;

[0115] Specifically, the server can assemble the updated source business data and extracted annotation information according to a predefined standard data protocol to generate standardized data. This can be standard formatted JSON data with a concise and clear hierarchical structure. Different data types, such as questions, resources, and essays in the education field, can define different standard data protocols.

[0116] For example, the standard data protocol for this problem can refer to the following form:

[0117]

[0118]

[0119] The standard data protocol for resources can refer to the following form:

[0120]

[0121]

[0122] As shown above, standardized data can include a series of "name / value" pairs. For example, "CPName": "Wanxiang" is a "name / value" pair. The "name" can be understood as an attribute name, corresponding to standard tags in the standard tag set, such as QuestionCategoryName, BookVersionName, and QuestionID mentioned above. The "value" can correspond to the annotation information extracted from the source business data, such as "single choice question," "People's Education Edition," and "119001." Therefore, standardized data generated using standard data protocols can be used to characterize the basic attributes of the source business data.

[0123] Step S208: Store the standardized data in a database and establish an association between the standardized data and the set of standard labels in the database.

[0124] Specifically, the database can be a relational database, such as MySQL. A relational database is a database that uses a relational model to organize data, storing data in rows and columns for ease of understanding. The relational model can be simply understood as a two-dimensional table model, and a relational database is a data organization composed of two-dimensional tables and the relationships between them. MySQL is a relational database management system characterized by its small size, high speed, low cost, and open-source nature. The server can store standardized data in the MySQL database and, based on the mapping values ​​corresponding to the identifier fields obtained in step S204 above, associate the standardized data with the established set of standard labels in the MySQL database. Therefore, the MySQL database can store the basic attributes and relationships of data such as questions, resources, and essays.

[0125] For example, see the structure below for storing a set of standardized test questions in a database:

[0126] db_college.t_question_id_mapping

[0127]

[0128]

[0129] …

[0130] db_college.t_question

[0131] 2775976 2 1 4 36 1

[0132] …

[0133] db_college.t_question_stem

[0134]

[0135] As can be seen from the structure above, the two-dimensional table stores the annotation information extracted from the test questions (which may include the original content and mapping values ​​in the test questions), such as question_id, course_id, stem_text, etc., and there are also relationships between the tables.

[0136] This invention can acquire source business data and identify its data format type. Then, based on the data format type, appropriate methods can be used to extract annotation information from the source business data to obtain corresponding annotation information. Furthermore, it can standardize the formats of various types of multimedia sub-data to obtain updated source business data. The updated source business data and annotation information can then be associated and stored in a database, enabling smooth interaction and sharing of the same data between different business systems, thus improving data compatibility. Because the extraction of annotation information and the storage of data can be automated, manual costs can be minimized, and the efficiency of data annotation can be improved.

[0137] Please see Figure 5 This is a flowchart illustrating another data processing method provided in an embodiment of the present invention. The method may include the following steps:

[0138] Step S301: Obtain source service data and obtain the data format type of the source service data;

[0139] Specifically, the server can obtain source business data and then identify the data format type of the source business data. When the data format type is identified as a file format type, subsequent steps S302-S307 can be executed.

[0140] Step S302: Obtain the filename of the source service data;

[0141] Specifically, the server can obtain the filename corresponding to the source business data of the file format type, for example... Figure 5 The exam paper shown can be named "2006 Guangdong Province Shenzhen Experimental School Direct Admission Examination Mathematics Exam Paper".

[0142] Step S303: Extract annotation information from the file name to obtain the annotation information corresponding to the source business data;

[0143] Specifically, when a filename conforms to the naming format rules, that is, when the filename format matches the filtering logic of a regular expression, the server can extract the annotation information corresponding to the source business data from the filename based on the regular expression. For example, for a file named " / Junior High-Grade 9-First Semester-Mathematics-East China Normal University Edition / xxx.pptx", the server can extract the data using the following logic code snippet:

[0144] def search_by_regex(content):

[0145] results = []

[0146] #content:″ / Junior High School-Grade 9-First Semester-Mathematics-East China Normal University Edition / xxx.pptx″

[0147] regex=ur″([\u4e00-\u9fa5]+)-([\u4e00-\u9fa5]+)-([\u4e00-\u9fa5]+)-([\u4e00-\u9fa5]+)-([\u4e00-\u9fa5]+)″

[0148] …

[0149] return results

[0150] In the above code, `regex` is the regular expression for the filename. `[\u4e00-\u9fa5]` can match Chinese characters. Using this regular expression, the server can extract information such as "Junior High School," "Grade 9," "Subject: Mathematics," and "Textbook Version: East China Normal University Edition" as annotation information for the file. It can be understood that the regular expression can be constructed based on different filenames.

[0151] For example, if the file name is "2006 Guangdong Province Shenzhen Experimental School Direct Admission Examination Mathematics Paper", the server can extract the year "2006". See the following logic code snippet for details:

[0152] def_get_year_(text, PATTERN_YEAR = "20\d+year|20\d+graduation|19\d+year|19\d+graduation|19\d+academic year|20\d+academic year|\d+[-]+\d+academic year|\d+-\d+graduation|\d+-\d+year|^20\d{2}|^19\d{2}"):

[0153] yy=re.findall(PATTERN_YEAR, text)

[0154] year = ...

[0155] …

[0156] return year

[0157] The above logic code can be understood as searching for the year in the filename text using the expression PATTERN_YEAR.

[0158] Optionally, when a filename does not conform to the naming format rules (i.e., the filename format does not meet the filtering logic of the regular expression mentioned above), but the filename still contains annotation information, the server can extract it using fuzzy matching. Specifically, the server can iterate through the standard tags in the standard tag set in the database, and then determine the data in the filename that matches the value of the standard tag as the annotation information corresponding to the source business data. For example, a file named "People's Education Edition 2019 Academic Year Primary School Chinese Grade 1 Volume 1 7 'Frogs Writing Poems' Teaching Record" can be extracted using the following logic code:

[0159]

[0160] Using the code above, the server can extract the [study stage] elementary school, and similarly, it can also extract the [grade] first grade, [subject] Chinese, and [textbook version] People's Education Press edition.

[0161] Optionally, some files may also contain regional and institutional information, such as the region and school information contained in exam papers and past exam questions. The server can use an algorithm model to extract this region and school information. Specifically, the algorithm model can be based on the LTP (Language Technology Platform, an open-source Chinese natural language tool). The server can first perform word segmentation on the filename to obtain N words, then perform part-of-speech tagging on these N words to obtain their corresponding parts of speech. Further, based on the parts of speech, named entity recognition can be performed on the N words to obtain the entity types corresponding to each of the N words. Words with the entity type of geographic institution can then be identified as the annotation information corresponding to the source business data. Here, N is a positive integer greater than or equal to 1. Geographic institution entity types can include region entity types and institution entity types. The named entity recognition module in LTP can recognize the three entity types listed in the table below:

[0162] Nh Name Ni Organization Name Ns Place name

[0163] As shown in the table above, Ns can correspond to the regional entity type, and Ni can correspond to the institutional entity type.

[0164] For example, to extract region and school information from a file named "2006 Guangdong Province Shenzhen Experimental School Direct Admission Examination Mathematics Paper", the following is a partial logic code using the LTP algorithm model:

[0165] …

[0166] # Call the word segmentation interface

[0167] words = segmentor.segment('2006 Guangdong Province Shenzhen Experimental School Direct Admission Examination Mathematics Test Paper')

[0168] …

[0169] # Call the part-of-speech tagging interface

[0170] postags=postagger.postag(words)

[0171] …

[0172] # Call the Named Entity Recognition API

[0173] netags=recognizer.recognize(words, postags)

[0174] # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #

[0175] #Word segmentation results

[0176] In 2006, in Shenzhen, Guangdong Province, an experimental school offered direct admission to a higher level, and the exam paper was for mathematics.

[0177] #Part-of-speech tagging results

[0178] nt, ns, ns, v, n, v, v, n, n

[0179] # Named entity recognition results

[0180] O, B-Ns, I-Ns, I-Ni, E-Ni, O, O, O, O

[0181] Referring to the table above, the server can identify that "Guangdong Province" and "Shenzhen City" belong to the [Region] information, and "Experiment" and "School" belong to the [School] information. The corresponding logic code snippet is as follows:

[0182]

[0183] Optionally, the server can also obtain the file directory from the source business data with file format types, and then use the same method as described above for extracting filenames to obtain more annotation information. Optionally, when the source business data contains pre-annotated data, such as user-annotated document data, the server can extract the pre-annotated data into annotation information; please refer to [link to relevant documentation]. Figure 6 This is a schematic diagram of a scenario for pre-labeled data provided in an embodiment of the present invention, such as... Figure 6 As shown, the pre-labeled data has been annotated with text, and the text within the dashed boxes w1-w16 is highlighted. The server can also use regular expression matching and other text matching rules to extract annotation information such as [Question Type] Multiple Choice, [Difficulty] Easy, [Textbook Version] People's Education Press, [Year] 2020, and [Region] Pingding County.

[0184] It is understandable that, in contrast to the above Figure 4 Similar to step S204 in the corresponding embodiment, the server can determine whether there is an identifier field in the annotation information extracted from the file name. If there is an identifier field, the server can perform identifier mapping on the identifier field according to the identifier mapping relationship table in the database to obtain the mapping value corresponding to the identifier field, and then the mapping value can also be used as annotation information.

[0185] Step S304: Obtain multimedia sub-data with the target media type from the source service data;

[0186] Specifically, target media types can include formula types and audio / video types. For detailed steps, please refer to [link / reference]. Figure 4 Step S205 in the corresponding embodiment will not be described again here.

[0187] Step S305: Perform format standardization processing on the multimedia sub-data to obtain updated source service data;

[0188] For details, please refer to Figure 4 Step S206 in the corresponding embodiment will not be described again here.

[0189] Step S306: According to the standard data protocol, the updated source business data and the annotation information are assembled to generate standardized data;

[0190] For details, please refer to Figure 4 Step S207 in the corresponding embodiment will not be described again here.

[0191] Step S307: Store the standardized data in a database and establish an association between the standardized data and the set of standard labels in the database.

[0192] For details, please refer to Figure 4 Step S208 in the corresponding embodiment will not be described again here.

[0193] This invention can acquire source business data and identify its data format type. Then, based on the data format type, appropriate methods can be used to extract annotation information from the source business data to obtain corresponding annotation information. Furthermore, it can standardize the formats of various types of multimedia sub-data to obtain updated source business data. The updated source business data and annotation information can then be associated and stored in a database. By associating and storing the updated source business data and annotation information, different business systems can smoothly interact and share the same data, improving data compatibility. Since the extraction of annotation information and the storage of data can be automated, manual costs can be minimized, and the efficiency of data annotation can be improved.

[0194] Please see Figure 7 This is a flowchart illustrating another data processing method provided in an embodiment of the present invention. Figure 7 As shown, for source business data in the education field, the server can obtain the data format type of the source business data. The data pulled from the API interface (also known as the application programming interface) includes some data that cannot be directly displayed or used for algorithm training, such as HTML format data. The server can first perform text cleaning on this type of data. For details, please refer to [link to documentation]. Figure 4 In the corresponding embodiment, step S202 allows for the extraction of tag information (also known as annotation information) from the cleaned source business data. For annotated data (i.e., pre-annotated data), typically annotated Word document data, the server can convert the Word document data into JSON data, and then extract tag information from the converted JSON data. For teaching and research materials and other file data, the server can read file information, such as filename and directory, and then extract tag information from the file information. For details, please refer to [link to documentation]. Figure 5 The corresponding step S303 in the embodiment. During the extraction process, standard tag data needs to be read from a standard tag set pre-stored in a relational database, specifically an MDB database (MDB is a database format used by Microsoft Access). This data is used to match fields in the source business data to generate tag information. The server can further format the data, that is, assemble the source business data and tag information according to a standard data protocol to generate standard data (also called standardized data). For details, please refer to [link to relevant documentation]. Figure 4 Step S207 in the corresponding embodiment.

[0195] Depending on the type of standard data, the server can also choose different types of databases for storage. For example, for file resources such as images, audio / video, courseware, and documents, the server can store them in object storage, facilitating subsequent anti-hotlinking measures. The server can also quickly access object storage and obtain the corresponding resources through resource addresses (such as file URLs) stored in the standard data. The server can also store standard data in an MDB database and associate it with standard tag data in the MDB database. For search scenarios, the standard data can be synchronized to the Elasticsearch search server. Elasticsearch is a distributed, highly scalable, and real-time search and data analysis engine that easily enables the search, analysis, and exploration of large amounts of data. In cloud computing, Elasticsearch achieves real-time search, and is stable, reliable, fast, and easy to install and use. Furthermore, knowledge point fusion can be performed, storing the relationships between knowledge points and chapters in a graph database, specifically Neo4j. Neo4j is a high-performance NoSQL graph database that facilitates the subsequent construction of knowledge networks. Graph databases are non-relational databases that use graph structures for semantic queries and represent and store data using vertices, edges, and attributes, enabling fast graph retrieval operations. Through the above-described data import process, large amounts of resources can be quickly imported into the database.

[0196] It is understandable that the above method is also applicable to source business data in other fields that also require annotation information extraction and format standardization.

[0197] This invention can acquire source business data and identify the data format type of the source business data. Then, according to the above data format type, the corresponding method can be used to extract annotation information from the source business data to obtain the corresponding annotation information. This allows for the rapid batch storage of source business data and annotation information, while minimizing manual costs and improving the efficiency of data annotation.

[0198] Please see Figure 8 This is a schematic diagram of the structure of a data processing device provided in an embodiment of the present invention. Figure 8 As shown, the data processing device 1 can be applied to the above-mentioned... Figure 2 The server 100 in the corresponding embodiment may include: an acquisition module 11, an extraction module 12, a standardization module 13, and a storage module 14.

[0199] The acquisition module 11 is used to acquire source business data and acquire the data format type of the source business data;

[0200] Extraction module 12 is used to extract annotation information from the source business data according to the data format type, so as to obtain the annotation information corresponding to the source business data;

[0201] Standardization module 13 is used to obtain multimedia sub-data with target media type in the source service data, perform format standardization processing on the multimedia sub-data, and obtain updated source service data;

[0202] Storage module 14 is used to associate and store the updated source business data and the annotation information in the database.

[0203] The specific implementation of the acquisition module 11 can be found in the above description. Figure 3 The corresponding step S101 in the embodiment, or as described above, can be found in the above-described embodiment. Figure 4 The corresponding step S201 in the embodiment, or as described above. Figure 5 The specific functional implementation of step S301 in the corresponding embodiment, the extraction module 12, can be found in the above description. Figure 3 The corresponding embodiment of step S102, or you can refer to the above. Figure 4 For steps S203-S204 in the corresponding embodiments, the specific functional implementation of the standardized module 13 can be found above. Figure 3 The corresponding step S103 in the embodiment, or as described above. Figure 4 For steps S205-S206 in the corresponding embodiments, the specific functional implementation of the storage module 14 can be found above. Figure 3 The corresponding embodiment's step S104, or you can refer to the above. Figure 4 Steps S207-S208 in the corresponding embodiments will not be described again here.

[0204] Please see also Figure 8 The extraction module 12 may include: a parsing unit 121 and a webpage extraction unit 122;

[0205] The parsing unit 121 is used to parse the source business data if the data format type is a web page format type, and obtain the valid fields contained in the source business data;

[0206] The webpage extraction unit 122 is used to perform field matching in a standard tag set in the database based on the valid fields; obtain field annotation information based on the valid fields that match the standard tag set; if there is an identifier field in the field annotation information, perform identifier mapping on the identifier field in the field annotation information according to the identifier mapping relationship table in the database to obtain the mapping value corresponding to the identifier field; and determine the field annotation information and the mapping value corresponding to the identifier field as the annotation information corresponding to the source business data.

[0207] The specific implementation of the parsing unit 121 can be found in the above description. Figure 4 The specific functional implementation of step S203 in the corresponding embodiment, the webpage extraction unit 122, can be found above. Figure 4 Step S204 in the corresponding embodiment will not be described again here.

[0208] Please see also Figure 8 The extraction module 12 may include: a file name acquisition unit 123 and a first file extraction unit 124;

[0209] The filename acquisition unit 123 is used to acquire the filename of the source business data if the data format type is a file format type;

[0210] The first file extraction unit 124 is used to extract the annotation information corresponding to the source business data from the file name based on the regular expression if the naming format of the file name conforms to the filtering logic of the regular expression; if the naming format of the file name does not conform to the filtering logic of the regular expression, it traverses the standard tags in the standard tag set in the database and determines the data in the file name that matches the value of the standard tag as the annotation information corresponding to the source business data.

[0211] The specific implementation of the filename acquisition unit 123 can be found in the above description. Figure 5 The specific functional implementation of step S302 in the corresponding embodiment, the first file extraction unit 124, can be found above. Figure 5 Step S303 in the corresponding embodiment will not be described again here.

[0212] Please see also Figure 8 The extraction module 12 may include: a file name acquisition unit 125 and a second file extraction unit 126;

[0213] The filename acquisition unit 125 is used to acquire the filename of the source business data if the data format type is a file format type.

[0214] The second file extraction unit 126 is used to perform word segmentation on the file name to obtain N words; N is a positive integer greater than or equal to 1; perform part-of-speech tagging on the N words to obtain the part-of-speech tags corresponding to the N words; perform named entity recognition on the N words based on the part-of-speech tags to obtain the entity types corresponding to the N words respectively; and determine the word whose entity type is a geographic organization entity type as the annotation information corresponding to the source business data.

[0215] The specific implementation of the filename acquisition unit 125 can be found in the above description. Figure 5 The specific functional implementation of step S302 in the corresponding embodiment, the second file extraction unit 126, can be found above. Figure 5 Step S303 in the corresponding embodiment will not be described again here. Among them, the file name acquisition unit 123 and the file name acquisition unit 125 have the same function and can be merged into one file name acquisition unit; the first file extraction unit 124 and the second file extraction unit 126 can be merged into one file extraction unit.

[0216] Please see also Figure 8 The standardized module 13 may include: a sub-data acquisition unit 131, an image processing unit 132, and a table processing unit 133;

[0217] Sub-data acquisition unit 131 is used to acquire multimedia sub-data with target media type from the source service data;

[0218] Image processing unit 132 is configured to, if the target media type is an image type and the multimedia sub-data is webpage description data of the image type, extract image tags and image resource addresses from the webpage description data, convert the image tags and image resource addresses into standard image data according to the standard media resource protocol, filter out invalid tags in the webpage description data, and determine the filtered webpage description data containing the standard image data as the updated source business data;

[0219] The table processing unit 133 is configured to, if the target media type is a table type and the multimedia sub-data is webpage description data of the table type, extract table tags and table content from the webpage description data, convert the table tags and table content into standard table data according to the standard media resource protocol, filter out invalid tags in the webpage description data, and determine the filtered webpage description data containing the standard table data as the updated source business data.

[0220] The specific functional implementation of the sub-data acquisition unit 131 can be found in the above description. Figure 4 The specific functional implementation of step S205 in the corresponding embodiment, and the image processing unit 132, can be found above. Figure 4 The specific functional implementation of step S206 in the corresponding embodiment, and the table processing unit 133, can be found above. Figure 4 Step S206 in the corresponding embodiment will not be described again here.

[0221] Please see also Figure 8 The standardization module 13 may include: a sub-data acquisition unit 134 and a formula processing unit 135;

[0222] Sub-data acquisition unit 134 is used to acquire multimedia sub-data with target media type from the source service data;

[0223] The formula processing unit 135 is configured to: if the target media type is a formula type, identify the formula format of the multimedia sub-data; if the formula format is a general image format, call the formula recognition interface to convert the multimedia sub-data into standard text data to obtain updated source business data; if the formula format is a metafile format, convert the multimedia sub-data into general image format data, call the formula recognition interface to convert the multimedia sub-data with general image format into standard text data to obtain updated source business data.

[0224] The specific functional implementation of the sub-data acquisition unit 134 can be found in the above description. Figure 4 The specific functional implementation of step S205 in the corresponding embodiment, and the formula processing unit 135, can be found above. Figure 4 Step S206 in the corresponding embodiment will not be described again here.

[0225] Please see also Figure 8 The standardization module 13 may include: a sub-data acquisition unit 136 and an audio / video processing unit 137;

[0226] Sub-data acquisition unit 136 is used to acquire multimedia sub-data with target media type from the source service data;

[0227] The audio and video processing unit 137 is configured to, if the target media type is an audio and video type, obtain the audio and video encoding format corresponding to the multimedia sub-data; if the audio and video encoding format is not a standard audio and video encoding format, convert the multimedia sub-data into standard audio and video data to obtain updated source service data.

[0228] The specific functional implementation of the sub-data acquisition unit 136 can be found in the above description. Figure 4 The specific functional implementation of step S205 in the corresponding embodiment, and the specific function implementation of the audio / video processing unit 137, can be found in the above description. Figure 4 Step S206 in the corresponding embodiment will not be described again here. Among them, the sub-data acquisition unit 131, sub-data acquisition unit 134, and sub-data acquisition unit 136 have the same function and can be combined into one sub-data acquisition unit.

[0229] Please see also Figure 8 The storage module 14 may include: a standardized data generation unit 141 and a storage association unit 142;

[0230] The standardized data generation unit 141 is used to assemble the updated source business data and the annotation information according to the standard data protocol to generate standardized data.

[0231] The storage association unit 142 is used to store the standardized data in a database and establish an association between the standardized data and the set of standard labels in the database.

[0232] The specific functional implementation of the standardized data generation unit 141 can be found in the above description. Figure 4 For the specific functional implementation of step S207 in the corresponding embodiment, the storage association unit 142 can be found above. Figure 4 Step S208 in the corresponding embodiment will not be described again here.

[0233] This invention can acquire source business data and identify its data format type. Then, based on the data format type, appropriate methods can be used to extract annotation information from the source business data to obtain corresponding annotation information. Furthermore, it can standardize the formats of various types of multimedia sub-data to obtain updated source business data. The updated source business data and annotation information can then be associated and stored in a database, enabling smooth interaction and sharing of the same data between different business systems, thus improving data compatibility. Because the extraction of annotation information and the storage of data can be automated, manual costs can be minimized, and the efficiency of data annotation can be improved.

[0234] Please see Figure 9 This is a schematic diagram of another data processing device provided in an embodiment of the present invention. Figure 9 As shown, the data processing device 1000 can correspond to the above. Figure 2 The server 100 in the corresponding embodiment, the data processing device 1000 may include: a processor 1001, a network interface 1003, and a memory 1004. Furthermore, the data processing device 1000 may also include: at least one communication bus 1002. The communication bus 1002 is used to implement communication between these components. The network interface 1003 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1004 may be high-speed RAM or non-volatile memory, such as at least one disk storage device. Optionally, the memory 1004 may also be at least one storage device located remotely from the aforementioned processor 1001. Figure 9 As shown, the memory 1004, which is a computer-readable storage medium, may include an operating system and a network communication module.

[0235] In such Figure 9 In the data processing device 1000 shown, the network interface 1003 can provide network communication functionality to achieve the above-mentioned... Figure 3 , Figure 4 , Figure 5 The description of the data processing method in any corresponding embodiment will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated.

[0236] It should be understood that the data processing apparatus 1000 described in the embodiments of the present invention can perform the foregoing... Figure 3 , Figure 4 , Figure 5 The description of the data processing method in any corresponding embodiment can also be performed as described above. Figure 8 The description of the data processing device 1 in the corresponding embodiments will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated.

[0237] Furthermore, it should be noted that this embodiment of the invention also provides a computer-readable storage medium, which stores a computer program executed by the aforementioned data processing device 1. The computer program includes program instructions, and when the processor executes the program instructions, it can execute the aforementioned... Figure 3 , Figure 4 , Figure 5 The description of the data processing method in any corresponding embodiment is already provided, and therefore will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated. For technical details not disclosed in the computer-readable storage medium embodiments related to this invention, please refer to the description of the method embodiments of this invention.

[0238] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0239] The above description discloses only preferred embodiments of the present invention and should not be construed as limiting the scope of the present invention. Therefore, equivalent variations made in accordance with the claims of the present invention are still within the scope of the present invention.

Claims

1. A data processing method, characterized by, include: Obtain source business data and determine the domain to which the source business data belongs. Different domains have different sets of standard labels stored in the database. The domain to which the source business data belongs includes the education domain. The data format type of the source business data is obtained, including web page format type and file format type, and the source business data includes multimedia sub-data of various media types; If the data format type is a web page format type, then the source business data is parsed to obtain the valid fields contained in the source business data; and the valid fields are matched in the standard tag set corresponding to the education field to obtain the standard tags corresponding to the valid fields. By combining the standard label with the value corresponding to the valid field, the field labeling information of the source business data is obtained; If the field annotation information contains an identifier field, then according to the identifier mapping relationship table in the database, the identifier field is mapped to obtain the mapping value corresponding to the identifier field; the field annotation information and the mapping value corresponding to the identifier field are determined as the annotation information corresponding to the source business data; If the data format type is a file format type, then obtain the file name of the source business data; if the naming format of the file name does not conform to the filtering logic of the regular expression, then traverse the standard tags in the standard tag set in the database, and determine the data in the file name that matches the value of the standard tag as the annotation information corresponding to the source business data; Obtain multimedia sub-data with the target media type from the source service data; If the target media type is an image type, and the multimedia sub-data is webpage description data of the image type, then image tags and image resource addresses are extracted from the webpage description data. According to the standard media resource protocol, the image tags are converted into custom image placeholders, and the image resource addresses are stored according to the image array defined in the standard media resource protocol to obtain standard image data. The webpage description data containing the standard image data is determined as the updated source business data. If the target media type is a table type, and the multimedia sub-data is the webpage description data of the table type, then table tags and table content are extracted from the webpage description data. According to the standard media resource protocol, the table tags are converted into custom table placeholders, and the table content is stored according to the table array defined in the standard media resource protocol and serialized into JSON to obtain standard table data. The webpage description data containing the standard table data is determined as the updated source business data. Based on the data categories of the source business data in the education field, a corresponding standard data protocol is determined, whereby the data categories include questions, resources, and articles; According to the standard data protocol, the updated source business data and the annotation information are assembled to generate standardized data; The standardized data is fused with knowledge points to generate the relationship between knowledge points and chapters; The standard tags in the standard tag set corresponding to the education field include knowledge point tags; The file resources in the standardized data are stored in object storage, and the relationships between the knowledge points and chapters are stored in a graph database; The standardized data is associated with a set of standard labels in the database, and the association is stored in the database to which the set of standard labels belongs; the mapping value corresponding to the identifier field is used to establish the association.

2. The method of claim 1, wherein, The method for obtaining the annotation information corresponding to the source business data further includes: if the naming format of the file name conforms to the filtering logic of the regular expression, then based on the regular expression, extracting the annotation information corresponding to the source business data from the file name.

3. The method of claim 1, wherein, The method for obtaining the annotation information corresponding to the source business data also includes: The filename is segmented into N words; N is a positive integer greater than or equal to 1. Part-of-speech tagging is performed on the N word segments to obtain the part-of-speech tags corresponding to the N word segments; Based on the part-of-speech tagging, named entity recognition is performed on the N word segments to obtain the entity types corresponding to the N word segments respectively. The word segments whose entity type is a geographic organization entity type are identified as the annotation information corresponding to the source business data.

4. The method of claim 1, wherein, The multimedia sub-data is format-standardized to obtain updated source business data, and the method further includes: if the target media type is a formula type, then identifying the formula format of the multimedia sub-data; If the formula format is a general image format, then the formula recognition interface is called to convert the multimedia sub-data into standard text data to obtain the updated source business data; If the formula format is a metafile format, the multimedia sub-data is converted into general image format data, and the formula recognition interface is called to convert the multimedia sub-data with general image format into standard text data to obtain the updated source business data.

5. The method of claim 1, wherein, The step of standardizing the format of the multimedia sub-data to obtain updated source service data further includes: if the target media type is an audio / video type, then obtaining the audio / video encoding format corresponding to the multimedia sub-data; If the audio and video encoding format is not a standard audio and video encoding format, the multimedia sub-data is converted into standard audio and video data to obtain updated source service data.

6. A data processing apparatus, characterized by include: Processor and memory; The processor is connected to a memory, wherein the memory is used to store a computer program, and the processor is used to invoke the computer program to perform the method as described in any one of claims 1-5.

7. A computer readable storage medium characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a processor, perform the method as described in any one of claims 1-5.

Images