Corpus generation method, device, electronic device and storage medium

Abstract

The present disclosure provides a corpus generation method, device, electronic device and storage medium, which relate to the field of computer technology, specifically to the field of artificial intelligence technology such as AI medicine, knowledge graph, deep learning, etc. The specific implementation scheme is: obtaining reference data and corpus demand information, wherein the reference data has a corresponding reference data type, and according to the reference data type and the corpus demand information, determining the target process description information corresponding to the reference data, and then according to the reference data type and the target process description information, determining the target process content for processing the reference data, and processing the reference data according to the target process content to obtain the target corpus, thereby being able to effectively improve the corpus generation efficiency while meeting the corpus generation requirements.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, specifically to the field of artificial intelligence technologies such as AI medicine, knowledge graphs, and deep learning, and specifically to a corpus generation method, device, electronic device, and storage medium. Background Art

[0002] With the development of artificial intelligence technology, corpus is increasingly being used in business scenarios such as large model training and digital human training.

[0003] The generation effect and efficiency of corpus have an important impact on the business effect in downstream corpus application scenarios. Therefore, how to efficiently provide corpus application scenarios with corpus that meets their business needs has become a key research direction. Summary of the invention

[0004] The present invention provides a corpus generation method, device, electronic device and storage medium.

[0005] According to a first aspect of the present disclosure, a corpus generation method is provided, comprising: acquiring reference data and corpus requirement information, wherein the reference data has a corresponding reference data type; determining target process description information corresponding to the reference data according to the reference data type and the corpus requirement information; determining target process content for processing the reference data according to the reference data type and the target process description information; and processing the reference data according to the target process content to obtain a target corpus.

[0006] According to a second aspect of the present disclosure, a corpus generation device is provided, comprising: an acquisition module for acquiring reference data and corpus requirement information, wherein the reference data has a corresponding reference data type; a first determination module for determining target process description information corresponding to the reference data based on the reference data type and the corpus requirement information; a second determination module for determining target process content for processing the reference data based on the reference data type and the target process description information; and a processing module for processing the reference data based on the target process content to obtain a target corpus.

[0007] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0008] at least one processor; and

[0009] a memory communicatively connected to at least one processor; wherein,

[0010] The memory stores instructions that can be executed by at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the corpus generation method as described in the first aspect.

[0011] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, where the computer instructions are used to cause a computer to execute the corpus generation method of the first aspect.

[0012] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising computer instructions, which, when executed by a processor, implement the steps of the corpus generation method of the first aspect.

[0013] The corpus generation method, device, electronic device and storage medium provided by the present disclosure have the following beneficial effects:

[0014] In the disclosed embodiment, reference data and corpus demand information are obtained, wherein the reference data has a corresponding reference data type, and according to the reference data type and the corpus demand information, target process description information corresponding to the reference data is determined, and then according to the reference data type and the target process description information, the target process content for processing the reference data is determined, and the reference data is processed according to the target process content to obtain the target corpus. Since the overall process of corpus generation is split into multiple target processes, and the target process content for implementing each target process is determined according to the reference data type and the corpus demand information, the corpus generation efficiency can be effectively improved while meeting the corpus generation requirements, and since the overall process of corpus generation is split into multiple target processes, flexible adjustment of the corpus generation process can be supported.

[0015] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0017] Figure 1 It is a flowchart of a corpus generation method provided according to an embodiment of the present disclosure;

[0018] Figure 2 is a flowchart of a corpus generation method provided according to another embodiment of the present disclosure;

[0019] Figure 3 is a flowchart of a corpus generation method provided according to another embodiment of the present disclosure;

[0020] Figure 4 is a flowchart of a corpus generation method provided according to another embodiment of the present disclosure;

[0021] Figure 5is a flowchart of a corpus generation method provided according to another embodiment of the present disclosure;

[0022] Figure 6 is a flowchart of a corpus generation method provided according to another embodiment of the present disclosure;

[0023] Figure 7 is a structural schematic diagram of a corpus generation device provided according to an embodiment of the present disclosure;

[0024] Figure 8 It is a block diagram of an electronic device used to implement the corpus generation method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0026] The embodiments disclosed herein relate to artificial intelligence technology fields such as AI medicine, deep learning, and knowledge graphs.

[0027] AI healthcare refers to the application of artificial intelligence technology and big data services in the medical industry based on the Internet, through the construction of infrastructure and data collection, to improve the diagnostic efficiency and service quality of the medical industry.

[0028] Artificial Intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence.

[0029] Deep learning is the process of learning the inherent laws and representation levels of sample data. The information obtained in the learning process is of great help in interpreting data such as text, images, and sounds. The ultimate goal of deep learning is to enable machines to have analytical learning capabilities like humans and to be able to recognize data such as text, images, and sounds.

[0030] Knowledge graph is a technical method that uses a graph model to describe knowledge and model the relationship between all things in the world. It is a modern theory that achieves the purpose of multidisciplinary integration by combining the theories and methods of applied mathematics, graphics, information visualization technology, information science and other disciplines with methods such as metrology citation analysis and co-occurrence analysis, and using visual graphs to vividly display the core structure, development history, frontier fields and overall knowledge architecture of the discipline.

[0031] It should be noted that in the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0032] The following describes the corpus generation method, device, electronic device, and storage medium of the embodiments of the present disclosure with reference to the accompanying drawings.

[0033] It should be noted that the executor of the corpus generation method of this embodiment is a corpus generation device, which can be implemented by software and / or hardware. The device can be configured in an electronic device, which may include but is not limited to a terminal, a server, etc.

[0034] Figure 1 It is a flowchart of a corpus generation method provided according to an embodiment of the present disclosure.

[0035] like Figure 1 As shown, the corpus generation method includes:

[0036] S101: Acquire reference data and corpus demand information, wherein the reference data has a corresponding reference data type.

[0037] Among them, reference data refers to data used to generate demand expectations in an expectation generation scenario. The reference data can be, for example, medical data in a doctor's expectation generation scenario (for example, medical papers, medical literature, diagnosis and treatment videos, etc.), or the reference data can also be, for example, data in a conversation expectation generation scenario (for example, conversation videos, conversation texts, etc.), and there is no limitation on this.

[0038] Among them, the expected demand information can be used to describe the demand for expected generation results and / or corpus generation process in actual business scenarios. The demand can be, for example, generating question and answer corpus, the corpus does not contain modal particles, the corpus only contains scenario-related keywords, the corpus generation process requires less time, etc., and there is no restriction on this.

[0039] Among them, the reference data can be divided into different data types according to different data classification criteria. For example, the data can be divided into: "text type, video type, audio type, image type" according to the data representation form, and the data can also be divided into: "A platform type, B platform type, etc." according to the platform to which the data belongs. There is no restriction on this.

[0040] In the embodiment of the present disclosure, the corpus generation method can be executed by the corpus generation system, that is, in the embodiment of the present disclosure, the corpus generation system can provide a data transmission interface, and obtain data from other external platforms as reference data through the data transmission interface, and obtain the user's corpus demand information this time through the data transmission interface, or, a corresponding operation interface can be configured for the corpus generation system, and the user can provide corpus generation demand information to the corpus generation system through the operation interface, and there is no limitation on this.

[0041] In some embodiments, the corpus generation system can also support the storage of in-process data and historical corpus generated during the historical corpus generation process, and use the in-process data and historical corpus generated during the historical corpus generation process as reference data in the current corpus generation process, thereby fully leveraging the data value and reducing data acquisition costs to a certain extent.

[0042] S102: Determine target process description information corresponding to the reference data according to the reference data type and corpus requirement information.

[0043] Among them, in the overall corpus generation process of generating the target corpus based on the reference corpus, the information used to describe the target process of the overall corpus generation is the target process description information. In a corpus generation scenario, the number of target processes corresponding to the reference data can be multiple, that is, for example, the overall process of a complete corpus generation can be used as the target process, or the overall process of a complete corpus generation can be split into multiple target processes. For example, the overall process of a complete corpus generation can be split into multiple target processes such as "data collection process-key information extraction process-corpus understanding process".

[0044] Among them, the target process description information used to describe the target process can be, for example: the target process name (for example, data collection process), the arrangement order of the target process relative to the overall corpus generation process (for example, data collection process-order 1, key information extraction process-order 2, corpus understanding process-order 3), etc., without limitation.

[0045] In some embodiments, the target process description information corresponding to the reference data is determined based on the reference data type and corpus requirement information. The target process description information can be self-configured to meet the corpus requirement information, and can support processing the reference data of the reference data type to generate a corpus processing flow for the target corpus, and split the overall corpus processing flow into multiple target processes, and use the names of the multiple target processes and the execution order of each target process together as the target process description information.

[0046] For example, the data type of the reference data is a video data type, and the corpus requirement information is to generate a target corpus for use in a doctor scenario. Then, it can be determined that the multiple target processes corresponding to the reference data are: "video data acquisition process-text data recognition process-text data extraction process-doctor corpus screening process". Therefore, the names of multiple target processes and the execution order of each target process can be used together as the target process description information, and there is no restriction on this.

[0047] In other embodiments, target process description information corresponding to the reference data is determined based on the reference data type and corpus requirement information, and multiple initial processes can be pre-acquired, each initial process having a corresponding initial data type and initial corpus requirement information. Then, partial initial processes corresponding to the initial data type that matches the reference data type can be determined, and then the initial processes corresponding to the initial corpus requirement information that matches the corpus requirement information in the aforementioned partial initial processes can be used as target processes. Then, the target process name and the execution order of each target process can be determined together as the target process description information, and there is no limitation on this.

[0048] S103: Determine the target process content for processing the reference data according to the reference data type and the target process description information.

[0049] In the disclosed embodiment, the target process content includes at least one of the following: a rule for implementing the target process, a script file for implementing the target process, a service for implementing the target process, and a component for implementing the target process.

[0050] In the disclosed embodiment, since the target process content is pre-compiled, the components, script files, services and components used to implement the target process can be directly obtained and used in the corpus generation business, thereby effectively improving the corpus generation efficiency.

[0051] In some embodiments, the corpus generation system may locally pre-store multiple initial process contents, that is, it may pre-acquire multiple initial process contents and determine the initial data type and initial process description information corresponding to each initial process content. Then, the initial process contents may be stored in the corpus generation system, and the initial data type and initial process description information corresponding to each initial process content may be configured as index items of the corresponding initial process contents. Then, after determining the reference data type and the target process description information, the target process content may be obtained by indexing from the corpus generation system according to the reference data type and the target process description information.

[0052] In other embodiments, the target process content for processing the reference data is determined based on the reference data type and the target process description information. The initial data type and the initial process description information may also be used as sample data, and the initial process content corresponding to the initial data type and the initial process description information may be used as labeled data corresponding to the sample data. Then, the process content determination model may be trained based on the sample data and the labeled data corresponding to the sample data until the model converges. Then, the reference data type and the target process description information may be input into the process content determination model to obtain the target process content output by the process content determination model. There is no limitation on this.

[0053] S104: Process the reference data according to the target process content to obtain the target corpus.

[0054] After determining the target process content of the target process for processing the reference data according to the reference data type and the target process description information, the embodiment of the present disclosure can process the reference data according to the target process content to obtain the target process content.

[0055] That is to say, in the embodiments of the present disclosure, the reference data can be processed based on a script file, external service, component, or rule corresponding to the reference data to obtain a target corpus that matches the corpus demand information, thereby effectively meeting the corpus generation needs in actual business scenarios. Moreover, since the acquisition of the target process content does not require a lot of time and computing cost, the corpus generation efficiency can be effectively improved.

[0056] In the disclosed embodiment, there are multiple target process contents, and different target process contents have corresponding execution orders.

[0057] In the embodiments of the present disclosure, in the complete process of corpus generation, it can be obtained by processing multiple target process contents, that is, the complete process of corpus generation can be split into multiple sub-processes. For example, the complete process of corpus generation can be split into: "collection process-key information extraction process-corpus understanding process", etc. Each sub-process can have corresponding target process content, and the target process content can be used to implement the corpus processing task of the corresponding sub-process.

[0058] In the disclosed embodiments, different target process contents have corresponding execution orders, which can be the process order of the sub-process corresponding to the target process content. Taking the above-mentioned complete process of corpus generation as divided into: "collection process-key information extraction process-corpus understanding process" as an example, the execution order corresponding to the target process content (collection plug-in) of the collection process is 1, the execution order corresponding to the target process content (key information extraction plug-in) of the key information extraction process is 2, and the execution order corresponding to the target process content (corpus understanding plug-in) of the corpus understanding process is 3, and there is no restriction on this.

[0059] That is, in the embodiments of the present disclosure, a process plug-in design may be used in a corpus generation method, that is, the complete process of corpus generation may be split into multiple sub-processes, and a plug-in for implementing each sub-process may be determined, thereby controlling the operation of each plug-in in turn based on the execution order corresponding to each plug-in to realize the processing of reference data and obtain the target corpus.

[0060] Optionally, in some embodiments, the reference data is processed according to the target process content to obtain the target corpus, and the reference data may be processed according to multiple target process contents and multiple execution orders to obtain the target corpus.

[0061] That is to say, in the embodiments of the present disclosure, the corpus generation system may control the target process content of the corresponding execution order based on a pre-compiled control program, and process the reference data according to the execution order to fully implement the overall corpus generation process and obtain the target corpus.

[0062] In the disclosed embodiments, by using a process plug-in design in the corpus generation method, that is, the complete process of corpus generation can be split into multiple sub-processes, and a plug-in for implementing each sub-process can be determined, so that based on the execution order corresponding to each plug-in, the operation of each plug-in is controlled in turn to achieve phased processing of reference data, thereby supporting flexible adjustments at various stages of corpus generation. For example, after the target process of the previous stage is completed, the execution of the target process of the next stage can be suspended, or the target process of the next stage can be adjusted.

[0063] In the disclosed embodiment, after processing the reference data according to the target process content to obtain the target corpus, the corpus evaluation module in the corpus generation system can perform corpus evaluation on the generated target corpus, that is, it can perform multi-dimensional quality evaluation on the target corpus, including accuracy, effectiveness, completeness and uniqueness. By introducing these evaluation controls, it is ensured that the quality of the output target corpus meets the standards.

[0064] In the embodiment of the present disclosure, after evaluating the target corpus and determining that the target corpus meets the standards, the corpus storage module in the corpus generation system can store the corpus that meets the quality assessment standards, that is, the target that meets the quality assessment standards can be stored in the database for use in actual business scenarios.

[0065] In the disclosed embodiment, by obtaining reference data and corpus demand information, wherein the reference data has a corresponding reference data type, and according to the reference data type and the corpus demand information, determining the target process description information corresponding to the reference data, and then determining the target process content for processing the reference data according to the reference data type and the target process description information, and processing the reference data according to the target process content to obtain the target corpus, since the overall process of corpus generation is split into multiple target processes, and the target process content for implementing each target process is determined according to the reference data type and the corpus demand information, it is possible to effectively improve the corpus generation efficiency while meeting the corpus generation requirements, and since the overall process of corpus generation is split into multiple target processes, it is possible to support flexible adjustment of the corpus generation process.

[0066] For example, Figure 2 As shown, Figure 2 FIG. 1 is a flow chart of a corpus generation method provided according to another embodiment of the present disclosure, the corpus generation method comprising:

[0067] S201: Obtain reference data and corpus requirement information.

[0068] S202: Determine target process description information.

[0069] Specifically, the target process description information corresponding to the reference data may be determined as (collection process-extraction process-understanding process) based on the reference data type and corpus requirement information of the reference data.

[0070] S203: Determine a target plug-in from the plug-in library.

[0071] Specifically, a target plug-in (collection plug-in, extraction plug-in, understanding plug-in) for processing the reference data of the reference type may be determined from the plug-in library according to the reference data type of the reference data and the target process description information determined in step S202.

[0072] S204: Control the target plug-ins to execute sequentially.

[0073] Specifically, the target plug-ins corresponding to the reference data of the reference data type may be controlled by the executor to be executed sequentially. For example, the target plug-ins corresponding to the reference data of the reference data type A may be determined to be: "target plug-in 1 and target plug-in 2", that is, the executor may control the target plug-in 1 to process the reference data of the reference data type A to obtain a first data processing result, and then control the target plug-in 2 to process the first data processing result to obtain the target corpus. For example, the target plug-ins corresponding to the reference data of the reference data type B may be determined to be: "target plug-in 3 and target plug-in 2", that is, the executor may control the target plug-in 3 to process the reference data of the reference data type B to obtain a second data processing result, and then control the target plug-in 2 to process the second data processing result.

[0074] S205: Generate target corpus.

[0075] Specifically, the processing result of the target plug-in 2 in S204 may be used as the target corpus.

[0076] Figure 3 is a flowchart of a corpus generation method provided according to another embodiment of the present disclosure;

[0077] like Figure 3 As shown, the corpus generation method includes:

[0078] S301: Acquire reference data and corpus requirement information, wherein the reference data has a corresponding reference data type.

[0079] The specific description of S301 can be found in the above embodiment, which will not be repeated here.

[0080] S302: Determine at least one candidate process description information corresponding to the reference data type.

[0081] The candidate process information may be process description information obtained through preliminary screening based on the reference data type.

[0082] That is to say, in the embodiments of the present disclosure, candidate process description information corresponding to the reference data type can be obtained by screening out from multiple initial process description information, that is, each initial process description information can respectively have a corresponding initial data type, and an initial data type that is the same as the reference data type can be determined, and the initial process description information corresponding to the same initial data type can be used as at least one candidate process description information corresponding to the reference data type.

[0083] Optionally, in some embodiments, determining at least one candidate process description information corresponding to a reference data type may be by obtaining a first relationship table, wherein the first relationship table includes: multiple candidate data types, and at least one candidate process description information corresponding to each candidate data type, and determining a candidate data type that matches the reference data type from the multiple candidate data types, and using at least one candidate process description information corresponding to the matching candidate data type as at least one candidate process description information corresponding to the reference data type.

[0084] The first relationship table includes: a plurality of candidate data types and at least one candidate process description information corresponding to each candidate data type.

[0085] For example, the first relationship table may include: A platform type-candidate process description information a, B platform type-candidate process description information b, and there is no limitation to this.

[0086] That is to say, in the embodiments of the present disclosure, a candidate data type that is the same as a reference data type can be queried from the first relationship table, and at least one candidate process description information corresponding to the same candidate data type in the first relationship table can be used as at least one candidate process description information corresponding to the reference data type. Thus, at least one candidate process description information corresponding to the reference data type can be accurately queried from the first relationship table.

[0087] S303: Determine target process description information from at least one candidate process description information according to the corpus requirement information.

[0088] After determining at least one candidate process description information corresponding to the reference data type, the embodiment of the present disclosure can perform a secondary screening on the at least one candidate process description information determined above according to the corpus requirement information, so as to accurately determine the target process description information that is compatible with the corpus requirement information from the at least one candidate process description information.

[0089] In the embodiment of the present disclosure, each candidate process description information has a corresponding candidate processing type.

[0090] Optionally, in some embodiments, target process description information is determined from at least one candidate process description information based on corpus requirement information. The data processing type may be determined based on the corpus requirement information, and a candidate processing type matching the data processing type is determined from multiple candidate processing types, and the candidate process description information corresponding to the matching candidate processing type is used as the target process description information.

[0091] The candidate processing types include, for example, video data processing type, text data processing type, etc., and there is no limitation on this.

[0092] That is to say, in the embodiments of the present disclosure, the data processing type can be determined based on the corpus requirement information. For example, when the corpus requirement information indicates a requirement to process text data, it can be determined that the data processing type is a text processing type, and the text processing type is determined from multiple candidate processing types, and the candidate process description information corresponding to the matching candidate processing type is used as the target process description information. Thus, the target process description information that matches the data processing type indicated by the corpus requirement information can be accurately determined from multiple candidate process description information.

[0093] Optionally, in some embodiments, the data processing type is determined based on the corpus requirement information, and the corpus requirement information may be input into a processing type determination model to obtain the data processing type output by the processing type determination model.

[0094] Among them, the processing type determination model has learned the mapping relationship between corpus requirement information and data processing type.

[0095] That is to say, the processing type determination model can be a model that is pre-trained and converged based on the mapping relationship between corpus requirement information and data processing type. The model can be a large model, and there is no limitation on this.

[0096] In the disclosed embodiments, the corpus requirement information may be input into a processing type determination model, which processes the corpus requirement information and outputs a data processing type that is adapted to the corpus requirement information. Since the data processing type is determined by a joint data processing type determination model, the efficiency of determining the data processing type can be effectively improved, and the accuracy of determining the data type can be improved.

[0097] S304: Determine the target process content for processing the reference data according to the reference data type and the target process description information.

[0098] S305: Process the reference data according to the target process content to obtain the target corpus.

[0099] The specific description of S304 - S305 can be found in the above embodiment, which will not be repeated here.

[0100] In the disclosed embodiment, by obtaining reference data and corpus requirement information, wherein the reference data has a corresponding reference data type, after determining at least one candidate process description information corresponding to the reference data type, the at least one candidate process description information determined above can be secondary screened according to the corpus requirement information to accurately determine the target process description information that matches the corpus requirement information from the at least one candidate process description information, and then determine the target process content for processing the reference data according to the reference data type and the target process description information, and process the reference data according to the target process content to obtain the target corpus, thereby effectively improving the corpus generation efficiency, and since the overall corpus generation process is split into multiple target processes, it can support flexible adjustment of the corpus generation process.

[0101] Figure 4 is a flowchart of a corpus generation method provided according to another embodiment of the present disclosure;

[0102] like Figure 4 As shown, the corpus generation method includes:

[0103] S401: Acquire reference data and corpus requirement information, wherein the reference data has a corresponding reference data type.

[0104] S402: Determine target process description information corresponding to the reference data according to the reference data type and corpus requirement information.

[0105] The specific description of S401 - S402 can be found in the above embodiment, which will not be repeated here.

[0106] S403: Determine at least one candidate process content according to the reference data type.

[0107] Each candidate process content has corresponding candidate process description information.

[0108] That is to say, in the embodiment of the present disclosure, multiple process contents may be screened according to the reference data type to preliminarily screen out at least one candidate process content from the multiple process contents.

[0109] S404: Determine candidate process description information that matches the target process description information from at least one candidate process description information.

[0110] In the embodiment of the present disclosure, after determining at least one candidate process content according to the reference data type, the candidate process description information that matches the target process description information can be further screened from the at least one candidate process description information, thereby being able to accurately determine the candidate process description information that matches the target process description information from the at least one candidate process description information.

[0111] S405: Taking the candidate process content corresponding to the matching candidate process description information as the target process content.

[0112] After determining the candidate process description information that matches the target process description information from at least one candidate process description information, the embodiment of the present disclosure can use the candidate process content corresponding to the matching candidate process description information as the target process content. Thus, the candidate process content can be secondary screened based on the reference data type and the target process description information to accurately determine the target process content that matches the reference data type and the target process description information.

[0113] In the embodiment of the present disclosure, each candidate process content is in a first state, wherein the first state indicates that the corresponding candidate process content is not enabled.

[0114] Optionally, in some embodiments, taking the candidate process content corresponding to the matching candidate process description information as the target process content may be to configure the candidate process content corresponding to the matching candidate process description information from a first state to a second state, wherein the second state indicates that the corresponding candidate process content has been enabled, and taking the candidate process content configured to the second state as the target process content.

[0115] That is to say, in the embodiments of the present disclosure, the candidate process content corresponding to the matching candidate process description information can be configured from an unenabled state to an enabled state, and when it is determined that the state of the candidate process is an enabled state, the candidate process content configured to the second state is used as the target process content. Since the target process content has been configured to be an enabled state, the target process content can be directly controlled to run during the subsequent execution of the corpus generation method to ensure the continuity of the corpus generation process.

[0116] S406: Process the reference data according to the target process content to obtain the target corpus.

[0117] The specific description of S406 can be found in the above embodiment, which will not be repeated here.

[0118] In the disclosed embodiment, by acquiring reference data and corpus demand information, wherein the reference data has a corresponding reference data type, and according to the reference data type and the corpus demand information, determining the target process description information corresponding to the reference data, and then determining at least one candidate process content according to the reference data type, and after determining the candidate process description information matching the target process description information from at least one candidate process description information, the candidate process content corresponding to the matching candidate process description information can be used as the target process content, thereby, the candidate process content can be secondary screened based on the reference data type and the target process description information to accurately determine the target process content that matches the reference data type and the target process description information, and processing the reference data according to the target process content to obtain the target corpus, thereby being able to effectively improve the corpus generation efficiency while meeting the corpus generation requirements, and since the overall corpus generation process is split into multiple target processes, it can support flexible adjustment of the corpus generation process.

[0119] Figure 5 It is a flowchart of a corpus generation method provided according to another embodiment of the present disclosure.

[0120] like Figure 5 As shown, the corpus generation method includes:

[0121] S501: Acquire reference data and corpus requirement information, wherein the reference data has a corresponding reference data type.

[0122] S502: Determine target process description information corresponding to the reference data according to the reference data type and corpus requirement information.

[0123] S503: Determine the target process content for processing the reference data according to the reference data type and the target process description information.

[0124] The specific description of S501 - S503 can be found in the above embodiment, which will not be repeated here.

[0125] S504: Process the reference data according to the target flow content of the first execution order to obtain a first data processing result.

[0126] Among them, the first execution order belongs to multiple execution orders, and the target process content of the first execution order needs to be executed first in the overall corpus generation process.

[0127] Taking the above-mentioned overall corpus generation process divided into: "data collection process-key information extraction process-corpus understanding process" as an example, the target process content of the first execution order can be a script file for implementing the data collection process.

[0128] That is to say, in the embodiment of the present disclosure, the corpus generation system can collect corresponding reference data based on the pre-compiled main control program to control the script file used to implement the data collection process, and use the collected corresponding reference data as the first data processing result.

[0129] S505: Process the first data processing result according to the target flow content of the second execution order to obtain a second data processing result.

[0130] The second execution order belongs to a plurality of execution orders, the second execution order is after the first execution order, and the second execution order is adjacent to the first execution order.

[0131] Taking the above-mentioned overall corpus generation process divided into: "data collection process-key information extraction process-corpus understanding process" as an example, the target process content of the second execution order can be a script file for implementing the key information extraction process.

[0132] That is to say, in the embodiment of the present disclosure, after the main control program controls the target process content of the first execution order to process the reference data to obtain the first data processing result, it can control the target process content of the second execution order to process the first data processing result to obtain the second data processing result.

[0133] For example, the corpus generation system can control a script file for implementing a key information extraction process to process reference data based on a pre-compiled main control program, so as to extract corresponding key information from the reference data, and use the extracted key information as the second data processing result.

[0134] Optionally, in some embodiments, after processing reference data according to the target process content of the first execution order to obtain a first data processing result, and processing the first data processing result according to the target process content of the second execution order to obtain a second data processing result, a first association relationship can be established between the target process description information corresponding to the first execution order and the first data processing result, and / or a second association relationship can be established between the target process description information corresponding to the second execution order and the second data processing result, and the first association relationship and / or the second association relationship can be saved.

[0135] Among them, there may be a corresponding first association relationship between the target process description information corresponding to the first execution order and the first data processing result, and accordingly, there may be a corresponding second association relationship between the target process description information corresponding to the second execution order and the second data processing result.

[0136] That is to say, in the embodiments of the present disclosure, after the target process content of each execution order is executed, the data processing results and corresponding process description information generated by the target process content of each execution order can be associated and stored, so that in the subsequent corpus generation process, it can support the calling of process data in the historical corpus generation process, so that the process data in the historical corpus generation process can be reread and utilized, so as to give full play to its data value.

[0137] For example, the target process description information corresponding to the first execution order is execution order identifier 1, so that an associated index relationship between the execution order identifier 1 and the first data processing result can be established as the first associated relationship, and based on the first associated relationship, the execution order identifier 1 and the first data processing result are stored in the key-value database in the corpus generation system. Therefore, in subsequent corpus generation scenarios, it is possible to support convenient indexing from the key-value database based on the execution order identifier to obtain the data processing results associated with the target process of the execution order, thereby effectively improving the efficiency of determining the data processing results.

[0138] Optionally, in some embodiments, processing the first data processing result according to the target process content corresponding to the second execution order to obtain the second data processing result can be performed by obtaining content modification information corresponding to the target process content of the second execution order, and processing the target process content of the second execution order according to the content modification information to obtain new target process content, and processing the first data processing result according to the new target process content to obtain the second data processing result.

[0139] The content modification information may be used to modify the target process content of the second order. The content modification information may be, for example, a new script file, or the content modification information may be a script command for partially adjusting the script file of the second order. There is no limitation on this.

[0140] That is to say, in the embodiments of the present disclosure, it is possible to support adjustment of the target process of a certain link in the corpus generation process, that is, when a process adjustment instruction is received, the content modification information is determined, and the target process content of the second execution order is processed based on the content modification information to obtain a new target process content. Then, the main control program in the corpus generation system can control the new second-order target process content to process the first data processing result to obtain the second data processing result after the first-order target process content is executed and the first data processing result is obtained.

[0141] For example, taking the above-mentioned overall corpus generation process being divided into: "data collection process-corpus understanding process" as an example, after the data collection process is executed, a process adjustment instruction is received, and the process adjustment instruction indicates to change the corpus understanding process to a question and answer sentence understanding process, so that the content modification information can be determined as: a script file for implementing the question and answer sentence understanding process, and then, the target process content of the second execution order can be replaced with a script file for implementing the question and answer sentence understanding process, and the script file for implementing the question and answer sentence understanding process can be controlled to process the first data processing result to obtain a question and answer corpus that is adapted to actual needs.

[0142] In the disclosed embodiment, by obtaining content modification information corresponding to the target process content of the second execution order, and processing the target process content of the second execution order according to the content modification information to obtain new target process content, and processing the first data processing result according to the new target process content to obtain the second data processing result, the corpus generation process can be adaptively adjusted in a timely manner based on changes in corpus requirements in actual business scenarios, and the adjustment process is relatively convenient.

[0143] Optionally, in some embodiments, after obtaining the content modification information corresponding to the target process content of the second execution order, the target process content of the second execution order may be configured from the second state to the first state.

[0144] The first state indicates that the target process content is not enabled, and the second state indicates that the target process content is enabled.

[0145] That is to say, in the embodiment of the present disclosure, after obtaining the content modification information corresponding to the target process content of the second execution order, the target process content of the second execution order can also be switched from an enabled state to an unenabled state, and in this state, the target process content of the second execution order is adjusted based on the content modification information, thereby avoiding the waste of resources caused by the erroneous execution of the target process content of the second execution order.

[0146] Optionally, in some embodiments, after the target process content of the second execution order is processed according to the content modification information to obtain new target process content, the new target process content may be configured from being in the first state to being in the second state.

[0147] That is to say, in the embodiment of the present disclosure, after the target process content of the second execution order is processed according to the content modification information to obtain the new target process content, the new target process content is switched from an unenabled state to an enabled state, so as to continue to trigger the new target process content to run.

[0148] S506: Update the first execution order according to the second execution order until the reference data has been processed based on each target process content, and use the data processing result obtained by processing the reference data based on the target process content of the last execution order among the multiple execution orders as the target corpus.

[0149] That is to say, in the embodiments of the present disclosure, when each target process content is executed, the data processing result obtained by processing the reference data based on the target process content of the last execution order among multiple execution orders can be used as the target corpus.

[0150] For example, taking the above-mentioned overall corpus generation process being divided into: "data collection process-corpus understanding process" as an example, when the target process contents corresponding to the data collection process and the corpus understanding process are both executed, the data processing result obtained by processing the reference data with the target process content of the corpus understanding process in the last execution order is used as the target corpus. In this way, it can be ensured that each target process content is executed sequentially in the execution order to realize a complete corpus generation process.

[0151] In the embodiments of the present disclosure, the corpus generation system and the corpus generation method can be used in other business scenarios, for example:

[0152] (1) In the research and development of natural language processing, the corpus generation system can process large-scale corpus data, support multi-channel data source access, and has protocol standardization capabilities. It can become a research and development platform in the field of natural language processing. Researchers and developers can use the system's process plug-in characteristics to quickly build and test different text processing rules to improve text understanding and processing capabilities.

[0153] (2) In the context of intelligent customer service and robots, the corpus generation system’s ability to understand and manage data makes it an ideal foundation for intelligent customer service and virtual robots. It can be used to analyze user questions, answer questions, and even provide personalized recommendations based on historical data.

[0154] (3) In the scenarios of public opinion analysis and social media monitoring, the data scale support and data source access protocol of the corpus generation system make it very suitable for processing large amounts of social media data and news articles. Through the plug-in process, processing rules for specific events, topics or emotions can be defined, thus realizing powerful public opinion analysis and social media monitoring functions.

[0155] (4) In text classification and information retrieval scenarios, the processing power and plug-in process of the corpus generation system can help you build efficient text classifiers and information retrieval systems.

[0156] S507: Determine the scene category information corresponding to the target corpus according to the corpus requirement information.

[0157] Among them, the scene category information is used to describe the category of the scene in which the target corpus is used. The scenes in which the target corpus is used include: medical large model training scenes, digital human training scenes, etc., and there is no restriction on this.

[0158] In the embodiments of the present disclosure, the application scenario of the target corpus corresponding to the reference data may be pre-configured in the corpus generation system. For example, it may be configured that fields 1-5 in the target corpus corresponding to the reference data are used for the medical large model training scenario, and fields 6-10 in the target corpus corresponding to the reference data are used for the digital human training scenario, and there is no limitation on this.

[0159] In the embodiments of the present disclosure, after processing the reference data to obtain the target corpus, the target corpus can be provided to the corresponding application scenario based on the application scenario of the target corpus pre-configured corresponding to the reference data, thereby being able to meet the corpus requirements in different application scenarios in a timely manner.

[0160] In an embodiment of the present disclosure, by obtaining reference data and corpus demand information, wherein the reference data has a corresponding reference data type, and according to the reference data type and the corpus demand information, determining the target process description information corresponding to the reference data, and then determining the target process content for processing the reference data according to the reference data type and the target process description information, and then processing the reference data according to the target process content of the first execution order to obtain a first data processing result, and processing the first data processing result according to the target process content of the second execution order to obtain a second data processing result, and then updating the first execution order according to the second execution order until the reference data has been processed based on each target process content, and the data processing result obtained by processing the reference data based on the target process content of the last execution order in the multiple execution orders is used as the target corpus, thereby ensuring that each target process content is executed sequentially according to the execution order to realize a complete corpus generation process, and then determining the scenario category information corresponding to the target corpus according to the corpus demand information, so that the target corpus can be provided to the corresponding application scenario based on the pre-configured application scenario of the target corpus corresponding to the reference data, so that the corpus requirements in different application scenarios can be met in a timely manner.

[0161] Figure 6 It is a flowchart of a corpus generation method provided according to another embodiment of the present disclosure.

[0162] like Figure 6 As shown, the corpus generation method includes:

[0163] S601: Acquire reference data and corpus requirement information, wherein the reference data has a corresponding reference data type.

[0164] The specific description of S601 can be found in the above embodiment, which will not be repeated here.

[0165] S602: When the corpus demand scenario is a medical scenario and the reference data type is a text type, obtain a reference text corresponding to the medical scenario.

[0166] In the disclosed embodiment, the corpus demand information includes: a corpus demand scenario, where the corpus demand scenario refers to an application scenario of the target corpus generated by the corpus generation method, and the application scenario may be, for example, a medical scenario, which is not limited.

[0167] Among them, when the corpus demand scenario is a medical scenario, and when the reference data type of the reference data is a text type, the reference data can be, for example, medical papers, drug prescriptions, drug instructions, etc., and there is no limitation on this.

[0168] Among them, the reference text corresponding to the medical scene can be, for example, medical vocabulary in the medical scene (for example, professional medical vocabulary such as disease names and drug names), or medical sentences and paragraphs in the medical scene (for example, diagnosis and treatment discourse sentences, symptom description sentences, treatment method sentences, etc.), without any limitation.

[0169] That is to say, when it is determined that the corpus demand scenario is a medical scenario, professional medical vocabulary and medical sentences used for the corpus scenario can be obtained, and the aforementioned professional medical vocabulary and medical sentences can be used as reference texts corresponding to the medical scenario.

[0170] S603: Identify at least one text paragraph from the reference data, and determine a text paragraph that matches the reference text from the at least one text paragraph.

[0171] The reference data may consist of multiple text paragraphs.

[0172] Therefore, in the embodiments of the present disclosure, a text paragraph matching a reference text may be determined from at least one text paragraph, that is, when any reference text (medical vocabulary) is identified from a text paragraph, the text paragraph may be regarded as a text paragraph matching the reference text, or when the text similarity between a text paragraph and any reference text (medical sentence) is greater than a similarity threshold, the text paragraph may be regarded as a text paragraph matching the reference text, and there is no limitation on this.

[0173] For example, for text paragraph a, when it is determined that text paragraph a includes reference text (platelets), text paragraph a is taken as a text paragraph that matches the reference text, and there is no limitation to this.

[0174] S604: Delete other text paragraphs except the matched text paragraph from the reference data.

[0175] After identifying at least one text paragraph from the reference data and determining a text paragraph that matches the reference text from at least one text paragraph, the embodiment of the present disclosure can delete other text paragraphs except the matching text paragraph from the reference data. Thus, text paragraphs that are not related to the medical scenario can be deleted from the reference data, thereby effectively reducing the amount of data processing in the subsequent medical corpus generation method and saving time resources and computing resources.

[0176] It is understandable that, taking medical paper data as the reference data, the medical paper may contain text paragraphs that do not match the medical scenario, such as acknowledgment text paragraphs, literature citation paragraphs, etc. These text paragraphs that do not match the medical scenario can be considered as invalid data when used in the medical corpus generation method. Therefore, when the text paragraphs that do not match the medical scenario are used in the medical corpus generation method, the processing of invalid data may be introduced, thereby increasing the data processing amount of the subsequent medical corpus generation method.

[0177] Therefore, in the embodiment of the present disclosure, for example, other text paragraphs (acknowledgement text paragraphs, literature citation paragraphs) in the medical paper except for the matching text paragraphs can be deleted, thereby effectively reducing the data processing amount of the subsequent medical corpus generation method and saving time and computing resources.

[0178] S605: Determine target process description information corresponding to the reference data according to the reference data type and corpus requirement information.

[0179] S606: Determine the target process content for processing the reference data according to the reference data type and the target process description information.

[0180] The specific description of S605 - S606 can be found in the above embodiment, which will not be repeated here.

[0181] S607: When the reference data type is a medical data type, or when the corpus demand scenario is a medical scenario, obtain multiple diagnosis and treatment keywords corresponding to multiple diagnosis and treatment scenario types related to the medical scenario.

[0182] Among them, in the medical scenario, according to the different directions and needs of disease diagnosis and treatment, the medical scenario can be divided into multiple diagnosis and treatment scenario types, and the diagnosis and treatment scenario type can be, for example, thoracic surgery, otolaryngology, dermatology, etc., without any restriction.

[0183] Among them, multiple diagnosis and treatment keywords corresponding to the diagnosis and treatment scenario types refer to the names of drugs, disease names, diagnosis and treatment methods, etc. specifically used in each diagnosis and treatment department. Among them, the diagnosis and treatment keywords corresponding to thoracic surgery, such as chest computed tomography (CT), pleurisy, pneumothorax, etc., are not restricted.

[0184] S608: Match the target corpus with each diagnosis and treatment keyword to determine the matching degree between the target corpus and each diagnosis and treatment keyword.

[0185] In the disclosed embodiment, taking the reference data as a medical paper as an example, the medical paper may simultaneously involve descriptions in multiple diagnosis and treatment scenarios, so that when the reference data is processed according to the target process content, multiple target corpora for different diagnosis and treatment scenarios can be obtained.

[0186] The matching degree between the target corpus and each diagnosis and treatment keyword may be, for example, a similarity value between the target corpus and each diagnosis and treatment keyword, and there is no limitation to this.

[0187] In the embodiment of the present disclosure, when processing reference data according to the target process content, multiple target corpora for different diagnosis and treatment scenarios can be obtained, and then the target corpora can be similarly matched with each diagnosis and treatment keyword to determine the similarity value corresponding to the target corpus and each diagnosis and treatment keyword, and the similarity value can be used as the matching degree between the target corpus and each diagnosis and treatment keyword.

[0188] S609: Determine the diagnosis and treatment scenario type corresponding to the maximum matching degree among the multiple matching degrees, and establish a third association relationship between the corresponding diagnosis and treatment scenario type and the target corpus.

[0189] In the embodiment of the present disclosure, determining the diagnosis and treatment scenario type corresponding to the maximum match degree among multiple match degrees may be to determine the diagnosis and treatment scenario type corresponding to the maximum similarity value, and establish a third association relationship between the corresponding diagnosis and treatment scenario type and the target corpus. The third association relationship may be, for example, an association index relationship between the corresponding diagnosis and treatment scenario type and the target corpus.

[0190] S610: Save the third association relationship.

[0191] In the embodiment of the present disclosure, after establishing the third association relationship between the corresponding diagnosis and treatment scenario type and the target corpus, the third association relationship can be saved, and the corresponding diagnosis and treatment scenario type and the target corpus can be associated and stored based on the third association relationship. In this way, in subsequent corpus scenarios, it is possible to support the convenient call of the target corpus matching the target diagnosis and treatment scenario type according to the third association relationship, thereby effectively meeting the corpus requirements in different diagnosis and treatment scenarios.

[0192] In an embodiment of the present disclosure, by obtaining reference data and corpus demand information, wherein the reference data has a corresponding reference data type, and when the corpus demand scenario is a medical scenario and the reference data type is a text type, a reference text corresponding to the medical scenario is obtained, and then at least one text paragraph is identified from the reference data, and a text paragraph matching the reference text is determined from at least one text paragraph, and other text paragraphs except the matching text paragraphs are deleted from the reference data, thereby text paragraphs irrelevant to the medical scenario can be deleted from the reference data, thereby effectively reducing the data processing amount of the subsequent medical corpus generation method, saving time resources and computing resources, and then determining the target process description information corresponding to the reference data based on the reference data type and the corpus demand information, and determining the target process content for processing the reference data based on the reference data type and the target process description information, and then after establishing a third association relationship between the corresponding diagnosis and treatment scenario type and the target corpus, the corresponding diagnosis and treatment scenario type and the target corpus can be associated and stored, so that in subsequent corpus scenarios, it can support convenient calling of the target corpus matching the diagnosis and treatment scenario type based on the diagnosis and treatment scenario type, thereby effectively meeting the corpus requirements in different diagnosis and treatment scenarios.

[0193] Figure 7 is a structural schematic diagram of a corpus generation device provided according to an embodiment of the present disclosure;

[0194] like Figure 7 As shown, the corpus generating device 70 comprises:

[0195] An acquisition module 701 is used to acquire reference data and corpus demand information, wherein the reference data has a corresponding reference data type;

[0196] A first determination module 702 is used to determine target process description information corresponding to the reference data according to the reference data type and corpus requirement information;

[0197] A second determination module 703 is used to determine the target process content for processing the reference data according to the reference data type and the target process description information; and

[0198] The processing module 704 is used to process the reference data according to the target process content to obtain the target corpus.

[0199] In some embodiments of the present disclosure, the first determining module 702 is further configured to:

[0200] Determine at least one candidate process description information corresponding to the reference data type;

[0201] According to the corpus requirement information, target process description information is determined from at least one candidate process description information.

[0202] In some embodiments of the present disclosure, the first determining module 702 is further configured to:

[0203] Acquire a first relationship table, wherein the first relationship table includes: a plurality of candidate data types, and at least one candidate process description information corresponding to each candidate data type;

[0204] determining a candidate data type that matches the reference data type from among the plurality of candidate data types; and

[0205] At least one candidate process description information corresponding to the matched candidate data type is used as at least one candidate process description information corresponding to the reference data type.

[0206] In some embodiments of the present disclosure, each candidate process description information has a corresponding candidate processing type;

[0207] The first determining module 702 is further configured to:

[0208] Determine the data processing type based on the corpus demand information;

[0209] determining a candidate process type that matches the data process type from a plurality of candidate process types; and

[0210] The candidate flow description information corresponding to the matched candidate processing type is used as the target flow description information.

[0211] In some embodiments of the present disclosure, the first determining module 702 is further configured to:

[0212] The corpus requirement information is input into the processing type determination model to obtain the data processing type output by the processing type determination model, wherein the processing type determination model has learned the mapping relationship between the corpus requirement information and the data processing type.

[0213] In some embodiments of the present disclosure, the second determining module 703 is further configured to:

[0214] Determine at least one candidate process content according to the reference data type, wherein each candidate process content has corresponding candidate process description information;

[0215] Determining candidate process description information that matches the target process description information from at least one candidate process description information; and

[0216] The candidate process content corresponding to the matching candidate process description information is used as the target process content.

[0217] In some embodiments of the present disclosure, each candidate process content is in a first state, wherein the first state indicates that the corresponding candidate process content is not enabled;

[0218] The second determining module 703 is further used for:

[0219] configuring the candidate process content corresponding to the matching candidate process description information from a first state to a second state, wherein the second state indicates that the corresponding candidate process content has been enabled;

[0220] The candidate flow content configured in the second state is used as the target flow content.

[0221] In some embodiments of the present disclosure, the number of target process contents is multiple, and different target process contents have corresponding execution orders;

[0222] The processing module 704 is further used for:

[0223] The reference data is processed according to multiple target process contents and multiple execution orders to obtain a target corpus.

[0224] In some embodiments of the present disclosure, the processing module 704 is further configured to:

[0225] Processing reference data according to the target flow content of a first execution order to obtain a first data processing result, wherein the first execution order belongs to a plurality of execution orders;

[0226] Processing the first data processing result according to the target flow content of the second execution order to obtain a second data processing result, wherein the second execution order belongs to a plurality of execution orders, the second execution order is after the first execution order, and the second execution order is adjacent to the first execution order;

[0227] The first execution order is updated according to the second execution order until the reference data has been processed based on each target process content, and the data processing result obtained by processing the reference data based on the target process content of the last execution order among the multiple execution orders is used as the target corpus.

[0228] In some embodiments of the present disclosure, the corpus generating device 70 further includes:

[0229] An establishing module, used to establish a first association relationship between the target process description information corresponding to the first execution order and the first data processing result; and / or

[0230] Establishing a second association relationship between the target process description information corresponding to the second execution order and the second data processing result;

[0231] The saving module is used to save the first association relationship and / or the second association relationship.

[0232] In some embodiments of the present disclosure, the processing module 704 is further configured to:

[0233] Acquire content modification information corresponding to the target process content of the second execution order;

[0234] Modifying the information processing and the target flow content of the second execution order according to the content to obtain a new target flow content; and

[0235] The first data processing result is processed according to the new target flow content to obtain a second data processing result.

[0236] In some embodiments of the present disclosure, the processing module 704 is further configured to:

[0237] After obtaining content modification information corresponding to the target process content of the second execution order, configuring the target process content of the second execution order from being in the second state to being in the first state, wherein the first state indicates that the target process content is not enabled, and the second state indicates that the target process content is enabled;

[0238] After processing the target flow content of the second execution order according to the content modification information to obtain a new target flow content, the apparatus further comprises:

[0239] The new target process content is configured from being in the first state to being in the second state.

[0240] In some embodiments of the present disclosure, the corpus demand information includes: corpus demand scenario;

[0241] The acquisition module 701 is further used for:

[0242] After obtaining the reference data and corpus demand information, when the corpus demand scenario is a medical scenario and the reference data type is a text type, obtaining a reference text corresponding to the medical scenario;

[0243] identifying at least one text paragraph from the reference data, and determining, from the at least one text paragraph, a text paragraph that matches the reference text;

[0244] Delete all text paragraphs except the matching text paragraphs from the reference data.

[0245] In some embodiments of the present disclosure, the corpus demand information includes: corpus demand scenario;

[0246] The processing module 704 is further used for:

[0247] After processing the reference data according to the target process content to obtain the target corpus, when the reference data type is a medical data type, or when the corpus demand scenario is a medical scenario, obtaining multiple diagnosis and treatment keywords corresponding to multiple diagnosis and treatment scenario types related to the medical scenario;

[0248] Match the target corpus with each diagnosis and treatment keyword to determine the matching degree between the target corpus and each diagnosis and treatment keyword;

[0249] Determine the diagnosis and treatment scenario type corresponding to the maximum matching degree among the multiple matching degrees, and establish a third association relationship between the corresponding diagnosis and treatment scenario type and the target corpus;

[0250] Save the third relationship.

[0251] In some embodiments of the present disclosure, the corpus generating device 70 further includes:

[0252] The third determination module is used to determine the scene category information corresponding to the target corpus according to the corpus requirement information, wherein the scene category information is used to describe the category of the scene in which the target corpus is applied.

[0253] In some embodiments of the present disclosure, the target process content includes at least one of the following:

[0254] Rules for achieving the goal process;

[0255] Script file to implement the target process;

[0256] Services to achieve target processes;

[0257] Components that implement the target process.

[0258] It should be noted that the above explanation of the corpus generation method is also applicable to the corpus generation device of this embodiment and will not be repeated here.

[0259] In the disclosed embodiment, by obtaining reference data and corpus demand information, wherein the reference data has a corresponding reference data type, and according to the reference data type and the corpus demand information, determining the target process description information corresponding to the reference data, and then determining the target process content for processing the reference data according to the reference data type and the target process description information, and processing the reference data according to the target process content to obtain the target corpus, since the overall process of corpus generation is split into multiple target processes, and the target process content for implementing each target process is determined according to the reference data type and the corpus demand information, it is possible to effectively improve the corpus generation efficiency while meeting the corpus generation requirements, and since the overall process of corpus generation is split into multiple target processes, it is possible to support flexible adjustment of the corpus generation process.

[0260] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0261] Figure 8A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0262] like Figure 8 As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0263] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0264] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as a corpus generation method. For example, in some embodiments, the corpus generation method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the corpus generation method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the corpus generation method in any other appropriate manner (e.g., by means of firmware).

[0265] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0266] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0267] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0268] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0269] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0270] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0271] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0272] In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. In the description of the present disclosure, the words "if" and "if" used can be interpreted as "at the time of" or "when" or "in response to determination" or "under the circumstances of".

[0273] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A corpus generation method, comprising: Acquire reference data and corpus demand information, wherein the reference data has a corresponding reference data type; Determining target process description information corresponding to the reference data according to the reference data type and the corpus requirement information; Determining target process content for processing the reference data according to the reference data type and the target process description information; and Processing the reference data according to the target process content to obtain target corpus; The step of determining target process description information corresponding to the reference data according to the reference data type and the corpus requirement information includes: Determine at least one candidate process description information corresponding to the reference data type; Determining the target process description information from the at least one candidate process description information according to the corpus requirement information; The determining of at least one candidate process description information corresponding to the reference data type includes: Acquire a first relationship table, wherein the first relationship table includes: a plurality of candidate data types, and at least one candidate process description information corresponding to each of the candidate data types; determining a candidate data type that matches the reference data type from among the plurality of candidate data types; and using at least one candidate process description information corresponding to the matched candidate data type as at least one candidate process description information corresponding to the reference data type; The step of determining the target process content for processing the reference data according to the reference data type and the target process description information includes: Determine at least one candidate process content according to the reference data type, wherein each candidate process content has corresponding candidate process description information; Determining candidate process description information that matches the target process description information from at least one of the candidate process description information; and The candidate process content corresponding to the matching candidate process description information is used as the target process content.

2. According to the method of claim 1, each of the candidate process description information has a corresponding candidate processing type; in, The step of determining the target process description information from the at least one candidate process description information according to the corpus requirement information includes: Determine the data processing type according to the corpus demand information; Determining a candidate processing type that matches the data processing type from the plurality of candidate processing types; and The candidate process description information corresponding to the matched candidate processing type is used as the target process description information.

3. The method according to claim 2, wherein: Determining the data processing type according to the corpus demand information includes: The corpus requirement information is input into a processing type determination model to obtain a data processing type output by the processing type determination model, wherein the processing type determination model has learned a mapping relationship between the corpus requirement information and the data processing type.

4. The method according to claim 1, wherein each of the candidate process contents is in a first state, wherein: The first state indicates that the corresponding candidate process content is not enabled; The step of taking the candidate process content corresponding to the matching candidate process description information as the target process content includes: configuring the candidate process content corresponding to the matching candidate process description information from the first state to a second state, wherein the second state indicates that the corresponding candidate process content has been enabled; The candidate process content configured as the second state is used as the target process content.

5. The method according to claim 1, wherein the number of the target process contents is multiple, and different target process contents have corresponding execution orders; in, The processing of the reference data according to the target process content to obtain the target corpus includes: The reference data is processed according to the plurality of target process contents and the plurality of execution orders to obtain the target corpus.

6. The method according to claim 5, wherein: The processing of the reference data according to the plurality of target process contents and the plurality of execution orders to obtain the target corpus includes: Processing the reference data according to the target flow content of a first execution order to obtain a first data processing result, wherein the first execution order belongs to a plurality of the execution orders; Processing the first data processing result according to the target flow content of a second execution order to obtain a second data processing result, wherein the second execution order belongs to a plurality of the execution orders, the second execution order is after the first execution order, and the second execution order is adjacent to the first execution order; The first execution order is updated according to the second execution order until the reference data has been processed based on each of the target process contents, and a data processing result obtained by processing the reference data based on the target process content of the last execution order among the multiple execution orders is used as the target corpus.

7. The method according to claim 6, wherein: The method further comprises: Establishing a first association relationship between the target process description information corresponding to the first execution order and the first data processing result; and / or Establishing a second association relationship between the target process description information corresponding to the second execution order and the second data processing result; The first association relationship and / or the second association relationship is saved.

8. The method according to claim 6, wherein: The processing of the first data processing result according to the target flow content corresponding to the second execution order to obtain the second data processing result includes: Acquire content modification information corresponding to the target process content of the second execution order; Processing the target flow content in the second execution order according to the content modification information to obtain new target flow content; and The first data processing result is processed according to the new target flow content to obtain the second data processing result.

9. The method according to claim 8, wherein: After acquiring the content modification information corresponding to the target process content of the second execution order, the method further includes: configuring the target process content with the second execution order from being in the second state to being in the first state, wherein the first state indicates that the target process content is not enabled, and the second state indicates that the target process content is enabled; After processing the target flow content in the second execution order according to the content modification information to obtain new target flow content, the method further includes: The new target process content is configured from being in the first state to being in the second state.

10. The method according to any one of claims 1 to 9, wherein the corpus demand information comprises: Corpus demand scenarios; Wherein, after obtaining the reference data and corpus demand information, the method further includes: When the corpus demand scenario is a medical scenario and the reference data type is a text type, obtaining a reference text corresponding to the medical scenario; identifying at least one text paragraph from the reference data, and determining, from the at least one text paragraph, a text paragraph that matches the reference text; The other text paragraphs except the matched text paragraph are deleted from the reference data.

11. The method according to any one of claims 1 to 9, wherein the corpus demand information comprises: Corpus demand scenarios; Wherein, after processing the reference data according to the target process content to obtain the target corpus, the method further includes: When the reference data type is a medical data type, or when the corpus demand scenario is a medical scenario, obtaining a plurality of diagnosis and treatment keywords respectively corresponding to a plurality of diagnosis and treatment scenario types related to the medical scenario; Matching the target corpus with each of the diagnosis and treatment keywords to determine a matching degree between the target corpus and each of the diagnosis and treatment keywords; Determining a diagnosis and treatment scenario type corresponding to a maximum matching degree among the plurality of matching degrees, and establishing a third association relationship between the corresponding diagnosis and treatment scenario type and the target corpus; The third association relationship is saved.

12. The method according to any one of claims 1 to 9, wherein: The method further comprises: According to the corpus requirement information, scene category information corresponding to the target corpus is determined, wherein the scene category information is used to describe the category of the scene in which the target corpus is applied.

13. The method according to any one of claims 1 to 9, wherein: The target process content includes at least one of the following: Rules for achieving the goal process; Script file to implement the target process; Services to achieve target processes; Components that implement the target process.

14. A corpus generating device, comprising: An acquisition module, used to acquire reference data and corpus demand information, wherein the reference data has a corresponding reference data type; A first determination module, configured to determine target process description information corresponding to the reference data according to the reference data type and the corpus requirement information; A second determination module is used to determine the target process content for processing the reference data according to the reference data type and the target process description information; and A processing module, used for processing the reference data according to the target process content to obtain a target corpus; Wherein, the first determining module is further used for: Determine at least one candidate process description information corresponding to the reference data type; Determining the target process description information from the at least one candidate process description information according to the corpus requirement information; Wherein, the first determining module is further used for: Acquire a first relationship table, wherein the first relationship table includes: a plurality of candidate data types, and at least one candidate process description information corresponding to each of the candidate data types; determining a candidate data type that matches the reference data type from among the plurality of candidate data types; and using at least one candidate process description information corresponding to the matched candidate data type as at least one candidate process description information corresponding to the reference data type; Wherein, the second determining module is further used for: Determine at least one candidate process content according to the reference data type, wherein each candidate process content has corresponding candidate process description information; Determining candidate process description information that matches the target process description information from at least one of the candidate process description information; and The candidate process content corresponding to the matching candidate process description information is used as the target process content.

15. The apparatus according to claim 14, wherein each of the candidate process description information has a corresponding candidate processing type; in, The first determining module is further used for: Determine the data processing type according to the corpus demand information; Determining a candidate processing type that matches the data processing type from the plurality of candidate processing types; and The candidate process description information corresponding to the matched candidate processing type is used as the target process description information.

16. The device according to claim 15, wherein: The first determining module is further used for: The corpus requirement information is input into a processing type determination model to obtain a data processing type output by the processing type determination model, wherein the processing type determination model has learned a mapping relationship between the corpus requirement information and the data processing type.

17. The apparatus according to claim 14, wherein each of the candidate process contents is in a first state, wherein: The first state indicates that the corresponding candidate process content is not enabled; Wherein, the second determining module is further used for: configuring the candidate process content corresponding to the matching candidate process description information from the first state to a second state, wherein the second state indicates that the corresponding candidate process content has been enabled; The candidate process content configured as the second state is used as the target process content.

18. The device according to claim 14, wherein the number of the target process contents is multiple, and different target process contents have corresponding execution orders; in, The processing module is further used for: The reference data is processed according to the plurality of target process contents and the plurality of execution orders to obtain the target corpus.

19. The device according to claim 18, wherein: The processing module is further used for: Processing the reference data according to the target flow content of a first execution order to obtain a first data processing result, wherein the first execution order belongs to a plurality of the execution orders; Processing the first data processing result according to the target flow content of a second execution order to obtain a second data processing result, wherein the second execution order belongs to a plurality of the execution orders, the second execution order is after the first execution order, and the second execution order is adjacent to the first execution order; The first execution order is updated according to the second execution order until the reference data has been processed based on each of the target process contents, and a data processing result obtained by processing the reference data based on the target process content of the last execution order among the multiple execution orders is used as the target corpus.

20. The device according to claim 19, wherein The device also includes: an establishing module, used to establish a first association relationship between the target process description information corresponding to the first execution order and the first data processing result; and / or Establishing a second association relationship between the target process description information corresponding to the second execution order and the second data processing result; A saving module is used to save the first association relationship and / or the second association relationship.

21. The device according to claim 19, wherein The processing module is further used for: Acquire content modification information corresponding to the target process content of the second execution order; Processing the target process content in the second execution order according to the content modification information to obtain new target process content; as well as The first data processing result is processed according to the new target flow content to obtain the second data processing result.

22. The device according to claim 21, wherein The processing module is further used for: After acquiring the content modification information corresponding to the target process content of the second execution order, configuring the target process content of the second execution order from being in the second state to being in the first state, wherein the first state indicates that the target process content is not enabled, and the second state indicates that the target process content is enabled; After processing the target flow content with the second execution order according to the content modification information to obtain new target flow content, the device further includes: The new target process content is configured from being in the first state to being in the second state.

23. The device according to any one of claims 14 to 22, wherein the corpus demand information comprises: Corpus demand scenarios; Wherein, the acquisition module is also used for: After obtaining the reference data and the corpus demand information, if the corpus demand scenario is a medical scenario and the reference data type is a text type, obtaining a reference text corresponding to the medical scenario; identifying at least one text paragraph from the reference data, and determining, from the at least one text paragraph, a text paragraph that matches the reference text; The other text paragraphs except the matched text paragraph are deleted from the reference data.

24. The device according to any one of claims 14 to 22, wherein the corpus demand information comprises: Corpus demand scenarios; Wherein, the processing module is also used for: After processing the reference data according to the target process content to obtain the target corpus, when the reference data type is a medical data type, or when the corpus demand scenario is a medical scenario, obtaining a plurality of diagnosis and treatment keywords respectively corresponding to a plurality of diagnosis and treatment scenario types related to the medical scenario; Matching the target corpus with each of the diagnosis and treatment keywords to determine a matching degree between the target corpus and each of the diagnosis and treatment keywords; Determining a diagnosis and treatment scenario type corresponding to a maximum matching degree among the plurality of matching degrees, and establishing a third association relationship between the corresponding diagnosis and treatment scenario type and the target corpus; The third association relationship is saved.

25. The device according to any one of claims 14 to 22, wherein: The device also includes: The third determination module is used to determine the scene category information corresponding to the target corpus according to the corpus requirement information, wherein the scene category information is used to describe the category of the scene in which the target corpus is applied.

26. The device according to any one of claims 14 to 22, wherein: The target process content includes at least one of the following: Rules for achieving the goal process; Script file to implement the target process; Services to achieve the target process; Components that implement the target process.

27. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 13.

28. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 13.

29. A computer program product comprising computer instructions, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 13.

Images