Data processing method and device

Abstract

The invention discloses a data processing method and device, and relates to the technical field of computers. A specific embodiment of the method comprises the following steps: acquiring basic data; analyzing the basic data to obtain the data type of the basic data and the data content of the basic data; and according to the data type, the data content and the target processing type, generating a cue word, and inputting the cue word into a pre-trained large model to obtain a data processing result. According to the implementation mode, the basic data can be analyzed, the cue word is automatically generated based on the data type and the data content obtained through analysis and the target processing type pre-configured by the user to serve as input of the large model, and the automatic analysis process of the data is achieved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for data processing. Background Art

[0002] In the existing data processing process, when using a large model for content analysis, it is usually necessary to manually upload the file to be analyzed (such as PDF or pictures, etc.) and enter prompt words at the same time so that the large model can analyze the file to be analyzed based on the prompt words. The degree of automation is low and the automated analysis process of the data cannot be realized. Summary of the invention

[0003] In view of this, an embodiment of the present invention provides a method and device for data processing, which can parse basic data and automatically generate prompt words as input of a large model based on the data type, data content and target processing type pre-configured by the user, thereby realizing an automatic data analysis process.

[0004] To achieve the above objective, according to one aspect of an embodiment of the present invention, a method for data processing is provided.

[0005] A data processing method of an embodiment of the present invention includes: acquiring basic data; parsing the basic data to obtain the data type of the basic data and the data content of the basic data; generating prompt words according to the data type, the data content and the target processing type, and inputting the prompt words into a pre-trained large model to obtain data processing results.

[0006] Optionally, parsing the basic data to obtain the data type of the basic data and the data content of the basic data includes: parsing to obtain multiple fields from an Internet protocol corresponding to the basic data; when there is a target field matching a preset field in the field, determining the data type of the basic data according to the field value of the target field; wherein the data type includes a text type or a picture type; and determining the data content from the basic data using an identification strategy corresponding to the data type.

[0007] Optionally, the use of an identification strategy corresponding to the data type to determine data content from the basic data includes: when the data type is a text type, concatenating text content corresponding to the basic data with the same Internet Protocol address to obtain the data content.

[0008] Optionally, the use of an identification strategy corresponding to the data type to determine data content from the basic data includes: when the data type is a picture type, determining the picture type, purpose and picture content of each of the basic data; and generating the data content based on the picture type, purpose and picture content.

[0009] Optionally, generating the prompt word includes: determining a target prompt word template corresponding to the data type and the target processing type from a plurality of preset prompt word templates; and filling the data content into the target prompt word template to obtain the prompt word.

[0010] Optionally, the method further includes: when the basic data is derived from a security gateway, obtaining a preset execution strategy of the security gateway; analyzing whether the data processing result complies with the preset execution strategy, generating an execution result, and feeding back the execution result to the security gateway.

[0011] To achieve the above objective, according to another aspect of an embodiment of the present invention, a data processing device is provided.

[0012] A data processing device according to an embodiment of the present invention includes: an acquisition module for acquiring basic data; a parsing module for parsing the basic data to obtain a data type of the basic data and a data content of the basic data; a processing module for generating prompt words according to the data type, the data content and the target processing type, and inputting the prompt words into a preset model to obtain a data processing result; wherein the preset model is obtained by training a large model based on historical data.

[0013] To achieve the above objective, according to another aspect of an embodiment of the present invention, an electronic device for data processing is provided.

[0014] An electronic device for data processing according to an embodiment of the present invention comprises: one or more processors; a storage device for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement a data processing method according to an embodiment of the present invention.

[0015] To achieve the above objective, according to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided.

[0016] A computer-readable storage medium according to an embodiment of the present invention stores a computer program, and when the program is executed by a processor, a data processing method according to an embodiment of the present invention is implemented.

[0017] An embodiment of the above invention has the following advantages or beneficial effects: basic data can be parsed, and prompt words can be automatically generated as input of the large model based on the data type, data content and target processing type pre-configured by the user, thereby realizing the automatic analysis process of the data.

[0018] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0020] Figure 1 is a schematic diagram of the main flow of a method for data processing according to an embodiment of the present invention;

[0021] Figure 2 is a schematic diagram of the main process of parsing data types and data contents according to an embodiment of the present invention;

[0022] Figure 3 is a schematic diagram of a main process of determining data content from basic data according to an embodiment of the present invention;

[0023] Figure 4 is a schematic diagram of a main process of determining a prompt word according to an embodiment of the present invention;

[0024] Figure 5 It is a schematic diagram of the main process of generating execution results based on a security gateway according to an embodiment of the present invention;

[0025] Figure 6 is a schematic diagram of main modules of a data processing device according to an embodiment of the present invention;

[0026] Figure 7 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0027] Figure 8 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0029] It should be pointed out that the embodiments of the present invention and the technical features therein may be combined with each other without conflict.

[0030] It should be noted that the collection, collection, updating, analysis, processing, use, transmission, storage and other aspects of user personal information involved in the technical solution of this disclosure are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information, network security and national security.

[0031] Figure 1 It is a schematic diagram of main steps of a data processing method according to an embodiment of the present invention.

[0032] like Figure 1 As shown, the data processing method of the embodiment of the present invention mainly includes the following steps:

[0033] Step S101: Obtain basic data;

[0034] Step S102: parsing the basic data to obtain the data type and data content of the basic data;

[0035] Step S103: Generate prompt words according to the data type, data content and target processing type, and input the prompt words into the pre-trained large model to obtain data processing results.

[0036] Among them, the basic data obtained by the embodiment of the present invention can be internal enterprise transmission data obtained from the security gateway of the enterprise or organization. For enterprises, in order to protect the internal enterprise network from external threats, each enterprise usually uses a security gateway to monitor the traffic. Therefore, the basic data corresponding to the traffic can be directly obtained from the security gateway.

[0037] In an optional embodiment, the process of parsing the data type and data content in step 102 can be as follows: Figure 2 As shown, specifically including:

[0038] Step S201: parsing the Internet protocol corresponding to the basic data to obtain multiple fields;

[0039] Step S202: when there is a target field matching the preset field in the field, determine the data type of the basic data according to the field value of the target field; wherein the data type includes a text type or a picture type;

[0040] Step S203: using the identification strategy corresponding to the data type, determine the data content from the basic data.

[0041] Among them, the Internet protocol contains multiple fields, which are used to represent multiple transmission parameters in the transmission process. Usually, there are differences in the fields of different Internet protocols, but there are preset fields for representing data types. For example, taking the http protocol as an example, the Content-Type field in the protocol is the field representing the data type. Therefore, in the data parsing process, the Internet protocol can be parsed and then matched with the preset fields representing the data type corresponding to each Internet protocol. When the target field is matched, the data type of the basic data can be determined according to the field value of the target field.

[0042] It is understandable that for different data types, the identification strategy and the process of determining the data content are also different. The following is a specific analysis of how to determine the data content of text type and image type:

[0043] (I) In an optional embodiment, step S203 includes: when the data type is a text type, concatenating the text content corresponding to the basic data with the same Internet Protocol address to obtain the data content.

[0044] In the prior art, when data analysis of text content is required, each data is usually used as an analysis object. For example, user A and user B use the internal communication application of the enterprise to send a message. A sends "How is the weather today?" and B replies "The weather is very good". Then two pieces of data will be generated accordingly, one for "How is the weather today?" and the other for "The weather is very good". Then data analysis is performed on the two pieces of data respectively, for example, to determine whether sensitive words are contained. However, in the process of using communication applications, data is usually coherent. It is difficult to ensure the accuracy of data analysis by only analyzing a single piece of data. Similarly, taking user A sending a message as an example, when the user sends "TMD", the system usually detects that it contains sensitive words. When user A sends three messages in succession, namely "T", "M", and "D", each piece of data is detected separately, and no sensitive words are detected, but the meaning of the continuous data still belongs to sensitive words. Therefore, in the embodiment of the present invention, the basic data with the same Internet Protocol address is spliced, so that the content sent by each user can be connected, achieving the purpose of connecting the context, and then the spliced ​​content is analyzed as the data content, which can obtain a more accurate analysis effect. Each application device has a unique Internet Protocol address, that is, when user A uses terminal device 1 to communicate with user B using terminal device 2, each piece of data of user A will carry the Internet address of terminal device 1, and each piece of data of user B will carry the Internet address of terminal device 2. Therefore, the embodiment of the present invention realizes splicing of only the content sent by the same user by splicing the basic data with the same Internet Protocol address, and the data sent by user A and user B will not be spliced ​​together.

[0045] (ii) In another optional embodiment, step S203 is as follows Figure 3 As shown, including:

[0046] Step S301: when the data type is a picture type, determining the picture type, purpose and picture content of each basic data;

[0047] Step S302: Generate data content according to the type, purpose and content of the picture.

[0048] Different from the prior art, when the prior art analyzes the picture, it only considers the picture content identified in the picture, and does not consider other parameters of the picture. In this application, the types of pictures and the uses of pictures are added. Specifically, the types of pictures may include: photo pictures, design drawings, hand-drawn pictures, instrument detection pictures, etc. Taking sensitive word detection as an example, in the hand-drawn picture, although there is no text content, the hand-drawn content is visually presented in the style of "TMD" through different colors. Only by identifying the text in the picture with the prior art, sensitive words cannot be identified. By inputting the type of picture (hand-drawn picture), the purpose of the picture, and the content of the picture as data content into the large model, more accurate recognition can be achieved.

[0049] Through the above process, how to determine the data content of the basic data of text type and picture type is explained, and the corresponding data content of different data types can be obtained. However, as the input of the large model, other factors such as the format need to be unified so that the model can successfully recognize the input data. Therefore, in an optional embodiment, step S103 can be as follows Figure 4 As shown, including:

[0050] Step S401: determining a target prompt word template corresponding to a data type and a target processing type from a plurality of preset prompt word templates;

[0051] Step S402: Fill the data content into the target prompt word template to obtain the prompt word.

[0052] Among them, the pre-set multiple prompt word templates correspond to the data type and the processing type, for example, prompt word template 1 corresponds to sensitive word detection for image type data, prompt word template 2 corresponds to image expansion for image type data, prompt word template 3 corresponds to image drawing for text type data, etc. It can be understood that the prompt word template may include the processed data type, the processed data content, the analysis rules, the purpose, and the returned content format, etc. By filling the data content obtained from the basic data parsing into the target prompt word template through step S402, the prompt word that is finally used as the input of the large model can be obtained, and then the data processing result fed back by the large model can be obtained.

[0053] In an optional embodiment, when the basic data comes from a security gateway, it can also be Figure 5 As shown, including:

[0054] Step S501: Obtaining a preset execution policy of a security gateway;

[0055] Step S502: Analyze whether the data processing result complies with the preset execution strategy, generate an execution result, and feed back the execution result to the security gateway.

[0056] It is understandable that the security gateway usually only has the ability to execute, but not the ability to perform logical processing. When the large model generates the data processing results, it is also necessary to determine the execution results according to the preset execution policy of the security gateway. For example, the preset execution policies corresponding to the security gateways set by different enterprises are different. For example, when the preset execution policy of an enterprise is not allowed to display sensitive words, then when the data processing results contain sensitive words, it can be generated: an execution result that displays text with "*" for the security gateway to perform related operations. When the preset execution policy of the enterprise is not to restrict sensitive words, even if the data processing results contain sensitive words, the generated execution results are displayed normally. For the preset execution policy, it can be specifically set according to the actual needs and usage of the enterprise, and the present invention does not make specific limitations on this.

[0057] According to the data processing method of the embodiment of the present invention, basic data can be parsed, and prompt words can be automatically generated as input of the large model based on the data type, data content and target processing type pre-configured by the user, thereby realizing the automatic analysis process of the data.

[0058] Figure 6 4 is a schematic diagram of main modules of a data processing device according to an embodiment of the present invention.

[0059] like Figure 6 As shown, the data processing device 600 of the embodiment of the present invention includes:

[0060] Acquisition module 601, used to acquire basic data;

[0061] A parsing module 602 is used to parse the basic data to obtain the data type of the basic data and the data content of the basic data;

[0062] The processing module 603 is used to generate prompt words according to the data type, the data content and the target processing type, and input the prompt words into a preset model to obtain a data processing result; wherein the preset model is obtained by training a large model based on historical data.

[0063] In an optional embodiment of the present invention, the parsing module 602 is also used to parse and obtain multiple fields from the Internet protocol corresponding to the basic data; when there is a target field matching a preset field in the field, determine the data type of the basic data according to the field value of the target field; wherein the data type includes a text type or a picture type; and determine the data content from the basic data using an identification strategy corresponding to the data type.

[0064] In an optional embodiment of the present invention, the parsing module 602 is further used to, when the data type is a text type, concatenate the text content corresponding to the basic data with the same Internet Protocol address to obtain the data content.

[0065] In an optional embodiment of the present invention, the parsing module 602 is also used to, when the data type is a picture type, determine the picture type, purpose and picture content of each of the basic data; and generate the data content according to the picture type, purpose and picture content.

[0066] In an optional embodiment of the present invention, the processing module 603 is further used to determine a target prompt word template corresponding to the data type and the target processing type from a plurality of pre-set prompt word templates; and fill the data content into the target prompt word template to obtain the prompt word.

[0067] In an optional embodiment of the present invention, the device also includes: a generation module, which is used to obtain the preset execution policy of the security gateway when the basic data comes from the security gateway; analyze whether the data processing result complies with the preset execution policy, generate an execution result, and feed back the execution result to the security gateway.

[0068] The data processing device according to the embodiment of the present invention can parse basic data and automatically generate prompt words as input of the large model based on the data type, data content and target processing type pre-configured by the user, thereby realizing the automatic analysis process of the data.

[0069] Figure 7 An exemplary system architecture 700 is shown to which the data processing method or data processing apparatus according to the embodiment of the present invention can be applied.

[0070] like Figure 7 As shown, system architecture 700 may include terminal devices 701, 702, 703, network 704 and server 705. Network 704 is used to provide a medium for communication links between terminal devices 701, 702, 703 and server 705. Network 704 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0071] The user can use the terminal devices 701, 702, 703 to interact with the server 705 through the network 704 to receive or send data, etc. Various communication client applications can be installed on the terminal devices 701, 702, 703, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0072] The terminal devices 701 , 702 , and 703 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0073] The server 705 may be a server that provides various services, such as a background management server that provides support for basic data sent by users using the terminal devices 701, 702, and 703. The background management server may analyze the received basic data and feed back the processing results (such as data processing results) to the terminal device.

[0074] It should be noted that the data processing method provided in the embodiment of the present invention is generally executed by the server 705 , and accordingly, the data processing device is generally set in the server 705 .

[0075] It should be understood that Figure 7 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0076] Referring to FIG. YY, it shows a schematic diagram of the structure of a computer system YY00 suitable for implementing a terminal device of an embodiment of the present invention. The terminal device shown in FIG. YY is only an example and should not bring any limitation to the functions and scope of use of the embodiment of the present invention.

[0077] like Figure 8 As shown, the computer system 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage part 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the system 800 are also stored. The CPU 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) first interface 805 is also connected to the bus 804.

[0078] The following components are connected to the I / O first interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network first interface card such as a LAN card, a modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O first interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed, so that a computer program read therefrom is installed into the storage section 808 as needed.

[0079] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 809, and / or installed from the removable medium 811. When the computer program is executed by the central processing unit (CPU) 801, the above-mentioned functions defined in the system of the present invention are executed.

[0080] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0081] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0082] The modules involved in the embodiments of the present invention may be implemented by software or hardware. The modules described may also be arranged in a processor, for example, it may be described as follows: a processor includes an acquisition module, a parsing module, and a processing module. The names of these modules do not constitute a limitation on the modules themselves in some cases, for example, the acquisition module may also be described as a "module for acquiring basic data".

[0083] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: obtaining basic data; parsing the basic data to obtain the data type of the basic data and the data content of the basic data; generating prompt words according to the data type, the data content and the target processing type, and inputting the prompt words into the pre-trained large model to obtain data processing results.

[0084] According to the technical solution of the embodiment of the present invention, the basic data can be parsed, and prompt words can be automatically generated as input of the large model based on the data type, data content and target processing type pre-configured by the user, thereby realizing the automatic analysis process of the data.

[0085] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for data processing, characterized in that: include: Get basic data; Parsing the basic data to obtain the data type of the basic data and the data content of the basic data; According to the data type, the data content and the target processing type, a prompt word is generated, and the prompt word is input into the pre-trained large model to obtain a data processing result.

2. The method according to claim 1, characterized in that The step of parsing the basic data to obtain the data type of the basic data and the data content of the basic data includes: Parsing the Internet protocol corresponding to the basic data to obtain multiple fields; In the case that there is a target field matching the preset field in the field, determining the data type of the basic data according to the field value of the target field; wherein the data type includes a text type or a picture type; The data content is determined from the basic data using an identification strategy corresponding to the data type.

3. The method according to claim 2, characterized in that The determining the data content from the basic data by using the identification strategy corresponding to the data type includes: When the data type is a text type, the text contents corresponding to the basic data with the same Internet Protocol address are concatenated to obtain the data content.

4. The method according to claim 2, characterized in that: The determining the data content from the basic data by using the identification strategy corresponding to the data type includes: In the case where the data type is a picture type, determining the picture type, purpose and picture content of each of the basic data; The data content is generated according to the type, purpose and content of the picture.

5. The method according to claim 1, characterized in that The generating prompt words comprises: Determine a target prompt word template corresponding to the data type and the target processing type from a plurality of preset prompt word templates; The data content is filled into the target prompt word template to obtain the prompt word.

6. The method according to claim 1, characterized in that Also includes: In the case where the basic data originates from a secure gateway, Obtaining a preset execution policy of the security gateway; Analyze whether the data processing result complies with the preset execution strategy, generate an execution result, and feed back the execution result to the security gateway.

7. A data processing device, characterized in that: include: Acquisition module, used to obtain basic data; A parsing module, used to parse the basic data to obtain the data type of the basic data and the data content of the basic data; A processing module is used to generate prompt words according to the data type, the data content and the target processing type, and input the prompt words into a preset model to obtain a data processing result; wherein the preset model is obtained by training a large model based on historical data.

8. An electronic device for data processing, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

9. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method according to any one of claims 1 to 7.

Images