Data processing method and device, server and medium
By parsing files that users cannot parse on the server side and sending parsed files, the problem of inefficient users to parse files in specific formats on local devices is solved, and a more efficient data analysis process is achieved.
Patent Information
- Application Number
- CN202411927588.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-25
- Publication Date
- 2025-05-30
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
In the prior art, when users perform data analysis, they need to download and parse files of a specific format on the local device, but the configuration environment of the local device cannot read these files, resulting in inefficient data analysis.
The server-side parsing files that users cannot parse, generate the parsed files and send them to the user's target device, and the user directly obtains the parsed files to improve data analysis efficiency.
It improves the efficiency of data analysis, reduces the process of users converting files between different configuration environments, and reduces the time of network transmission and potential link congestion risks.
Smart Images
Figure CN120067419A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing, and in particular, to a data processing method, apparatus, server, and medium. Background Art
[0002] There is an increasing amount of data on the Internet, and the emerging data crawling technology can obtain data from the Internet based on specified rules for data analysis. Currently, with the improvement of the standardization and privacy of data management, most existing data crawls are based on the public data provided by application programming interfaces (APIs).
[0003] In related technologies, when a user uses web crawling technology for data analysis, the data interaction process is as follows: The user first configures the crawling rules and keywords on a local device, and then the local device sends the crawling rules and keywords to a server for the server to perform data crawling; after the server finishes crawling, it obtains the source link of the target data and sends it to the user's local device, and the local device then downloads the data based on the source link.
[0004] After the user downloads the crawled target data to the local device, a series of data analysis tasks are often required, such as data screening and filtering. However, there may be some files in a specific format in the target data, and these specific format files often cannot be read in the configuration environment of the normal local device; if the user transfers to another configuration environment to read these specific format files and then transfers them back to the local device, the data analysis efficiency will be slow. Therefore, how to improve the efficiency of data analysis is an urgent problem to be solved. Summary of the Invention
[0005] To improve the efficiency of data analysis, this application provides a data processing method, apparatus, server, and medium.
[0006] In a first aspect, this application provides a data processing method, adopting the following technical solution:
[0007] A data processing method includes:
[0008] Receiving a link set of target data sent by a user's target device, where there is at least one source link in the link set, and each source link points to at least one source file;
[0009] Performing a screening condition obtaining step to obtain a screening condition;
[0010] Determining whether there is a file to be parsed that meets the screening condition in the file set pointed to by the link set, where the file set includes the source files pointed to by each source link;
[0011] If it exists, obtain each file to be parsed, and parse each file to be parsed to obtain each parsed file;
[0012] Send each of the parsed files to the target device of the user;
[0013] Among them, the screening condition obtaining step includes:
[0014] Send a test script to the target device to obtain the target type, where the target type is the file type that the target device cannot parse, and record the sending time when the test script is sent to the target device and the end time when the sending is completed;
[0015] Based on the sending time, the end time, and the volume of the test script, determine the network transmission rate with the target device, and obtain a volume threshold based on the network transmission rate;
[0016] Determine that the screening condition is: files with a file volume exceeding the volume threshold and not meeting the target type are the first type of files, files meeting the target type are the second type of files, and the files to be parsed include the first type of files and the second type of files.
[0017] In a possible implementation manner, determining whether there are files to be parsed that meet the screening conditions in the file set pointed to by the link set includes:
[0018] Obtain the attribute information of each source file in the file set, where the attribute information includes the file type and the file volume;
[0019] Based on the attribute information of each source file, determine whether there are files to be parsed in the file set pointed to by the link set.
[0020] In a possible implementation manner, parsing each file to be parsed to obtain each parsed file includes:
[0021] Allocate parsing space for each of the files to be parsed based on the file type, and the parsing space is configured as a container space;
[0022] Parse the corresponding file to be parsed in the parsing space to obtain each parsed file.
[0023] In a possible implementation manner, parsing the corresponding file to be parsed in the parsing space to obtain each parsed file includes:
[0024] Perform compression processing on the first type of files in the parsing space, and determine the compressed file as the parsed file corresponding to the first type of files; and / or
[0025] In the parsing space, standardize and convert the second type of file based on the file type to obtain a parsed file of the second type of file, where the standardization conversion includes at least one of screenshot, screen recording, and text translation.
[0026] In a possible implementation manner, the standardization conversion of the second type of file includes:
[0027] If the standardization conversion includes screenshot, capture N basic images, and determine the image with the highest clarity from the N basic images as the parsed file, where the parameters of any two basic images are different, and the parameters include screenshot position, screenshot perspective, and scaling ratio;
[0028] If the standardization conversion includes screen recording, the screen recording duration is determined based on the file size of the second type of file, and screen recording is performed based on the screen recording duration to obtain a parsed file;
[0029] If the standardization conversion includes text translation, translate text files in at least two languages as the parsed file.
[0030] In a possible implementation manner, sending the respective parsed files to the user's target device includes:
[0031] Compress the respective parsed files and send them to the user's target device; or
[0032] Generate download links corresponding to the respective parsed files;
[0033] Send the download links corresponding to the respective parsed files to the user's target device.
[0034] In a second aspect, the present application provides a data processing device, adopting the following technical solution:
[0035] A data processing device includes:
[0036] A receiving module, configured to receive a link set of target data sent by the user's target device, where there is at least one source link in the link set, and each source link points to at least one source file;
[0037] A filtering condition obtaining module, configured to execute a filtering condition obtaining step to obtain a filtering condition;
[0038] A file to be parsed determining module, configured to determine whether there is a file to be parsed that meets the filtering condition in the file set pointed to by the link set, where the file set includes the source files pointed to by each source link;
[0039] An analysis module, configured to obtain each file to be parsed and analyze each file to be parsed to obtain each parsed file;
[0040] A sending module, configured to send the respective parsed files to a target device of a user;
[0041] Among them, the filtering condition obtaining module is specifically configured to:
[0042] Send a test script to the target device to obtain a target type, where the target type is a file type that the target device cannot parse, and record the sending time when the test script is sent to the target device and the end time when the sending is completed;
[0043] Determine a network transmission rate with the target device based on the sending time, the end time, and the volume of the test script, and obtain a volume threshold based on the network transmission rate;
[0044] Determine the filtering condition as: files with a file volume exceeding the volume threshold and not conforming to the target type are the first type of files, files conforming to the target type are the second type of files, and the files to be parsed include the first type of files and the second type of files.
[0045] In a possible implementation manner, when the file to be parsed determining module determines whether there are files to be parsed that meet the filtering conditions in the file set pointed to by the link set, it is specifically configured to:
[0046] Obtain the attribute information of each source file in the file set, where the attribute information includes the file type and the file volume;
[0047] Determine whether there are files to be parsed that meet the filtering conditions in the file set pointed to by the link set based on the attribute information of each source file.
[0048] In a third aspect, the present application provides a server, adopting the following technical solution:
[0049] A server, the server includes:
[0050] At least one processor;
[0051] A memory;
[0052] At least one application program, where the at least one application program is stored in the memory and is configured to be executed by at least one processor, and the at least one application program is configured to: execute the data processing method described in the first aspect.
[0053] In a fourth aspect, the present application provides a computer-readable storage medium, adopting the following technical solution:
[0054] A computer-readable storage medium, including: a computer program stored with the ability to be loaded and executed by a processor to execute the data processing method described in the first aspect.
[0055] In summary, the present application includes at least one of the following beneficial technical effects:
[0056] 1. The server can identify the file to be parsed based on the link set and the preset rules. For files that cannot be parsed by the user's target device, that is, the second type of files, the server can obtain the second type of files for parsing and then send them to the user's target device. Then the files directly obtained by the user are the parsed files after parsing the second type of files. When the server obtains the second type of files, since the file transfer efficiency between servers is relatively fast, and it is even possible that the server executing this method is the same as the server storing the second type of files. In this way, the server can directly retrieve the second type of files stored in the server for parsing. Therefore, the efficiency of the server in obtaining the second type of files is relatively fast.
[0057] 2. For some files with large volumes, although the user's target device can parse them, due to the limited network transmission rate between the user's target device and the server in general, when the target device downloads these large-volume files, it usually takes more time; moreover, when transmitting large-volume files, the probability of communication link congestion and interruption is also higher. In order to improve the efficiency of the user in obtaining large-volume files, therefore, files with a volume exceeding the volume threshold are used as the first type of files to be parsed in the files to be parsed, that is, the first type of files are compressed to obtain the compressed files and sent to the user's target device, so that the user directly downloads the compressed files in the server executing this method, so as to improve the efficiency of the user's target device in obtaining large-volume files. BRIEF DESCRIPTION OF THE DRAWINGS
[0058] Figure 1 is a schematic diagram of the information interaction relationship between a local device and a crawling server in the related art;
[0059] Figure 2 is a schematic flowchart of the data processing method in an embodiment of the present application;
[0060] Figure 3 is a schematic diagram of the information interaction relationship between a target device and a server in an embodiment of the present application;
[0061] Figure 4 is a schematic diagram of the interaction between a target device and a server in the step of obtaining screening conditions in an embodiment of the present application;
[0062] Figure 5 is a schematic diagram of the structure of the data processing device in an embodiment of the present application;
[0063] Figure 6 is a schematic diagram of the structure of the server in an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS
[0064] The following sets are attached Figure 1 - Attachment Figure 6 This application will be further described in detail below.
[0065] After reading this specification, those skilled in the art may make modifications to this embodiment without creative contributions as needed, but as long as they are within the scope of the claims of this application, they are protected by the patent law.
[0066] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the accompanying drawings in the embodiments of this application will be assembled below, and the technical solutions in the embodiments of this application will be described clearly and completely. Obviously, the described embodiments are some, but not all, of the embodiments of this application. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of this application.
[0067] In addition, the term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this article, unless otherwise specified, generally represents an "or" relationship between the associated objects before and after.
[0068] Using data crawling technology, data can be obtained from the Internet based on specified rules. At present, with the improvement of the standardization and privacy of data management, most existing data crawls are based on the public data provided by application programming interfaces (APIs).
[0069] As Figure 1 shown, in the related art, when a user uses crawling technology for data analysis, the data interaction process is as follows: The user first configures crawling rules and keywords on a local device, and then the local device sends the crawling rules and keywords to a server to enable the server to perform data crawling; after the server crawls the data, it obtains the source link of the data and sends it to the user's local device, and the local device then downloads the data from the server based on the source link.
[0070] After downloading the crawled target data to the user's local device, a series of data analysis tasks often need to be performed manually, such as data screening and filtering. However, there may be some files in specific formats in the target data, and these specific format files often cannot be read in the configuration environment of a normal local device; if the user transfers to another configuration environment to read these specific format files and then transfers them back to the local device, the data analysis efficiency will be slow. Therefore, how to improve the efficiency of data analysis is an urgent problem to be solved.
[0071] An embodiment of the present application provides a data processing method, which is executed by a server. The server can be a single server or a server cluster composed of multiple slave servers, and the embodiment of the present application does not limit this. Refer to Figure 2 , the data processing method includes steps S10 - S50, where:
[0072] Step S10: Receive a set of links of target data sent by a user's target device. There is at least one source link in the set of links, and each source link points to at least one source file.
[0073] For the embodiment of the present application, if the server used by the user to crawl data is not the same as the server executing this method, the set of links can be sent by the user through the target device or other electronic devices and received by the server. If the crawling server used by the user to crawl data is the same as the server executing this method, the set of links can be obtained locally from the server, that is, directly crawl based on the crawling rules and keywords published by the user to obtain the set of links. Among them, there is one or more source links in the set of links. The source link can be a URL, or a port link of an API, and of course it can also be other links that can point to the file location. The embodiment of the present application does not limit this.
[0074] Step S20: Execute a screening condition acquisition step to obtain screening conditions.
[0075] For the embodiment of the present application, the screening conditions are: files with a file volume exceeding the volume threshold and not meeting the target type are the first type of files, and files meeting the target type are the second type of files. The files to be parsed include the first type of files and the second type of files, where the target type is a file type that the user's target device cannot parse, and the volume threshold is determined based on the network transmission rate with the target device.
[0076] Step S30: Determine whether there are files to be parsed that meet the screening conditions in the file set pointed to by the set of links. The file set includes the source files pointed to by each source link;
[0077] Step S40: If there are files to be parsed in the file set, obtain each file to be parsed and parse each file to be parsed to obtain each parsed file.
[0078] For the embodiment of the present application, the file corresponding to each source link in the set of links is a source file, and the source files corresponding to each source link in the set of links are the file set corresponding to the set of links.
[0079] After determining that there are files to be parsed, obtain each file to be parsed, that is, download the corresponding source file based on the source link of the file to be parsed. Parse each file to be parsed, which may include reading the file to be parsed using a reading software corresponding to the file type of the file to be parsed. Of course, the parsing may also include decompressing the file to be parsed, and even the parsing may include converting the format of the file to be parsed, etc.
[0080] Step S50: Send each parsed file to the target device of the user.
[0081] For the embodiments of the present application, it may be to uniformly package and then compress each parsed file to generate an overall compressed package, and then send the compressed package to the target device of the user. It may also be to generate download links corresponding to each parsed file, and then send the download links corresponding to each parsed file to the target device of the user. The specific sending method and format are not limited in the embodiments of the present application.
[0082] As Figure 3 shown in the schematic diagram of the information interaction relationship between the target device of the user and the server, the server can identify the file to be parsed based on the link set and the preset rules. For files that the target device of the user cannot parse, that is, the second type of files, the server can obtain the second type of files, parse them, and send them to the target device of the user. Then the files directly obtained by the user are the parsed files after parsing the second type of files. When the server obtains the second type of files, since the file transfer efficiency between servers is relatively fast, and even the server executing this method may be the same as the server storing the second type of files, the server can directly retrieve the second type of files stored in the server for parsing. Therefore, the efficiency of the server in obtaining the second type of files is relatively fast.
[0083] For some files with large volumes, although the target device of the user can parse them, due to the limited network transmission rate between the target device of the user and the server in general, when the target device downloads these large-volume files, it often takes more time; and when transmitting large-volume files, the probability of communication link congestion and interruption is also higher. To improve the efficiency of the user in obtaining files with larger volumes, therefore, files with a volume exceeding the volume threshold are used as the first type of files among the files to be parsed for parsing, that is, the first type of files are compressed to obtain the compressed files and sent to the target device of the user, so that the user directly downloads the compressed files in the server executing this method to improve the efficiency of the target device of the user in obtaining files with larger volumes.
[0084] Further, referring to Figure 4, in step S20, the screening condition obtaining step may specifically include steps S21 - S23, where:
[0085] Step S21: Send a test script to the target device to obtain the target type, where the target type is a file type that the target device cannot parse, and record the sending time when the test script is sent to the target device and the end time when the sending is completed.
[0086] Configuring the environment information means that the server sends a test script to the target device to actively obtain the configuration environment information feedback by the target device. The test script is used to obtain the file types that the target device can parse and generate the configuration environment information. Among them, the test script can be a registry retrieval program. After the target device receives the test script, the test script scans the registry program of the target device, obtains the installed applications on the target device, and then determines the file types that the target device can parse based on the installed applications on the target device, and then generates the configuration environment information of the target device and feedbacks it to the server.
[0087] Furthermore, multiple files in different formats can also be embedded in the test script. The test script is used to randomly select a non-repeated file as the test file each time, and then call the corresponding recommended program or default program of the target device to open it, and supervise the occupancy status of the test file in the task process; if the occupancy status is occupied, it indicates that the target device can read and parse the test file, and if the occupancy status is unoccupied, it indicates that the target device cannot read and parse the test file. Then, after the test script extracts all the test files, it determines the configuration environment information of the target device and feedbacks it to the server.
[0088] In a possible implementation, set corresponding tags or identifiers in the test script, or the server sends the test script to the target device through a protocol agreed with the target device or a predetermined communication link, so that when the target device receives the test script, it can automatically run the test script.
[0089] Step S22: Determine the network transmission rate with the target device based on the sending time, end time, and the volume of the test script, and obtain the volume threshold based on the network transmission rate.
[0090] Specifically, pre-set the corresponding relationship between different network transmission rate intervals and volume thresholds. After determining the network transmission rate between the user's target device and the server, determine the interval where the network transmission rate is located, and then determine the corresponding volume threshold.
[0091] Determining the volume threshold through the network transmission rate is convenient for determining which files belong to large-volume files based on the actual situation of the user's device, and thus is convenient for more accurately screening out the first type of files in the files to be parsed.
[0092] Step S23: Determine the screening conditions as follows: Files with a file size exceeding the size threshold and not meeting the target type are classified as the first type of files, and files meeting the target type are classified as the second type of files. The files to be parsed include the first type of files and the second type of files.
[0093] In the embodiments of the present application, the effects of the test script are reflected in two aspects. Among them: The first aspect is that by sending the test script to the target device of the user, the screening conditions can be obtained, so as to enable different target devices to determine the files to be parsed based on different screening rules to meet the actual needs of different target devices. The second aspect is that based on sending the test script to the target device, the network transmission rate between the server and the target device can also be tested, and then the size threshold can be determined based on the network transmission rate.
[0094] By determining the screening conditions, the files to be parsed existing in the file set can be further determined. Therefore, further, in step S30, it is determined whether there are files to be parsed that meet the screening conditions in the file set pointed to by the link set, which may specifically include step S31 (not shown in the figure) and step S32 (not shown in the figure), where:
[0095] Step S31: Obtain the attribute information of each source file in the file set. The attribute information includes the file type and the file size;
[0096] Step S32: Determine whether there are files to be parsed in the file set pointed to by the link set based on the attribute information of each source file.
[0097] Specifically, the attribute information of the file is the fixed attribute of the file itself. For each source link, the source link is parsed to obtain the file attributes of the corresponding source file. For example, if the source link is a URL, the corresponding file extension in the URL link can be read, and the file type of the source file can be determined based on the file extension. The server obtains the attribute information of the file by reading and parsing the metadata of the source file.
[0098] Further, in step S40, each file to be parsed is parsed to obtain each parsed file, which may specifically include: allocating parsing space for each file to be parsed based on the file type. The parsing space is configured as a container space, and the corresponding file to be parsed is parsed in the parsing space to obtain each parsed file.
[0099] Specifically, the parsing space can be an independent isolated container configured in the server, that is, a container space is configured separately for each file to be parsed, and the reading software corresponding to the file to be parsed and other software used for standardization conversion are configured in the isolated container. Of course, the parsing space can also be a slave server, that is, each file to be parsed corresponds to a slave server, and the configuration environment of the slave server is the same as that of the above-mentioned isolated container. Parsing in the parsing space can reduce the harm caused by some harmful files to the server itself. At the same time, it can reduce the conflicts brought by parsing different types of files in the same environment, such as conflicts and incompatibilities of software processes, so as to improve the efficiency, stability and security of parsing.
[0100] Furthermore, the configuration environment of the isolated container or the slave server should be pre-configured. Obtain the mapping table of file types and parsing spaces, then determine the corresponding parsing space for each file to be parsed based on the file type and the mapping table, and then allocate each file to be parsed to the corresponding parsing space based on the parsing device of each file to be parsed.
[0101] Furthermore, when parsing the file to be parsed in the parsing space, it can specifically include:
[0102] Compress the first type of file, and use the compressed file as the parsing file corresponding to the first type of file. After reading the second type of file, perform standardization conversion on the second type of file based on the file type to obtain the parsing file of the second type of file. The standardization conversion includes at least one of screenshot, screen recording, and text translation.
[0103] Specifically, when performing standardization conversion on the second type of file, it can specifically include:
[0104] If the standardization conversion includes screenshot, intercept N basic images, and determine the image with the highest clarity from the N basic images as the parsing file, where the parameters of any two basic images are different, and the parameters include screenshot position, screenshot perspective, and zoom ratio;
[0105] If the standardization conversion includes screen recording, the screen recording duration is determined based on the file size of the second type of file, and screen recording is performed based on the screen recording duration to obtain the parsing file;
[0106] If the standardization conversion includes text translation, translate the text files in at least two languages as the parsing file.
[0107] In the process of specific implementation, the corresponding relationship between file types and standardization operation methods should be pre-configured, and then corresponding standardization processing operations should be performed on the files to be parsed that are read and opened based on the file type.
[0108] Among them, different file types express different contents after being read. For example, if the file to be parsed is a.DWG file read by CAD software, the file shows the structural relationship in the form of two-dimensional lines. Based on this, screenshots can be taken of the.DWG file after it is read and opened. In order to make the parsed file more accurate and comprehensively display the content corresponding to the file to be parsed, when taking screenshots, multiple basic images with different parameters can be captured. The clarity of each basic image is identified through a clarity recognition algorithm or model, and then the basic image with the highest clarity is selected as the parsed file.
[0109] For another example, for a 3D model object described by a.3ds format file corresponding to 3DMAX software, simply taking screenshots cannot display the main features of the 3D model object. Therefore, for such files, a screen recording operation can be performed, that is, during the screen recording process, multiple-dimensional perspectives of the 3D model object need to be shown. In the embodiments of the present application, only these two types of file formats are used to exemplarily illustrate the process of parsing files, but this does not form a limitation on the file types that can be parsed by the embodiments of the present application.
[0110] Further, in step S50, sending each parsed file to the target device of the user may specifically include: compressing each parsed file and then sending it to the target device of the user.
[0111] In another embodiment, sending each parsed file to the target device of the user may further include: generating a download link corresponding to each parsed file, and then sending the download link corresponding to each parsed file to the target device of the user.
[0112] Specifically, generating the download link for each parsed file does not directly send each parsed file to the target device. After the user receives the download link, the user can select which files to download based on the download link. If the user chooses not to download, the corresponding file can be discarded to reduce the occupation of network resources.
[0113] The above embodiments introduce a data processing method from the perspective of the method flow. The following embodiments introduce a data processing device from the perspective of virtual modules or virtual units. For details, see the following embodiments.
[0114] The embodiments of the present application provide a data processing device, as Figure 5 shown. The data processing device may specifically include a receiving module 501, a screening condition acquisition module 502, a file to be parsed determination module 503, a parsing module 504, and a sending module 505, where:
[0115] A receiving module 501, configured to receive a set of links of target data sent by a user's target device, where there is at least one source link in the set of links, and each source link points to at least one source file;
[0116] A filtering condition obtaining module 502, configured to perform a filtering condition obtaining step to obtain a filtering condition;
[0117] A file to be parsed determining module 503, configured to determine whether there is a file to be parsed that meets the filtering condition in the set of files pointed to by the set of links, where the set of files includes the source files pointed to by each source link;
[0118] An analysis module 504, configured to obtain each file to be parsed and analyze each file to be parsed to obtain each parsed file;
[0119] A sending module 505, configured to send each parsed file to the user's target device;
[0120] Among them, the filtering condition obtaining module 502 is specifically configured to:
[0121] Send a test script to the target device to obtain a target type, where the target type is a file type that the target device cannot parse, and record the sending time when the test script is sent to the target device and the end time when the sending is completed;
[0122] Based on the sending time, the end time, and the volume of the test script, determine the network transmission rate with the target device, and obtain a volume threshold based on the network transmission rate;
[0123] Determine the filtering condition as: files with a file volume exceeding the volume threshold and not meeting the target type are the first type of files, files meeting the target type are the second type of files, and the files to be parsed include the first type of files and the second type of files.
[0124] In a possible implementation manner, when the file to be parsed determining module 503 determines whether there is a file to be parsed that meets the filtering condition in the set of files pointed to by the set of links, it is specifically configured to:
[0125] Obtain the attribute information of each source file in the set of files, where the attribute information includes the file type and the file volume;
[0126] Based on the attribute information of each source file, determine whether there is a file to be parsed that meets the filtering condition in the set of files pointed to by the set of links.
[0127] In a possible implementation manner, when the analysis module 504 analyzes each file to be parsed to obtain each parsed file, it is specifically configured to:
[0128] Allocate analysis space for each file to be parsed based on the file type, and the analysis space is configured as a container space;
[0129] Parse the corresponding file to be parsed in the parsing space to obtain each parsed file.
[0130] In a possible implementation manner, when the parsing module 504 parses the corresponding file to be parsed in the parsing space to obtain each parsed file, it is specifically used for:
[0131] Compress the first type of files in the parsing space, and determine the compressed files as the parsed files corresponding to the first type of files; and / or
[0132] In the parsing space, perform standard conversion on the second type of files based on the file type to obtain the parsed files of the second type of files, and the standard conversion includes at least one of screenshot, screen recording, and text translation.
[0133] In a possible implementation manner, when the parsing module 504 performs standard conversion on the second type of files, it is specifically used for:
[0134] If the standard conversion includes screenshot, intercept N basic images, and determine the image with the highest clarity among the N basic images as the parsed file, where the parameters of any two basic images are different, and the parameters include screenshot position, screenshot perspective, and scaling ratio;
[0135] If the standard conversion includes screen recording, the screen recording duration is determined based on the file size of the second type of files, and screen recording is performed based on the screen recording duration to obtain the parsed file;
[0136] If the standard conversion includes text translation, translate the text files in at least two languages as the parsed files.
[0137] In a possible implementation manner, when the sending module 505 sends each parsed file to the target device of the user, it is specifically used for:
[0138] Compress each parsed file and send it to the target device of the user; or
[0139] Generate a download link corresponding to each parsed file;
[0140] Send the download link corresponding to each parsed file to the target device of the user.
[0141] In the embodiments of the present application, a server is provided, as Figure 6 shown, Figure 6The server 600 shown includes: a processor 601 and a memory 603. Among them, the processor 601 and the memory 603 are connected, such as through a bus 602. Optionally, the server 600 may further include a transceiver 604. It should be noted that in practical applications, the transceiver 604 is not limited to one, and the structure of the server 600 does not constitute a limitation to the embodiments of the present application.
[0142] The processor 601 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logic blocks, modules, and circuits described in the disclosure of the present application. The processor 601 may also be a combination that implements computing functions, such as a combination including one or more microprocessors, a combination of a DSP and a microprocessor, etc.
[0143] The bus 602 may include a path for transmitting information between the above components. The bus 602 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus 602 may be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, Figure 6 only a thick line is used to represent it in the figure, but it does not mean that there is only one bus or one type of bus.
[0144] The memory 603 can be a ROM (Read Only Memory), or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory), or other types of dynamic storage devices that can store information and instructions. It can also be an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory), or other optical disc storage, optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media, or other magnetic storage devices, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.
[0145] The memory 603 is used to store the application program code for executing the solution of this application, and is controlled by the processor 601 for execution. The processor 601 is used to execute the application program code stored in the memory 603 to implement the content shown in the foregoing method embodiments.
[0146] It should be understood that although the steps in the flowchart of the accompanying drawings are shown in sequence according to the indication of the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise clearly stated in this article, there is no strict order restriction for the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same moment, but can be executed at different moments, and their execution order is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or sub-steps or stages of other steps.
[0147] The above are only partial embodiments of this application. It should be noted that for those of ordinary skill in the art in this technical field, without departing from the principle of this application, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of this application.
Claims
1. A data processing method, characterized in that: Executed by a server, the method includes: Receive a link set of target data sent by a target device of a user, wherein the link set contains at least one source link, and each source link points to at least one source file; Execute the filtering condition obtaining step to obtain the filtering condition; Determine whether there is a file to be parsed that meets the screening condition in the file set pointed to by the link set, wherein the file set includes the source file pointed to by each source link; If so, obtain each file to be parsed, and parse each file to be parsed to obtain each parsed file; Sending each parsed file to a target device of a user; The step of obtaining the screening conditions includes: Sending a test script to the target device to obtain a target type, where the target type is a file type that the target device cannot parse, and recording a sending time of sending the test script to the target device and an end time of sending completion; Determine a network transmission rate between the target device and the target device based on the sending time, the end time, and the volume of the test script, and obtain a volume threshold based on the network transmission rate; The screening condition is determined as follows: files whose file volume exceeds a volume threshold and do not conform to the target type are first-category files, and files that conform to the target type are second-category files, and the files to be parsed include first-category files and second-category files.
2. A data processing method according to claim 1, characterized in that: The step of determining whether there is a file to be parsed that meets the screening condition in the file set pointed to by the link set includes: Acquire attribute information of each source file in the file set, wherein the attribute information includes file type and file size; Based on the attribute information of each source file, it is determined whether there is a file to be parsed in the file set pointed to by the link set.
3. A data processing method according to claim 2, characterized in that: The step of parsing each to-be-parsed file to obtain each parsed file includes: Allocate a parsing space for each of the to-be-parsed files based on the file type, the parsing space being configured as a container space; The corresponding to-be-parsed files are parsed in the parsing space to obtain various parsed files.
4. A data processing method according to claim 3, characterized in that: The step of parsing the corresponding files to be parsed in the parsing space to obtain the parsed files includes: compressing the first type of files in the parsing space, and determining the compressed files as parsed files corresponding to the first type of files; and / or In the parsing space, the second category files are subjected to standardized conversion based on the file type to obtain parsed files of the second category files, wherein the standardized conversion includes at least one of screenshot, screen recording and translated text.
5. A data processing method according to claim 4, characterized in that: The step of performing standardized conversion on the second type of files includes: If the normalization conversion includes screenshots, N basic images are captured, and an image with the highest definition is determined as the parsed file from the N basic images, wherein parameters of any two basic images are different, and the parameters include a screenshot position, a screenshot viewing angle, and a zoom ratio; If the standardized conversion includes screen recording, the screen recording duration is determined based on the file volume of the second type of file, and the screen is recorded based on the screen recording duration to obtain a parsed file; If the standardization conversion includes translating text, text files in at least two languages are translated as parsed files.
6. A data processing method according to any one of claims 2 to 5, characterized in that: The sending of each parsed file to a target device of the user comprises: Compressing each parsed file and sending it to the user's target device; or Generate a download link corresponding to each of the parsed files; The download links corresponding to the respective parsed files are sent to the target device of the user.
7. A data processing device, characterized in that: include: A receiving module, configured to receive a link set of target data sent by a target device of a user, wherein the link set contains at least one source link, and each source link points to at least one source file; A screening condition acquisition module, used to execute a screening condition acquisition step to acquire the screening condition; A to-be-parsed file determination module, used to determine whether there is a to-be-parsed file that meets the screening condition in the file set pointed to by the link set, the file set including the source file pointed to by each source link; A parsing module, used for obtaining each file to be parsed, and parsing each file to be parsed to obtain each parsed file; A sending module, used for sending each parsed file to a target device of a user; Among them, the screening condition acquisition module is specifically used for: Sending a test script to the target device to obtain a target type, where the target type is a file type that the target device cannot parse, and recording a sending time of sending the test script to the target device and an end time of sending completion; Determine a network transmission rate between the target device and the target device based on the sending time, the end time, and the volume of the test script, and obtain a volume threshold based on the network transmission rate; The screening condition is determined as follows: files whose file volume exceeds a volume threshold and do not conform to the target type are first-category files, and files that conform to the target type are second-category files, and the files to be parsed include first-category files and second-category files.
8. A data processing device according to claim 7, characterized in that: When determining whether there is a file to be parsed that meets the screening condition in the file set pointed to by the link set, the to-be-parsed file determination module is specifically used to: Acquire attribute information of each source file in the file set, wherein the attribute information includes file type and file size; Based on the attribute information of each source file, it is determined whether there is a file to be parsed that meets the screening condition in the file set pointed to by the link set.
9. A server, characterized in that: The server includes: at least one processor; Memory; At least one application, wherein the at least one application is stored in a memory and configured to be executed by at least one processor, and the at least one application is configured to: execute the data processing method according to any one of claims 1-6.
10. A computer-readable storage medium, characterized in that: include: The computer program is stored and can be loaded by a processor to execute the data processing method according to any one of claims 1 to 6.