File generation method and device based on character coding, equipment, medium and program product

Abstract

The invention provides a file generation method and device based on character coding, equipment, a storage medium and a program product, and can be applied to the technical field of big data. The file generation method based on character coding comprises the following steps: analyzing a received target parameter from an upstream application, and determining a connection string for accessing a target database; the target parameter is used for defining a file generation format so as to ensure that the generated file can be correctly analyzed by a downstream application; calling a target interface in a target database based on the connection string so as to read an initial byte array from the target database based on the target interface; the target interface is an interface which is provided by a target database and is used for streaming data reading; coding the initial byte array based on the target character coding information to obtain a target byte array; and writing the target byte array into a pre-created initial file to obtain a target file.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and more specifically to a method, apparatus, device, medium, and program product for generating a file based on character encoding. Background Art

[0002] In industries like finance, data is often dispersed across multiple business systems. To support data analysis, report generation, or inter-system interaction, this dispersed data needs to be generated into files in a unified format for use by downstream systems. Character encoding maps characters in a character set into binary numbers that computers can store and process. This allows for the exchange of text data between different systems and programs. However, file formats are often hard-coded within programs, and upstream parameter changes require code modifications and redeployment, resulting in limited flexibility. Furthermore, the byte data stored in the database may differ from the target file encoding, and implicit conversions can easily lead to garbled characters. Summary of the Invention

[0003] In view of the above problems, the present application provides a character encoding-based file generation method, apparatus, device, medium and program product that improve the flexibility of file generation.

[0004] According to a first aspect of the present application, a method for generating a file based on character encoding is provided, comprising: parsing a target parameter received from an upstream application to determine a connection string for accessing a target database; the target parameter is used to define a format for file generation to ensure that the generated file can be correctly parsed by a downstream application; calling a target interface in the target database based on the connection string to read an initial byte array from the target database based on the target interface; the target interface is an interface provided by the target database for streaming data;

[0005] The initial byte array is encoded based on target character encoding information to obtain a target byte array; the target byte array is written into a pre-created initial file to obtain a target file.

[0006] According to an embodiment of the present application, calling a target interface in a target database based on a connection string so as to read an initial byte array from the target database based on the target interface includes: establishing a connection with the target database through the connection string and calling the target interface of the target database; reading raw data from the target database in a streaming manner through the target interface and buffering the raw data into an initial byte array.

[0007] According to an embodiment of the present application, raw data is read from a target database in a streaming manner through a target interface and the raw data is buffered as an initial byte array, including: reading the raw data in the target database in blocks based on a preset cache area; encoding the raw data read in blocks based on source character encoding information in a connection string and storing the encoded raw data as an initial byte array, wherein the source character encoding information is used to unify the encoding format of the raw data in the database.

[0008] According to an embodiment of the present application, encoding an initial byte array based on target character encoding information to obtain a target byte array includes: decoding the initial byte array based on source character encoding information in a connection string to obtain an intermediate character string; encoding the intermediate character string based on the target encoding information to obtain a target byte array.

[0009] According to an embodiment of the present application, a target byte array is written into a pre-created initial file to obtain a target file, including: generating an initial file according to file configuration information in a target parameter; and writing a target byte array into the initial file based on target encoding information to generate a target file.

[0010] According to an embodiment of the present application, a target byte array is written into an initial file based on target encoding information to generate a target file, including: formatting the initial content written into the initial file based on format information in the target parameters to generate a target file containing structured data.

[0011] According to an embodiment of the present application, it also includes: before writing the target byte array into the initial file, opening a file output stream, the file output stream is used to pass the target byte array to the initial file; calling a write method through the opened file output stream to write the data in the target byte array into the initial file; in response to completion of writing all data in the target file, closing the file output stream.

[0012] The second aspect of the present application provides a file generation device based on character encoding, including: a determination module, used to parse the target parameters received from the upstream application and determine the connection string for accessing the target database; the target parameters are used to define the format of file generation to ensure that the generated file can be correctly parsed by the downstream application; a calling module, used to call the target interface in the target database based on the connection string, so as to read the initial byte array from the target database based on the target interface; the target interface is an interface provided by the target database for streaming data; an encoding module, used to encode the initial byte array based on the target character encoding information to obtain a target byte array; and a writing module, used to write the target byte array into a pre-created initial file to obtain a target file.

[0013] The third aspect of the present application provides an electronic device, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0014] The fourth aspect of the present application further provides a computer-readable storage medium having a computer program or instructions stored thereon, which implements the steps of the above method when the computer program or instructions are executed by a processor.

[0015] The fifth aspect of the present application further provides a computer program product, comprising a computer program or instructions, which implement the steps of the above method when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above contents and other objects, features and advantages of the present application will become more apparent through the following description of the embodiments of the present application with reference to the accompanying drawings, in which:

[0017] Figure 1 Schematically illustrates an application scenario diagram of a character encoding-based file generation method, apparatus, device, medium, and program product according to an embodiment of the present application;

[0018] Figure 2 A flowchart of a method for generating a file based on character encoding according to an embodiment of the present application is schematically shown;

[0019] Figure 3 A flowchart schematically illustrates calling a target interface in a target database based on a connection string, so as to read an initial byte array from the target database based on the target interface, according to an embodiment of the present application;

[0020] Figure 4 A flowchart of reading original data from a target database in a streaming manner through a target interface and buffering the original data into an initial byte array according to an embodiment of the present application is schematically shown;

[0021] Figure 5 A flowchart of encoding an initial byte array based on target character encoding information to obtain a target byte array according to an embodiment of the present application is schematically shown;

[0022] Figure 6 A flowchart of writing a target byte array into a pre-created file to obtain a target file according to an embodiment of the present application is schematically shown;

[0023] Figure 7 A flowchart of writing data in a target byte array into an initial file based on a file output stream according to an embodiment of the present application is schematically shown;

[0024] Figure 8 A block diagram schematically illustrates a structure of a device for generating a file based on character encoding according to an embodiment of the present application; and

[0025] Figure 9 A block diagram of an electronic device suitable for implementing a file generation method based on character encoding according to an embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0026] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present application. In the detailed description below, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present application. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessarily confusing the concepts of the present application.

[0027] The terms used herein are only for describing specific embodiments and are not intended to limit the present application. The terms "comprise," "include," etc. used herein indicate the presence of features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0028] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0029] When expressions such as "at least one of A, B, and C, etc." are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0030] It should be noted that the character encoding-based file generation method and device provided in this application can be used in the field of financial technology, and can also be used in any field other than the field of financial technology. The application field of the character encoding-based file generation method and device provided in this application is not limited.

[0031] The embodiment of the present application provides a file generation method based on character encoding conversion, comprising: parsing target parameters received from an upstream application to determine a connection string for accessing a target database; the target parameters are used to define the format of file generation to ensure that the generated file can be correctly parsed by downstream applications; calling a target interface in the target database based on the connection string to read an initial byte array from the target database based on the target interface; the target interface is an interface provided by the target database for streaming data reading; encoding the initial byte array based on the target character encoding information to obtain a target byte array; writing the target byte array into a pre-created initial file to obtain a target file

[0032] Figure 1 The application scenario diagram of the character encoding-based file generation method according to an embodiment of the present application is schematically shown.

[0033] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links or optical fiber cables.

[0034] A user may use a first terminal device 101, a second terminal device 102, or a third terminal device 103 to interact with a server 105 via a network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, or the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (for example only).

[0035] The first terminal device 101 , the second terminal device 102 , and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0036] The server 105 may be a server that provides various services, such as a background management server (for example only) that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process received data such as user requests, and feed back processing results (e.g., web pages, information, or data obtained or generated based on user requests) to the terminal devices.

[0037] It should be noted that the character encoding-based file generation method provided in the embodiment of the present application can generally be executed by the server 105. Accordingly, the character encoding-based file generation device provided in the embodiment of the present application can generally be set in the server 105. The character encoding-based file generation method provided in the embodiment of the present application can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the character encoding-based file generation device provided in the embodiment of the present application can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0038] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0039] The character encoding-based file generation method and apparatus provided in this application can be used in the field of financial technology. For example, in banking systems, a large number of transaction records are generated daily, including deposits, withdrawals, transfers, and other transactions. These transaction records need to be generated into files according to a specific format so that they can be sent to other financial institutions (such as payment and clearing institutions) for clearing, or used by other internal bank systems (such as financial systems and risk management systems) for subsequent processing and analysis. Alternatively, in securities trading, securities companies need to provide timely feedback on client transaction information to clients and also report transaction data to regulatory agencies such as exchanges. Transaction report files must contain important information such as transaction date, security code, transaction volume, transaction price, and buy and sell direction, and must be generated in a prescribed format to ensure accurate information transmission and regulatory compliance. Furthermore, the character encoding-based file generation method and apparatus provided in this application can also be used in any field beyond financial technology, and the application field of the character encoding-based file generation method and apparatus provided in this application is not limited.

[0040] The following will be based on Figure 1 The scene described by Figures 2 to 7 A character encoding-based file generation method according to an embodiment of the present application is described in detail.

[0041] Figure 2 The flowchart of the character encoding-based file generation method according to an embodiment of the present application is schematically shown.

[0042] like Figure 2 As shown, the character encoding-based file generation of this embodiment includes operations S210 to S230.

[0043] In operation S210 , the target parameters received from the upstream application are parsed to determine a connection string for accessing the target database.

[0044] In some embodiments, the target parameters may include, for example, target parameters from upstream applications, such as command line parameters, Application Programming Interface Request (API) requests, configuration files, etc. The target parameters are used to define the format of file generation, which may be specified by the upstream application based on the needs of the downstream application to ensure that the generated file can be correctly parsed by the downstream application. Exemplarily, the upstream application may be a data producer or a core business system, which is responsible for the creation, storage or processing of raw data. The downstream application may be a data consumer or a new business support system, which relies on upstream data to achieve specific business goals. Exemplarily, the upstream application may be an order management system, which needs to regularly export order data to files in a specified data exchange format for import into the financial system.

[0045] Exemplarily, the target parameters may include: a query statement for data extraction (by executing the statement in the target database, the desired exported data result set can be locked, including a specified field list and order), a connection string for the target database (which may include the target database address, port, user name, password, and source character encoding information, etc.), specified character encoding parameters for the generated file (i.e., target character encoding information), the directory for the generated file, the file name, line break character, field separator, and other parameters.

[0046] In operation S220 , a target interface in the target database is called based on the connection string, so as to read an initial byte array from the target database based on the target interface; the target interface is an interface provided by the target database for streaming data reading.

[0047] In some embodiments, the target database may be a Gaussian database, and the target interface may be the Gaussian database's CopyManager interface. The CopyManager interface is a high-performance data export interface provided by the Gaussian database that enables streaming data export. The original data of the result set, i.e., the initial byte array, may be obtained by executing a query statement. The encoding of the data returned by the target database may be determined by the source character encoding information in the connection string.

[0048] In operation S230 , the initial byte array is encoded based on the target character encoding information to obtain a target byte array.

[0049] In some embodiments, the initial byte array is binary data read from the target database, and its encoding is usually determined by the target database connection string or the system default encoding. If the initial byte array is directly written to a file without encoding conversion, it may cause garbled characters to be parsed by the target system. For example, the Chinese character "测试" is B2 E2 CA D4 in the first encoding format. If it is directly written to a file in the second encoding format, it will be displayed as garbled characters ²âÊ´. Therefore, it is necessary to encode the initial byte array according to the target character encoding information to obtain the target byte array. It should be noted that the target character encoding information is the encoding information consistent with the expected encoding of the target file, and the target character encoding information can be specified by the upstream application according to the requirements of the downstream application.

[0050] In operation S240, write the target byte array to the pre-created initial file to obtain the target file.

[0051] In some embodiments, create a file according to the parameters to ensure that the file directory exists and has write permission, and write the target byte array to the pre-created initial file to obtain the target file. Exemplarily, the downstream application can periodically check the directory for storing the target file. After discovering a new target file, read the target file and import the data of the target file into its own database to complete a specific business objective.

[0052] The file generation method provided by the embodiments of the present application ensures the correct representation of data in the file through explicit encoding conversion. Different databases and file formats can be flexibly adapted through target parameters (such as connection strings, target character encoding information, etc.). All configurations are passed in through parameters, which can reduce hard coding in the code. The same upstream can provide files in a unified format for different downstreams, and only by adjusting the target parameters can the requirements of different downstream applications be adapted, effectively improving the flexibility of file generation and the efficiency of data flow.

[0053] Figure 3 A flowchart is schematically shown for calling a target interface in a target database based on a connection string according to an embodiment of the present application, so as to read an initial byte array from the target database based on the target interface.

[0054] As Figure 3 shown, the embodiment of calling a target interface in a target database based on a connection string to read an initial byte array from the target database based on the target interface includes operations S310 to S320.

[0055] In operation S310, establish a connection with the target database through the connection string and call the target interface of the target database.

[0056] In some embodiments, the upstream application selects the connection string format based on the database type. In this embodiment, the target database may be a Gaussian database, and the connection string may be the Gaussian database's jbdc connection string. The connection string may include information such as the target database server address and the target database name. A connection to the target database can be established through a connection pool or a direct connection. Establishing a database connection through a connection pool involves pre-creating and maintaining a set of reusable database connections, avoiding the performance overhead of frequently creating and destroying connections and improving system throughput and response speed.

[0057] Exemplarily, the process of establishing a database connection through a connection pool may include: adding a connection pool dependency to the project, configuring database connection information (for example, it may include a Gaussian database connection string, user name / password, maximum number of connections, idle connection timeout, connection lifetime, etc.), initializing the connection pool and creating a connection, obtaining the connection through the specified method of the connection pool and converting the connection into a PGConnection object, establishing a connection with the target database, and explicitly closing the connection after the connection pool is used up (marking the connection as idle, not actually destroying it, for reuse by subsequent requests). Establishing a connection to the target database by means of a direct connection includes: establishing a new physical connection with the target database based on the connection string, and destroying the connection after the physical connection is used up.

[0058] In some embodiments, the target interface can be the CopyManager interface. CopyManager is an extended interface provided by Gaussian Database that supports efficient data import / export via the COPY command, making it particularly suitable for batch data operations. A CopyManager instance can be obtained through a PGConnection object to call the target database's target interface.

[0059] In operation S320 , original data is read from the target database in a streaming manner through the target interface and buffered as an initial byte array.

[0060] In some embodiments, source character encoding information is obtained from a configuration file, database connection configuration (such as the characterEncoding parameter in a specific connection string), or parameters passed in during an interface call. If no encoding is explicitly specified, the system default encoding or the database driver's default encoding may be used. In this application, the source character encoding can be obtained from the JDBC connection string, and the source character encoding can be the Extended Standard for Chinese Internal Code (GBK). The target database driver's streaming read method is used to directly obtain the data stream. For example, data can be obtained from the stream block by block through a buffer, resulting in multiple discrete data blocks. This streaming method directly obtains the data stream rather than the entire data, eliminating the need for single-byte reads and reducing the number of I / O operations. The discrete data blocks read in the streaming method are merged into a complete byte array (i.e., the initial byte array) for subsequent processing. For example, an expandable buffer (such as a structure similar to a "dynamic array") can be used to gradually concatenate data blocks. The initial capacity can be set to a fixed value (e.g., 8KB), which automatically expands based on the data size. Each data block is sequentially appended to the end of the buffer, ultimately generating a continuous byte array. Check whether the data is read completely (for example, whether the database field is NULL) to avoid generating empty or truncated byte arrays.

[0061] The embodiments of the present application avoid loading large files into memory all at once through streaming reading, processing them in blocks only when needed, significantly reducing memory usage. Dynamic buffering is used to expand memory on demand, reducing memory waste in fixed-size arrays and achieving low memory usage. By explicitly parsing the source code and enforcing unified encoding, the problem of garbled characters across databases or systems is resolved, and implicit encoding settings that rely on database connection strings are avoided, improving encoding accuracy. Coding flexibility is expanded through dynamic configuration of the code, allowing the same program to process databases with different encodings.

[0062] Figure 4 The flowchart of reading original data from a target database in a streaming manner through a target interface and buffering the original data into an initial byte array according to an embodiment of the present application is schematically shown.

[0063] like Figure 4 As shown, the embodiment of the present invention includes operations S410 to S420, which include reading original data from a target database in a streaming manner through a target interface and buffering the original data into an initial byte array.

[0064] In operation S410 , original data in a target database is read in blocks based on a preset buffer area.

[0065] In some embodiments, a fixed-size byte array (e.g., 8KB, 64KB, or 1MB) can be defined as a buffer. The buffer size can be determined based on data characteristics and memory limitations. For example, a larger buffer can reduce I / O operations but increase memory usage. The data stream can be obtained through the streaming interface (i.e., the CopyManager interface) provided by the database driver. Blocks of data equal to the buffer size can be read from the stream until all data has been read (the stream returns -1 to indicate completion). During the reading process, the current read offset (e.g., the total number of bytes read) can be recorded for resuming transfers or tracking progress. If the data block size is insufficient for the buffer capacity (e.g., the last block is only 1KB), the processing logic must be dynamically adjusted to avoid reading invalid data. Furthermore, the database field is checked for NULL. If so, the read is skipped or an empty byte array is returned.

[0066] In operation S420 , the original data read in blocks is encoded based on the source character encoding information in the connection string and the encoded original data is stored as an initial byte array. The source character encoding is used to re-encode the original data of the database.

[0067] In some embodiments, for each read data block, it can be converted according to the source character encoding (such as decoding binary data into a string, and then re-encoding it into a byte array as needed). The original data of the database may be mixed encoding, and the original data can be uniformly converted into a standardized byte array through the source character encoding information. For example, if the original data is binary (such as a picture, PDF), no encoding is required, and the byte array can be directly spliced. If the original data is text (such as JSON, XML), it needs to be converted into a string according to the encoding, and then converted back to a byte array (or directly spliced ​​into byte blocks). A dynamic buffer (such as ByteArrayOutputStream) or a pre-allocated large array can be used to splice all encoded data blocks into a complete byte array in sequence to ensure that the splicing order is consistent with the reading order to avoid data confusion.

[0068] This embodiment of the application uses block-by-block reading and dynamic splicing to avoid loading large files (such as a 10GB video) into memory all at once. Only the currently processed data block is retained in memory, significantly reducing peak memory usage. By gradually expanding storage space through a dynamic buffer (such as ByteArrayOutputStream), the memory waste of pre-allocated large arrays can be reduced.

[0069] Figure 5 The flowchart of encoding an initial byte array based on target character encoding information to obtain a target byte array according to an embodiment of the present application is schematically shown.

[0070] like Figure 5As shown, the embodiment encodes the initial byte array based on the target character encoding information to obtain the target byte array, including operations S510 to S520.

[0071] In operation S510 , the initial byte array is decoded based on source character encoding information in the connection string to obtain an intermediate character string.

[0072] In operation S520 , the intermediate character string is encoded based on the target encoding information to obtain a target byte array.

[0073] In some embodiments, each byte or byte combination in the byte array is interpreted as a corresponding character based on the rules of the source character encoding, and the characters are concatenated into an intermediate string. For example, the initial bytes Source_Bytes[] can be parsed into a Unicode string (i.e., a unified character representation within the program) based on the source encoding. It should be noted that the source character encoding must be consistent with the actual encoding of the byte array; otherwise, decoding errors will result in garbled characters in the string.

[0074] Each character in the intermediate string is converted into a corresponding byte or byte combination according to the rules of the target encoding to generate a target byte array. For example, the Unicode string can be re-encoded into a new byte array Destination_Bytes[] according to the target encoding information, i.e., the target byte array.

[0075] In some embodiments, encoding conversion through an intermediate string can effectively improve the security of encoding conversion. For example, the source character encoding can be GBK encoding. GBK includes a total of 21,003 Chinese characters. GB18030 is a newer character set standard that expands more characters, including traditional Chinese characters, rare characters, Japanese and Korean characters, and minority languages. GB18030-2000 includes a total of 27,533 Chinese characters. Therefore, when converting from a large character set to a small character set, such as when converting from GB18030 to GBK, some Chinese characters exceed the GBK character set range. Forcibly using the GB18030 encoding as the GBK encoding will result in garbled characters for these Chinese characters. However, by using an intermediate character transition method, the GB18030 byte array is first converted to the Unicode encoding of the String, and then converted to the GBK character encoding. This can filter out these out-of-range fields during the encoding process and convert them into spaces (and then remove the spaces), thereby ensuring that valid data can be successfully converted. Compared with traditional pattern matching replacement operations, it is also faster and more efficient.

[0076] The embodiments of the present application safely convert byte data of different encodings through intermediate strings, avoiding garbled characters caused by direct operation of bytes. The complex byte-to-byte conversion is split into two clear operations (byte → string → byte), which is convenient for debugging and problem location. For example: if the final result is garbled, you can check separately whether the encoding parameters of the decoding or encoding step are correct. Strings as intermediate results are easier to read and verify (such as printing logs), while it is difficult to intuitively understand the content by directly operating byte arrays. By parameterizing the control of source / target encoding, it adapts to the data exchange needs of multiple sources and multiple platforms. Strings as intermediate results are convenient for debugging and verification, reducing the complexity of encoding conversion and the accuracy of encoding conversion.

[0077] Figure 6 The flowchart of writing a target byte array into a pre-created initial file to obtain a target file according to an embodiment of the present application is schematically shown.

[0078] like Figure 6 As shown, in this embodiment, writing the target byte array into the pre-created initial file to obtain the target file includes operations S610 to S620.

[0079] In operation S610, an initial file is generated according to the file configuration information in the target parameters.

[0080] In operation S620, the target byte array is written into the initial file based on the target encoding information to generate a target file.

[0081] In some embodiments, the target file's storage location and name can be determined based on target parameters, and an initial file can be created based on the specified path and file name. For example, the initial file can be generated by overwriting a file with the same name or creating a new one. The target byte array is written as a whole to the initial file to form the initial content. The target encoding must be specified during writing to ensure that the bytes and encoding match. If the byte array already contains line breaks, a multi-line structure will automatically be formed after writing.

[0082] This application explicitly controls the target file generation process through target encoding information. The initial file is dynamically generated based on target parameters, which provides high flexibility and dynamic adaptability. The target encoding information is passed as an independent parameter, and when writing, the byte array is uniformly processed according to the encoding specified by the parameter, achieving decoupling of encoding and format, reducing the coupling degree.

[0083] In some embodiments, writing the target byte array into the initial file may further include: adjusting the format of the initial content written into the initial file based on the format information in the target parameters to generate a target file including structured data.

[0084] In some embodiments, the initial content read from the database can be converted into a structured file based on pre-defined formatting rules (i.e., format information). For example, tabular data can be converted into a CSV file that can be directly read by downstream systems, ensuring that field delimiters, encoding, null value handling, and other specifications meet downstream requirements. Exemplarily, format information can be determined by upstream applications based on downstream requirements to guide data conversion operations.

[0085] Exemplarily, the format information may include, for example, field separation mode, line break separation mode, etc. The text may be segmented into multiple lines of data according to line breaks (e.g., ["line 1 content", "line 2 content"]). Each line of data is further segmented according to the field separator to generate structured data (e.g., [["field 1", "field 2"], ["field A", "field B"]]. The processed structured data is rewritten to the file (optionally in overwrite or append mode) to generate the target file. Exemplarily, line characters and separators may be pre-replaced when writing to the byte array to avoid the overhead of subsequent segmentation and reorganization. If the amount of data is extremely large, it may be written line by line instead of being loaded into memory as a whole.

[0086] The embodiments of this application ensure that the target file format meets the requirements of the target system by explicitly defining line breaks and field separators, adapting to the differences in line breaks between different operating systems and achieving data normalization. Compared to character-by-character operations, writing directly into a byte array is more efficient and is particularly suitable for large files. In addition, line breaks and separators can be dynamically adjusted through parameters to adapt to different scenarios, increasing the flexibility of target file generation.

[0087] Figure 7 The flowchart of writing data in a target byte array into an initial file based on a file output stream according to an embodiment of the present application is schematically shown.

[0088] like Figure 7 As shown, writing the data in the target byte array into the initial file based on the file output stream in this embodiment includes operations S710 to S730.

[0089] In operation S710 , before writing the target byte array to the initial file, a file output stream is opened, where the file output stream is used to transfer the target byte array to the initial file.

[0090] In operation S720, a write method is called through the opened file output stream to write the data in the target byte array into the initial file.

[0091] In operation S730 , in response to completion of writing all data in the target file, the file output stream is closed.

[0092] In some embodiments, when writing the target byte array to a pre-created initial file, a file output stream is opened. Exemplarily, in code (or at the logical level), a File object is generated based on the file path, and methods related to the file output stream are called to open the file in write mode. An output stream object pointing to the file is initialized, and subsequent data is written to the file through this output stream object. Exemplarily, the file output stream can be a buffered stream, which can cache data in memory, reducing direct disk I / O.

[0093] By using an output stream object, the target byte array is passed to the original file all at once or in chunks.

[0094] After all data has been read from the target database and written to the initial file, explicitly call the file output stream's close method to close the file output stream and release system resources. If the file output stream is a buffered stream, in response to the close method being called, the remaining data cached in memory is forcibly written to disk to ensure data integrity. For example, the file output stream can be closed first, and then the connection to the target database can be closed. If an exception occurs during the writing process, the file output stream can also be forcibly closed in the exception handling logic to avoid resource leaks.

[0095] The embodiments of the present application write data via an output stream, ensuring that all data in the byte array is written sequentially, improving data integrity and reliability. Explicitly closing the file output stream promptly releases the file handle and memory buffer, preventing system resource exhaustion. Forcing the buffer data to be written to disk before closing effectively prevents data loss in the event of a program crash and maintains data integrity.

[0096] Based on the above-mentioned file generation method based on character encoding, the present application also provides a file generation device based on character encoding. Figure 8 The device is described in detail.

[0097] Figure 8 The structural block diagram of the character encoding-based file generation device according to an embodiment of the present application is schematically shown.

[0098] like Figure 8 As shown, the character encoding-based file generation device 800 of this embodiment includes a determination module 810 , a calling module 820 , an encoding module 830 and a writing module 840 .

[0099] Determination module 810 is used to parse the target parameters received from the upstream application and determine the connection string used to access the target database. The target parameters are used to define the format of the generated file to ensure that the generated file can be correctly parsed by the downstream application. In one embodiment, determination module 810 can be used to perform operation S210 described above, and will not be repeated here.

[0100] Calling module 820 is configured to call a target interface in the target database based on the connection string to read an initial byte array from the target database based on the target interface. The target interface is an interface provided by the target database for streaming data. In one embodiment, calling module 820 can be configured to perform operation S220 described above, which will not be further described here.

[0101] The encoding module 830 is used to encode the initial byte array based on the target character encoding information to obtain the target byte array. In one embodiment, the encoding module 830 can be used to perform the operation S230 described above, which will not be repeated here.

[0102] The writing module 840 is used to write the target byte array into the pre-created initial file to obtain the target file. In one embodiment, the writing module 840 can be used to perform the operation S240 described above, which will not be repeated here.

[0103] According to embodiments of the present application, any multiple modules among the determination module 810, the call module 820, the encoding module 830, and the writing module 840 may be combined into a single module, or any one of these modules may be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules may be combined with at least part of the functionality of other modules and implemented in a single module. According to embodiments of the present application, at least one of the determination module 810, the call module 820, the encoding module 830, and the writing module 840 may be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or may be implemented in hardware or firmware through any other reasonable means of circuit integration or packaging, or may be implemented in any one of the three implementation methods of software, hardware, and firmware, or any appropriate combination of these. Alternatively, at least one of the determination module 810, the call module 820, the encoding module 830, and the writing module 840 may be at least partially implemented as a computer program module that, when executed, performs the corresponding functionality.

[0104] Figure 9 A block diagram of an electronic device suitable for implementing a file generation method based on character encoding according to an embodiment of the present application is schematically shown.

[0105] like Figure 9As shown, an electronic device 900 according to an embodiment of the present application includes a processor 901, which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 902 or programs loaded from a storage unit 908 into a random access memory (RAM) 903. The processor 901 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or related chipsets and / or a dedicated microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to the embodiment of the present application.

[0106] Various programs and data required for the operation of the electronic device 900 are stored in the RAM 903. The processor 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. The processor 901 performs various operations of the method flow according to the embodiment of the present application by executing the programs in the ROM 902 and / or the RAM 903. It should be noted that the programs may also be stored in one or more memories other than the ROM 902 and the RAM 903. The processor 901 may also perform various operations of the method flow according to the embodiment of the present application by executing the programs stored in one or more memories.

[0107] According to an embodiment of the present application, electronic device 900 may further include an input / output (I / O) interface 905, which is also connected to bus 904. Electronic device 900 may also include one or more of the following components connected to I / O interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including devices such as a cathode ray tube (CRT), liquid crystal display (LCD), and speakers; a storage section 908 including a hard disk; and a communication section 909 including a network interface card such as a LAN card or modem. Communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to I / O interface 905 as needed. Removable media 911, such as a magnetic disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed in drive 910 as needed, so that computer programs read from the removable media can be installed into storage section 908 as needed.

[0108] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not be incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, and when the one or more programs are executed, the method according to the embodiments of this application is implemented.

[0109] According to an embodiment of the present application, a computer-readable storage medium may be a non-volatile computer-readable storage medium, and may include, for example, but not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present application, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present application, a computer-readable storage medium may include the ROM 902 and / or RAM 903 described above and / or one or more memories other than ROM 902 and RAM 903.

[0110] The embodiments of the present application also include a computer program product, which includes a computer program containing program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the character encoding-based file generation method provided in the embodiments of the present application.

[0111] The computer program executes the above functions defined in the system / device of the embodiment of the present application when the processor 901 executes the computer program. According to the embodiment of the present application, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0112] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal on a network medium, downloaded and installed via the communication portion 909, and / or installed from a removable medium 911. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0113] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 909, and / or installed from a removable medium 911. When the computer program is executed by the processor 901, the above-mentioned functions defined in the system of the embodiment of the present application are performed. According to the embodiment of the present application, the systems, devices, means, modules, units, etc. described above can be implemented by computer program modules.

[0114] According to an embodiment of the present application, the program code for executing the computer program provided by the embodiment of the present application can be written in any combination of one or more programming languages. Specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).

[0115] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of the boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0116] Those skilled in the art will appreciate that the features described in the various embodiments of this application may be combined and / or coupled in various ways, even if such combinations or couplings are not explicitly described in this application. In particular, the features described in the various embodiments of this application may be combined and / or coupled in various ways without departing from the spirit and teachings of this application. All such combinations and / or couplings fall within the scope of this application.

Claims

1. A method for generating a file based on character encoding, characterized in that: The method comprises: Parse the target parameters received from the upstream application to determine the connection string used to access the target database; the target parameters are used to define the format of the generated file to ensure that the generated file can be correctly parsed by the downstream application; calling a target interface in a target database based on the connection string, so as to read an initial byte array from the target database based on the target interface; the target interface is an interface provided by the target database for streaming data reading; Encoding the initial byte array based on target character encoding information to obtain a target byte array; The target byte array is written into a pre-created initial file to obtain a target file.

2. The file generation method according to claim 1, wherein: The calling of a target interface in a target database based on the connection string so as to read an initial byte array from the target database based on the target interface includes: Establishing a connection with the target database through the connection string and calling the target interface of the target database; Raw data is read from a target database in a streaming manner through a target interface and buffered as an initial byte array.

3. The file generation method according to claim 2, wherein: The step of reading raw data from a target database in a streaming manner through a target interface and buffering the raw data into an initial byte array includes: Reading the original data in the target database in blocks based on a preset buffer area; The original data read in blocks is encoded based on the source character encoding information in the connection string and the encoded original data is stored as an initial byte array. The source character encoding information is used to unify the encoding format of the original data in the database.

4. The file generation method according to claim 1, wherein: The encoding of the initial byte array based on the target character encoding information to obtain the target byte array includes: Decoding the initial byte array based on source character encoding information in the connection string to obtain an intermediate character string; The intermediate character string is encoded based on the target encoding information to obtain a target byte array.

5. The file generation method according to claim 1, wherein: The step of writing the target byte array into a pre-created initial file to obtain the target file comprises: Generate an initial file according to the file configuration information in the target parameters; The target byte array is written into the initial file based on the target encoding information to generate a target file.

6. The file generation method according to claim 5, characterized in that: The step of writing the target byte array into the initial file based on the target encoding information to generate the target file comprises: The format of the initial content written into the initial file is adjusted based on the format information in the target parameters to generate a target file containing structured data.

7. The file generation method according to claim 5, characterized in that: The method further comprises: Before writing the target byte array into the initial file, opening a file output stream, wherein the file output stream is used to transfer the target byte array to the initial file; Call the write method through the opened file output stream to write the data in the target byte array into the initial file; In response to completion of writing all data in the target file, the file output stream is closed.

8. A file generation device based on character encoding, characterized in that: The device comprises: A determination module is used to parse target parameters received from an upstream application and determine a connection string for accessing a target database; the target parameters are used to define the format of file generation to ensure that the generated file can be correctly parsed by the downstream application; a calling module, configured to call a target interface in a target database based on the connection string, so as to read an initial byte array from the target database based on the target interface; the target interface is an interface provided by the target database for streaming data; an encoding module, configured to encode the initial byte array based on target character encoding information to obtain a target byte array; and The writing module is used to write the target byte array into a pre-created initial file to obtain a target file.

9. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

11. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images