Heterogeneous file generation method and device, equipment, medium and product

Abstract

The invention provides a heterogeneous file generation method and device, equipment, a medium and a product. The method comprises the steps that before a binary code of any processor architecture is written into a heterogeneous file, whether a binary code of a first processor architecture corresponding to the processor architecture exists in the heterogeneous file or not is determined; if yes, metadata of binary codes of the processor architecture is stored in a storage area of the heterogeneous file, the binary codes of the processor architecture do not execute write-in operation, and the metadata comprises identification information of the processor architecture, offset and the size of a memory occupied by the binary codes. The offset and the occupied memory size of the binary code of the processor architecture are the same as the offset and the occupied memory size of the binary code of the first processor architecture. According to the method and the device, repeated binary codes are not stored repeatedly, and multiplexing of a multiprocessor architecture can be realized while the file volume is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, and in particular to a heterogeneous file generation method and device, equipment, medium and product. BACKGROUND

[0002] Executable and Linkable Format (ELF) is adapted to different processor architectures and operating systems, and is one of the main executable file formats in computer systems. A fat binary file is a file containing binary codes of multiple target architectures or device types generated based on ELF, and is usually used to support the running of programs on multiple processor architectures or device types.

[0003] In related technologies, all supported architectures are embedded into a fat binary file during program compilation, resulting in a file size that is too large. Therefore, how to reduce the file size while supporting heterogeneous systems is a problem that needs to be solved. SUMMARY

[0004] The present application provides a heterogeneous file generation method and device, equipment, medium and product for reducing the file size of a fat binary file.

[0005] In a first aspect, the present application provides a heterogeneous file generation method, comprising:

[0006] obtaining binary codes of multiple processor architectures;

[0007] Before writing the binary code of any one processor architecture into a heterogeneous file, determining whether the binary code of a first processor architecture corresponding to the processor architecture exists in the heterogeneous file; if so, storing metadata of the binary code of the processor architecture in a storage area of the heterogeneous file, and the binary code of the processor architecture does not perform a write operation, the metadata including identification information of the processor architecture, an offset and a memory size occupied by the binary code;

[0008] The offset and the memory size occupied by the binary code of the processor architecture are the same as the offset and the memory size occupied by the binary code of the first processor architecture, so that the processor architecture reuses the binary code of the first processor architecture; the offset is the length between the start position of the storage area of the heterogeneous file and the start position of the area occupied by the binary code of the processor architecture in the storage area; the binary code of the first processor architecture and the metadata of the binary code are stored in the heterogeneous file.

[0009] In the embodiment, before writing binary code of any processor architecture into a heterogeneous file, it is determined whether binary code of a first processor architecture corresponding to the processor architecture exists in the heterogeneous file; if yes, metadata of the binary code of the processor architecture is stored into a storage area of the heterogeneous file, the binary code of the processor architecture does not perform a writing operation, and an offset and a memory size occupied by the binary code of the processor architecture are the same as an offset and a memory size occupied by the binary code of the first processor architecture, so that the processor architecture can reuse the binary code of the first processor architecture, and the binary code of the first processor architecture and the metadata of the binary code are stored in the heterogeneous file. The application does not store repeated binary code repeatedly, and through storage of the metadata, reuse of multiple processor architectures can be realized while reducing file size.

[0010] In a possible implementation, the determining whether the binary code of the first processor architecture corresponding to the processor architecture exists in the heterogeneous file comprises:

[0011] The first field is used for describing basic information of the binary code, and the second field is used for describing annotation information of the binary code.

[0012] According to the first field and the second field, it is determined whether the binary code of the first processor architecture exists in the heterogeneous file.

[0013] In the embodiment, according to the field where the metadata of the binary code is located and the field where the annotation information is located, it is determined whether the first processor architecture compatible with the processor architecture exists, so that storage of repeated binary code can be reduced, and file size can be reduced.

[0014] In a possible implementation, the metadata further comprises a hash value of the binary code; and the determining whether the binary code of the first processor architecture exists in the heterogeneous file according to the first field and the second field comprises:

[0015] The first field and the second field of the binary code of at least one processor architecture stored in the heterogeneous file are acquired.

[0016] If there is a processor architecture meeting a condition in the at least one processor architecture, it is determined that the binary code of the first processor architecture exists in the heterogeneous file.

[0017] The condition comprises:

[0018] The first value in the first field of the binary code is the same as the first value in the first field of the binary code of the processor architecture, the first value is a value in the first field except for a third value, and the third value is a flag bit related to the processor architecture;

[0019] The second value in the second field of the binary code is the same as the second value in the second field of the binary code of the processor architecture, the second value is a value in the second field except for a fourth value, and the fourth value indicates a specific processor architecture environment required by the binary code to run;

[0020] The hash value of the binary code is the same as the hash value of the binary code of the processor architecture.

[0021] In the embodiment, whether the binary code of the first processor architecture applicable to the processor architecture exists is determined according to the fields of the binary code, so that the storage of repeated binary codes can be reduced, and the file size can be reduced.

[0022] In a possible implementation, the hash value of the binary code is determined according to fields in the binary code except for the first field and the second field.

[0023] In the embodiment, even if the binary code is reusable, the values of the first field and the second field are different, the hash value of the binary code can be determined according to fields in the binary code except for the first field and the second field, so that the binary code of the first processor architecture applicable to the processor architecture can be determined according to the hash value, and the multi-architecture reuse can be implemented while reducing the file size.

[0024] In a possible implementation, the binary codes of multiple processor architectures are obtained, including:

[0025] The device code of multiple processor architectures is obtained.

[0026] For any one processor architecture, the binary code of the processor architecture is generated according to the device code of the processor architecture.

[0027] In the embodiment, the binary code is generated according to the code device of the processor architecture, so that the binary code can be stored in a heterogeneous file in the subsequent process, and a basis for heterogeneous running is provided.

[0028] In a possible implementation, the method further includes:

[0029] The identification information of the target processor architecture is obtained.

[0030] According to the identification information of the target processor architecture, a first field and a second field of the binary code of the target processor architecture are extracted;

[0031] According to the first field and the second field of the binary code of the target processor architecture, it is determined whether the target processor architecture matches the target device;

[0032] If yes, it is determined that the target device can run the algorithm using the binary code of the target processor architecture.

[0033] In the embodiment, since different processor architectures need to support the same binary code, the identification information of the target processor architecture is relied on to determine whether the binary code of the target processor architecture matches the target device, so that the heterogeneous runtime implements that different processor architectures need to support the same binary code.

[0034] In a second aspect, the application provides a heterogeneous file generation device, comprising:

[0035] An acquisition module is configured to acquire binary codes of multiple processor architectures;

[0036] A storage module is configured to, before writing the binary code of any one processor architecture into a heterogeneous file, determine whether the binary code of a first processor architecture corresponding to the processor architecture exists in the heterogeneous file; if yes, store metadata of the binary code of the processor architecture into a storage area of the heterogeneous file, and the binary code of the processor architecture does not perform a write operation, wherein the metadata comprises identification information of the processor architecture, an offset and a memory size occupied by the binary code;

[0037] The offset and the memory size occupied by the binary code of the processor architecture are the same as the offset and the memory size occupied by the binary code of the first processor architecture, so that the processor architecture reuses the binary code of the first processor architecture; the offset is a length between a start position of the storage area of the heterogeneous file and a start position of an area occupied by the binary code of the processor architecture in the storage area; and the binary code of the first processor architecture and the metadata of the binary code are stored in the heterogeneous file.

[0038] In a possible implementation, the storage module is specifically configured to:

[0039] A first field and a second field of the binary code of the processor architecture are acquired, the first field is used to describe basic information of the binary code, and the second field is used to describe annotation information of the binary code;

[0040] According to the first field and the second field, it is determined whether the binary code of the first processor architecture exists in the heterogeneous file.

[0041] In a possible implementation, the metadata further includes a hash value of the binary code; and the storage module is specifically configured to:

[0042] Obtain a first field and a second field of the binary code of at least one processor architecture that has been stored in the heterogeneous file;

[0043] If there is a processor architecture that meets the condition in the at least one processor architecture, it is determined that the binary code of the first processor architecture exists in the heterogeneous file.

[0044] The condition includes:

[0045] A first value in the first field of the binary code is the same as a first value in the first field of the binary code of the processor architecture, the first value is a value in the first field except for a third value, and the third value is a flag related to a processor architecture.

[0046] A second value in the second field of the binary code is the same as a second value in the second field of the binary code of the processor architecture, the second value is a value in the second field except for a fourth value, and the fourth value indicates a specific processor architecture environment required by the binary code to run.

[0047] A hash value of the binary code is the same as a hash value of the binary code of the processor architecture.

[0048] In a possible implementation, the hash value of the binary code is determined according to fields in the binary code except for the first field and the second field.

[0049] In a possible implementation, the obtaining module is specifically configured to:

[0050] Obtain device code of a plurality of processor architectures;

[0051] For any one processor architecture, according to the device code of the processor architecture, the binary code of the processor architecture is generated.

[0052] In a possible implementation, the apparatus further includes a processing module configured to:

[0053] Obtain identification information of a target processor architecture;

[0054] According to the identification information of the target processor architecture, a first field and a second field of the binary code of the target processor architecture are extracted;

[0055] determining whether the target processor architecture matches the target device according to the first field and the second field of the binary code of the target processor architecture;

[0056] If yes, it is determined that the target device can run the algorithm using the binary code of the target processor architecture.

[0057] The program compiling apparatus provided by the embodiments of the present application has similar advantages to the first aspect, which will not be repeated here.

[0058] In a third aspect, the present application provides an electronic device, comprising a processor and a memory connected with the processor in communication;

[0059] The memory stores computer execution instructions.

[0060] The processor executes the computer execution instructions stored in the memory to implement the method of the first aspect.

[0061] In a fourth aspect, the present application provides a computer readable storage medium, wherein the computer readable storage medium stores computer execution instructions, and the computer execution instructions are used to implement the method of the first aspect when executed by a computer.

[0062] The computer readable storage medium provided by the embodiments of the present application can execute the technical solutions in the above method embodiments, and has similar advantages, which will not be repeated here.

[0063] In a fifth aspect, the present application provides a computer program product, comprising a computer program, and the computer program is used to implement the method of the first aspect when executed by a computer.

[0064] The computer program product provided by the embodiments of the present application can execute the technical solutions in the above method embodiments, and has similar advantages, which will not be repeated here.

[0065] The application provides a heterogeneous file generation method, device, equipment, medium and product. Binary codes of various processor architectures are acquired. Before binary codes of any one processor architecture are written into a heterogeneous file, it is determined whether binary codes of a first processor architecture corresponding to the processor architecture exist in the heterogeneous file. If yes, metadata of the binary codes of the processor architecture is stored in a storage area of the heterogeneous file, and the binary codes of the processor architecture do not perform a write-in operation. The metadata comprises identification information of the processor architecture, an offset and a memory size occupied by the binary codes. The offset and the memory size occupied by the binary codes of the processor architecture are the same as those of the binary codes of the first processor architecture, so that the binary codes of the processor architecture can reuse the binary codes of the first processor architecture. The binary codes of the first processor architecture and the metadata of the binary codes are stored in the heterogeneous file. The application does not store repeated binary codes repeatedly, and metadata storage can realize reuse of multiple processor architectures while reducing file size. BRIEF DESCRIPTION OF DRAWINGS

[0066] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate embodiments consistent with the present application and, together with the description, further serve to explain the principles of the application.

[0067] Figure 1 is a structural diagram of an ELF file and a Fat binary file in related technologies;

[0068] Figure 2 is a structural diagram of an exemplary Fat binary file;

[0069] Figure 3 is a flowchart of a heterogeneous file generation method provided by an embodiment of the application;

[0070] Figure 4 is a structural diagram of a storage area of a heterogeneous file according to an example of the application;

[0071] Figure 5 is a flowchart of another heterogeneous file generation method provided by an embodiment of the application;

[0072] Figure 6 is a structural diagram of a typical Fat binary file provided by an embodiment of the application;

[0073] Figure 7 is a flowchart of still another heterogeneous file generation method provided by an embodiment of the application;

[0074] Figure 8A flowchart of another method for generating a heterogeneous file according to an embodiment of the present application is provided;

[0075] Figure 9 A structural diagram of a device for generating a heterogeneous file according to an embodiment of the present application is provided;

[0076] Figure 10 A structural diagram of an electronic device according to an embodiment of the present application is provided.

[0077] The specific embodiments of the present application have been shown through the above-described drawings, and will be described in more detail hereinafter. The drawings and the written description are not intended to restrict the scope of the present application by any means, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0078] Exemplary embodiments will be described in detail herein with reference to the attached drawings. In the following description, the same numbers are used to indicate the same elements in different drawings. The embodiments described in the following exemplary embodiments are not meant to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present application as detailed in the appended claims.

[0079] The ELF file format is extensible and can accommodate different processor architectures and operating systems. In order to be compatible with heterogeneous programs, a Fat binary file is proposed based on the ELF file, which can accommodate binary codes of multiple target architectures or device types, so that the program can run on different devices without recompilation. This file format provides convenience for implementing program portability across platforms and devices, by storing multiple versions of binary codes in one Fat binary file, so that the code most suitable for the target device can be selected at runtime.

[0080] For the ELF file, all supported processor architecture binary codes are currently embedded in the ELF file at compile time, resulting in a file size that is too large. A large portion of the instruction sets of some processor architectures overlap, and only this portion of the instruction set is commonly used for basic applications. Therefore, for ordinary common files, the binary codes generated under different processor architectures can be theoretically universal across different platforms, and if the binary codes are universal, only one such binary code can be retained, which can avoid excessive file size expansion.

[0081] Figure 1is a structural diagram of an ELF file and a Fat binary file in the related art, and the Fat binary file further includes a structure, which can be a data section and can be referred to as a hip_fatbin section. The hip_fatbin section is used to store binary code of multiple processor structures, and the binary code can also be referred to as device-side code. The device side can be understood as a hardware or device side of a computer architecture that performs a computing task, and is usually used to describe accelerators, GPUs, FPGAs, and the like. The hip_fatbin section can also be understood as a file in the Fat binary file, and the hip_fatbin section can also be referred to as a hip_fatbin file.

[0082] For how to simplify the Fat binary file, the following problems need to be considered:

[0083] 1. How to determine that the binary code generated for different processor architectures can be universal.

[0084] 2. How to save the universal binary code to achieve the effect of reducing the file size.

[0085] 3. Since different processor architectures need to support the same binary code, how to modify the heterogeneous runtime to match the binary code.

[0086] Therefore, the present application provides a heterogeneous file generation method. When writing binary code of each processor architecture into a heterogeneous file, it can be determined whether there is reusable binary code of the processor architecture in the heterogeneous file. If there is, the repeated binary code is not stored repeatedly, and the storage of metadata can achieve the reduction of the file size while achieving the reuse of multiple processor architectures.

[0087] In the embodiments of the present application, the heterogeneous file can be a Fat binary file.

[0088] The heterogeneous file generation method of the embodiments of the present application can be applied to a heterogeneous system integrated with multiple processor architectures such as GPUs, FPGAs, or DSPs.

[0089] For ease of understanding, before the embodiments of the present application are described, the file structure of the current Fat binary file is first described.

[0090] The file structure of the Fat binary file can refer to Figure 2 , and Table 1 to Table 3. Among them, Figure 2 is a schematic diagram of the file structure of the exemplary Fat binary file, and Table 1 to Table 3 are definitions of the structure of the hip_fatbin section in the exemplary Fat binary file. In order to facilitate understanding, the definition of the structure of the hip_fatbin section is described in three tables.

[0091] In Table 1 to Table 3, the field in Table 1 is used to describe the format of the hip_fatbin file and the number of binary code packages of the processor architecture stored therein, the field in Table 2 is used to store the metadata of N processor architecture binary codes, and the field in Table 3 is used to store N processor architecture binary codes.

[0092] Table 1

[0093]

[0094] Table 2

[0095]

[0096]

[0097] Table 3

[0098]

[0099] In the embodiments of the present application, the starting position of the region occupied by the metadata of the binary code of the first bundled item can be used as the starting position of the storage area of the heterogeneous file.

[0100] The technical solutions of the present application and how the technical solutions of the present application solve the above technical problems will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes can not be described in detail in some embodiments. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0101] Figure 3 A flowchart of a heterogeneous file generation method provided in the embodiments of the present application is shown in the figure. The method can be executed by a heterogeneous file generation device. The device can be realized by a computer program, or by a medium storing a related computer program, such as a U disk and / or an optical disk, or by an entity device integrated or installed with a related computer program, such as a chip or an electronic device. The electronic device can be a server, a server cluster, a computer terminal device, etc. The following will be described taking a server as an example, which can be executed by a compiler in the server, such as Figure 3As shown, the method can include the following steps:

[0102] S301, obtaining binary codes of multiple processor architectures.

[0103] In a possible implementation, the server can obtain device codes of multiple processor architectures, and for any one processor architecture, the server can generate a binary code of the processor architecture according to the device code of the processor device.

[0104] For example, for any one processor architecture, the server can determine a suitable compilation tool for the processor architecture, and then compile the device code through the compilation tool to obtain the binary code of the processor architecture, so that the binary code can be stored in a heterogeneous file to provide a basis for heterogeneous running.

[0105] S302, before writing the binary code of any one processor architecture into the heterogeneous file, determining whether there is a first processor architecture corresponding to the processor architecture; if yes, storing metadata of the binary code of the processor architecture in a storage area of the heterogeneous file, and the binary code of the processor architecture does not perform a write operation.

[0106] The metadata includes identification information of the processor architecture, an offset, and a memory size occupied by the binary code.

[0107] The offset and the memory size occupied by the binary code of the processor architecture are the same as the offset and the memory size occupied by the binary code of the first processor architecture, so that the processor architecture reuses the binary code of the first processor architecture; the offset is the length between the start position of the storage area of the heterogeneous file and the start position of the area occupied by the binary code of the processor architecture in the storage area; the binary code of the first processor architecture and the metadata of the binary code are stored in the heterogeneous file.

[0108] That is, the first processor architecture is a processor architecture whose binary code has been stored in the heterogeneous file, and before storing the binary code of any one processor architecture, it can be determined from the binary code already stored in the heterogeneous file whether there is reusable binary code.

[0109] The identification information includes a memory size occupied by an ID of the processor architecture and a pointer of the ID, and through the pointer and the memory size, the ID of the processor architecture can be determined.

[0110] For example, the IDs of the plurality of processor architectures can be stored in a file, and the server can determine the starting position of an ID in the file according to a pointer of the ID, and then determine the ending position of the ID according to the memory size occupied by the ID, so as to obtain the ID, where the ID can be a string. For example, the pointer of the ID can be the starting symbol of the ID, for example, the ID is abcde001, and the pointer of the ID can be a.

[0111] It can be understood that if there is no first processor architecture corresponding to the processor architecture, the server can store the metadata and the binary code of the binary code of the processor architecture into the heterogeneous file.

[0112] In the embodiments of the present application, the first processor architecture corresponding to the processor architecture can be understood as that the binary code of the processor architecture is the same as the binary code of the first processor architecture, and the two processor architectures can reuse the same binary code.

[0113] Specifically, before the server writes the binary code of any processor architecture into the heterogeneous file, the server can determine whether the binary code of the first processor architecture that is the same as the binary code of the processor architecture exists in the heterogeneous file, if it exists, the server can not store the binary code of the processor architecture, and determine the metadata of the binary code of the processor architecture according to the metadata of the binary code of the first processor architecture, and store the metadata of the binary code of the processor architecture into the storage area of the heterogeneous file, so as to save the universal binary code, and achieve the effect of reducing the file size.

[0114] For example, the server can determine the offset and the memory size occupied by the binary code of the first processor architecture in the metadata of the binary code of the first processor architecture as the offset and the memory size occupied by the binary code of the processor architecture in the metadata of the binary code of the processor architecture, that is, the offset and the memory size occupied by the binary code of the processor architecture are the same as the offset and the memory size occupied by the binary code of the first processor architecture, so that the binary code of the processor architecture reuses the binary code of the first processor architecture, that is, when searching for the binary code of the processor architecture, the binary code of the first processor architecture can be found, so as to realize reuse.

[0115] Figure 4 For the above-mentioned offset, the storage area of the heterogeneous file of the embodiments of the present application is shown in the schematic diagram, for the convenience of understanding, please refer to Figure 4 If the first processor architecture is processor architecture 3, the offset of the binary code of the processor architecture 3 and the memory size occupied by the binary code are as follows Figure 4The offset is the length between the start position of the storage area of the heterogeneous file and the start position of the area occupied by the binary code of the processor architecture 3. It can be understood that, for the offset of the binary code of the processor architecture 1, the size of the offset is the length of the area occupied by the metadata of the binary code of all processor architectures, which may, for example, be a byte length in the embodiment of the present application. The server can obtain the binary code of the corresponding processor architecture in the storage area of the heterogeneous file according to the offset and the occupied memory size. For example, the server can determine the start position of the area occupied by the binary code of the processor architecture 3 in the storage area of the heterogeneous file according to the offset, and obtain the complete binary code of the processor architecture 3 according to the start position of the area occupied and the occupied memory size. Figure 4

[0116] In the embodiment, before writing the binary code of any processor architecture into the heterogeneous file, it is determined whether the binary code of the first processor architecture corresponding to the processor architecture exists in the heterogeneous file. If yes, the metadata of the binary code of the processor architecture is stored into the storage area of the heterogeneous file, and the binary code of the processor architecture does not perform a writing operation. The offset of the binary code of the processor architecture and the occupied memory size are the same as those of the binary code of the first processor architecture, so that the first processor architecture can reuse the binary code of the first processor architecture, and the binary code of the first processor architecture and the metadata of the binary code are stored in the heterogeneous file. The present application does not store the repeated binary code repeatedly, and the metadata can be stored to reduce the file size and reuse multiple processor architectures.

[0117] Next, how to determine whether the binary code of the first processor architecture corresponding to the processor architecture exists in the heterogeneous file, that is, how to determine whether the binary code generated for different processor architectures can be universal, is described.

[0118] For ease of understanding, the structure of the binary code of the processor architecture is first described. Table 4 is the structure of the binary code of the processor architecture in the example of the present application. Each stored binary code itself is a complete ELF file.

[0119] Table 4

[0120] Elf header …… Section note …… Section header table

[0121] In Table 4, only the header field, the note field and the section header table field of the binary code are shown, and other fields (such as the code field and the data field) are not shown. The header field (elf header) is used to describe the basic attributes of the binary code. ​

[0122] For example, the header field (elf header) can include an e_flags value, a flag bit related to a processor architecture, which is a bit mask encoding a specific processor architecture and a runtime feature configuration; the header field can also include other values, such as an e_ident value (used to identify an ELF file storing the binary code), an e_type value (used to identify a file type, such as a relocation file or an executable file, etc.), an e_machine value (used to specify an instruction set architecture to which the file is adapted), and the like.

[0123] The note field (section.note) is used to describe the annotation information of the binary code, and the annotation information can include, for example, a specific processor architecture environment required for running the binary code, a compiler version generating the binary code, an instruction set version supported by the binary code, copyright information, and the like.

[0124] Specifically, the note field can include an amdhsa.target value for describing a specific processor architecture environment required for running the binary code, and the note field can also include other fields, which are not described here.

[0125] The section header table field is used to describe the metadata of other fields not shown in Table 4 in the binary code, such as a section type, a section name, an offset, and a byte size occupied, and the section name can be, for example, a name of a code field, a name of a data field, and the like, and the section type is a code field, a data field, and the like.

[0126] Figure 5 Another flowchart of a method for generating a heterogeneous file provided by an embodiment of the present application is shown, which can be executed by a heterogeneous file generation device. The device can be implemented by a computer program, or can be implemented by a medium storing a related computer program, such as a U disk and / or an optical disk, or can also be implemented by an entity device integrated or installed with a related computer program, such as a chip or an electronic device. The electronic device can be a server, a server cluster, a computer terminal device, and the like. The following takes a server as an example for description, which can be executed by a compiler in the server, as shown in the figure, and the method can include the following steps: Figure 5

[0127] S501, obtaining a first field and a second field of a binary code of a processor architecture, the first field being used to describe metadata of the binary code, and the second field being used to describe annotation information of the binary code.

[0128] ​For any one of the multiple processor architectures, the server can obtain the first and second fields of the binary code of that processor architecture. The first field is used to describe the basic information of the binary code, and the second field is used to describe the annotation information of the binary code. For example, as shown in Table 4, the first field can be a header field (elf header), and the second field can be a note field (section.note).

[0129] S502. Based on the first field and the second field, determine whether binary code of the first processor architecture exists in the heterogeneous file.

[0130] The server can determine whether binary code for the first processor architecture exists in the heterogeneous file based on the first and second fields.

[0131] In one possible implementation, the metadata of the processor architecture's binary code also includes a hash value of the binary code, which the server can use to determine whether the binary code of the first processor architecture exists in the heterogeneous file, based on the hash value, the first field, and the second field.

[0132] For example, after adding the hash value, the structure of the Fat binary file is as follows: Figure 6 As shown, by Figure 6 It can be seen that the storage structure of the binary code's metadata has added a hash value field to store the hash value of the binary code.

[0133] Specifically, the server can obtain the first and second fields of the binary code of at least one processor architecture stored in the heterogeneous file. If there is a processor architecture that meets the conditions among the at least one processor architecture, it is determined that there is binary code of the first processor architecture in the heterogeneous file.

[0134] This condition includes the following three items:

[0135] The first value in the first field of the binary code is the same as the first value in the first field of the processor architecture's binary code. The first value is the value in the first field other than the third value. The third value is a flag bit related to the processor architecture.

[0136] The second value in the second field of the binary code is the same as the second value in the second field of the processor architecture's binary code. The second value is any value in the second field except for the fourth value, which is the processor architecture to which the comment information applies.

[0137] The hash value of the binary code is the same as the hash value of the binary code of the processor architecture.

[0138] The third value may be, for example, an e_flags value, and the fourth value may be, for example, an amdhas.target value. Taking processor architecture 1 as an example, the server may compare the first value in the first field of the binary code of processor architecture 1 with the first value in the first field of the binary code of each processor architecture stored in the heterogeneous file, compare the second value in the second field of the binary code of processor architecture 1 with the second value in the second field of the binary code of each processor architecture stored in the heterogeneous file, and compare the hash value of the binary code of processor architecture 1 with the hash value of the binary code of each processor architecture stored in the heterogeneous file, and find the binary code in the heterogeneous file that is identical to the three values of the binary code of processor architecture 1, and if such binary code exists, it may be determined that the binary code is the binary code of the first processor architecture.

[0139] By using the fields of the binary code, it is determined whether there is the binary code of the first processor architecture that is applicable to the processor architecture, thereby reducing the storage of repeated binary codes and reducing the file size.

[0140] In a possible implementation, based on the content indicated by the first field and the second field, even if the binary code is reusable, the values of the first field and the second field are different, the server may determine the hash value according to the fields other than the first field and the second field in the binary code, thereby determining whether there is the binary code of the first processor architecture that is applicable to the processor architecture according to the hash value, and achieving the reuse of multiple architectures while reducing the file size.

[0141] For example, the fields other than the first field and the second field in the binary code may include code fields and data fields, and the server may calculate the hash value of the binary code according to these fields.

[0142] In Figure 5 In the corresponding embodiment, by using the fields of the binary code, it is determined whether there is the binary code of the first processor architecture that is applicable to the processor architecture, thereby reducing the storage of repeated binary codes and reducing the file size.

[0143] In the related art, since the binary codes of various processor architectures are stored in the heterogeneous file, in the heterogeneous runtime, only the e_flags value and the amdhsa.target value in the system running parameter of the heterogeneous system are needed to judge whether the corresponding processor architecture matches the current device, but since the same binary code is reused for part of the processor architectures in the present application, the corresponding e_flags value and the amdhsa.target value need to be extracted based on the identification information, so as to judge whether the corresponding processor architecture matches the current device, to ensure the correct running of the program.

[0144] Next, in the heterogeneous runtime, how to match the binary code of the target processor architecture with the target device to support multiple processor architectures using the same binary code is described.

[0145] Figure 7 Another flowchart of a method for generating a heterogeneous file provided by the embodiments of the present application is provided. The method can be executed by a device for generating a heterogeneous file. The device can be implemented by a computer program. The device can also be implemented by a medium, such as a U disk and / or an optical disk, which stores a related computer program. Alternatively, the device can also be implemented by an entity device, such as a chip or an electronic device, which integrates or installs a related computer program. The electronic device can be a server, a server cluster, a computer terminal device, and the like. The following takes a server as an example for description. The method can be executed by a compiler in the server, as shown in the following figure, and can include the following steps: Figure 7

[0146] S701, obtaining identification information of a target processor architecture.

[0147] For example, a heterogeneous system is integrated in the server. For example, a central processing unit (CPU) in the server can be used to control and coordinate tasks, so that a program can run on different devices (such as GPU, FPGA, etc.).

[0148] The server can obtain the identification information of the target processor architecture. For example, the identification information can include a pointer of the ID of the target processor architecture and the memory size occupied by the ID. The server can determine the ID of the target processor architecture according to the identification information.

[0149] S702, extracting a first field and a second field of the binary code of the target processor architecture according to the identification information of the target processor architecture.

[0150] The server can obtain the metadata of the binary code of the target processor architecture in the region occupied by the metadata of the heterogeneous file according to the ID of the target processor architecture. For example, Figure 6 ​As shown, the server can extract the first and second fields from the metadata. Specifically, the server can extract the third value (i.e., the e_flags value) from the first field and the fourth value (amdhsa.target value) from the second field.

[0151] S703. Determine whether the target processor architecture matches the target device based on the first and second fields of the binary code of the target processor architecture.

[0152] After extracting the third value (e_flags value) from the first field and the fourth value (amdhsa.target value) from the second field, the server can determine whether the binary code of the target processor architecture matches the target device based on the third value of the first field and the fourth value of the second field. The target device could be, for example, a GPU or FPGA.

[0153] If so, then execute S704.

[0154] S704. Determine that the target device can use binary code of the target processor architecture to run the algorithm.

[0155] If the binary code of the target processor architecture matches the target device, then the server can run the algorithm using the binary code of the target processor architecture through the target device.

[0156] In this embodiment, since different processor architectures need to support the same binary code, the identification information of the target processor architecture is used to determine whether the binary code of the target processor architecture matches the target device, thereby enabling heterogeneous runtime to achieve the requirement that different processor architectures support the same binary code.

[0157] Figure 8 This is a flowchart illustrating another method for generating heterogeneous files provided in this application. The method can be executed by a heterogeneous file generation device, which can be implemented using a computer program; it can also be implemented using a medium storing the relevant computer program, such as a USB flash drive and / or optical disc; or it can be implemented using a physical device integrating or installing the relevant computer program, such as a chip or electronic device. The electronic device can be a server, server cluster, or computer, etc. The following description uses a server as an example. Specifically, the method can be executed by a compiler within the server. Figure 8 As shown, the method may include the following steps:

[0158] S801: Obtain binary code for various processor architectures.

[0159] S802, obtaining a first field and a second field of the binary code of the processor architecture before writing the binary code of any one processor architecture into the heterogeneous file, the first field being used for describing metadata of the binary code, and the second field being used for describing annotation information of the binary code.

[0160] S803, determining whether the binary code of the first processor architecture exists in the heterogeneous file according to the first field and the second field.

[0161] If yes, S804 is executed.

[0162] S804, storing the metadata of the binary code of the processor architecture into a storage area of the heterogeneous file, the binary code of the processor architecture not performing the writing operation.

[0163] S805, obtaining identification information of a target processor architecture.

[0164] S806, extracting the first field and the second field of the binary code of the target processor architecture according to the identification information of the target processor architecture.

[0165] S807, determining whether the target processor architecture matches the target device according to the first field and the second field of the binary code of the target processor architecture.

[0166] If yes, S808 is executed.

[0167] S808, determining that the target device can run the algorithm using the binary code of the target processor architecture.

[0168] The specific implementation manners and technical effects of the embodiment are similar to those of the above-described embodiment, and thus are not described herein.

[0169] Based on the above-described embodiments, the embodiments of the present application have the following beneficial effects:

[0170] 1. For the binary code of the universal processor architecture, the file size is greatly reduced.

[0171] 2. The organization structure of the file is not damaged, the correctness and integrity of the program are ensured, and the program can be correctly run.

[0172] 3. The heterogeneous runtime is implemented, and different processor architectures support the same binary code.

[0173] Figure 9 A structure diagram of a heterogeneous file generation device provided by the embodiment of the present application is shown in FIG. 9. Figure 9 As shown in FIG. 9, the device 90 includes an obtaining module 901 and a storage module 902.

[0174] The acquisition module 901 is configured to acquire binary codes of multiple processor architectures.

[0175] The storage module 902 is configured to, before writing the binary code of any one processor architecture into the heterogeneous file, determine whether the binary code of the first processor architecture corresponding to the processor architecture exists in the heterogeneous file; if yes, store metadata of the binary code of the processor architecture into a storage area of the heterogeneous file, and the binary code of the processor architecture does not perform a writing operation, and the metadata includes identification information of the processor architecture, an offset, and a memory size occupied by the binary code.

[0176] The offset and the memory size occupied by the binary code of the processor architecture are the same as the offset and the memory size occupied by the binary code of the first processor architecture, so that the processor architecture reuses the binary code of the first processor architecture; the offset is a length between a start position of the storage area of the heterogeneous file and a start position of an area occupied by the binary code of the processor architecture in the storage area; and the binary code of the first processor architecture and the metadata of the binary code are stored in the heterogeneous file.

[0177] In a possible implementation, the storage module 902 is specifically configured to:

[0178] acquire a first field and a second field of the binary code of the processor architecture, the first field being used to describe basic information of the binary code, and the second field being used to describe annotation information of the binary code;

[0179] determine, according to the first field and the second field, whether the binary code of the first processor architecture exists in the heterogeneous file.

[0180] In a possible implementation, the metadata further includes a hash value of the binary code; and the storage module 902 is specifically configured to:

[0181] acquire the first field and the second field of the binary code of at least one processor architecture that has been stored in the heterogeneous file.

[0182] if there is a processor architecture that meets a condition in the at least one processor architecture, it is determined that the binary code of the first processor architecture exists in the heterogeneous file.

[0183] The condition includes:

[0184] a first value in the first field of the binary code is the same as a first value in the first field of the binary code of the processor architecture, the first value being a value in the first field except for a third value, and the third value being a flag bit related to the processor architecture.

[0185] The second value in the second field of the binary code is the same as the second value in the second field of the binary code of the processor architecture, the second value is a value in the second field other than the fourth value, and the fourth value indicates a specific processor architecture environment required by the binary code to run.

[0186] The hash value of the binary code is the same as the hash value of the binary code of the processor architecture.

[0187] In a possible implementation, the hash value of the binary code is determined according to fields in the binary code other than the first field and the second field.

[0188] In a possible implementation, the obtaining module 901 is specifically configured to:

[0189] Obtain the device code of a plurality of processor architectures.

[0190] For any one processor architecture, generate the binary code of the processor architecture according to the device code of the processor architecture.

[0191] In a possible implementation, the apparatus 90 further includes a processing module 903 configured to:

[0192] Obtain the identification information of the target processor architecture.

[0193] According to the identification information of the target processor architecture, extract the first field and the second field of the binary code of the target processor architecture.

[0194] According to the first field and the second field of the binary code of the target processor architecture, determine whether the target processor architecture matches the target device.

[0195] If yes, it is determined that the target device can run the algorithm using the binary code of the target processor architecture.

[0196] The apparatus of the embodiment can be used to execute the technical solutions of the method embodiments, and the specific implementation manners and technical effects are similar, which will not be described here.

[0197] Figure 10 A structural schematic diagram of an electronic device provided in the embodiment of the present application is shown in FIG. 1. Figure 10 As shown in FIG. 1, the electronic device 100 can include at least one processor 1001 and a memory 1002.

[0198] The memory 1002 is configured to store a program. Specifically, the program can include program code, and the program code includes computer execution instructions.

[0199] The memory 1002 can include a random access memory (RAM) and can also include a non-volatile memory, such as at least one disk memory.

[0200] The processor 1001 is configured to execute the computer-executable instructions stored in the memory 1002, so as to implement the method described in the foregoing method embodiments. The processor 1001 can be a CPU, or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0201] Optionally, the electronic device 100 can further include a communication interface 1003. In a specific implementation, if the communication interface 1003, the memory 1002 and the processor 1001 are independently implemented, the communication interface 1003, the memory 1002 and the processor 1001 can be connected with each other through a bus and complete communication between each other. The bus can be an industry standard architecture (ISA) bus, a peripheral component (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc., but it does not mean that there is only one bus or one type of bus.

[0202] Optionally, in a specific implementation, if the communication interface 1003, the memory 1002 and the processor 1001 are integrated on a chip, the communication interface 1003, the memory 1002 and the processor 1001 can complete communication through an internal interface.

[0203] The electronic device 100 can be a server, a server cluster, a computer terminal device and the like.

[0204] The electronic device of the embodiment can be used to execute the technical solutions of the foregoing method embodiments, and the specific implementation manners and technical effects are similar, which will not be described herein.

[0205] The embodiment of the present application provides a computer readable storage medium, which can include: a U disk, a mobile hard disk, a read-only memory (ROM), a RAM, a magnetic disk or an optical disk and various media that can store computer execution instructions, specifically, the computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by a computer to execute the technical solutions shown in the method embodiment, and specific implementation manners and technical effects are similar, and details are not described herein.

[0206] The embodiment of the present application provides a computer program product, including a computer program, and the computer program is executed by a computer to execute the technical solutions shown in the method embodiment, and specific implementation manners and technical effects are similar, and details are not described herein.

[0207] It should be noted that, for the foregoing method embodiments, in order to simply describe, the method embodiments are described as a series of action combinations, but those skilled in the art should know that the present application is not limited to the action sequence described, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0208] It should be further noted that, although each step in the flowchart is displayed in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless otherwise stated in this paper, the execution of these steps has no strict sequence limitation, and these steps can be executed in other sequences. Moreover, at least part of the steps in the flowchart can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these sub-steps or stages is not necessarily sequential, but can be executed with at least part of other steps or other steps or stages of sub-steps or stages, or alternately.

[0209] It should be understood that the above-mentioned device embodiment is only schematic, and the device of the present application can also be realized by other ways. For example, the division of units / modules in the above-mentioned embodiment is only a logical function division, and another division mode can be used in actual implementation. For example, a plurality of units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.

[0210] In addition, each functional unit / module in the various embodiments of the present application can be integrated in one unit / module, or each unit / module can exist physically, or two or more units / modules can be integrated together. The integrated unit / module can be implemented in the form of hardware or in the form of a software program module.

[0211] If the integrated unit / module is implemented in the form of hardware, the hardware can be a digital circuit, an analog circuit, etc. The physical implementation of the hardware structure includes but is not limited to transistors, memristors, etc. Unless otherwise specified, the processor can be any suitable hardware processor, such as a CPU, a graphics processing unit (GPU), a field programmable gate array (FPGA), a digital signal processing (DSP) chip, an application-specific integrated circuit (ASIC), etc. Unless otherwise specified, the storage unit can be any suitable magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory (RRAM), a dynamic random access memory (DRAM), a static random access memory (SRAM), an enhanced dynamic random access memory (EDRAM), a high-bandwidth memory (HBM), a hybrid memory cube (HMC), etc.

[0212] If the integrated unit / module is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application, essentially or the part that makes a contribution to the prior art, or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a ROM, a RAM, a mobile hard disk, a magnetic disk or an optical disk, and various other media that can store program codes.

[0213] In the above embodiments, the description of each of the embodiments focuses on different aspects of the embodiments. The parts not described in detail in a certain embodiment can be seen in the relevant description of the other embodiments. The technical features of the above embodiments can be combined in any manner. In order to make the description simple, not all possible combinations of the technical features in the above embodiments are described, however, as long as the combinations of the technical features do not exist contradictory, they shall be considered as falling within the scope of the disclosure.

[0214] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the specification and examples be considered as exemplary only, with the true scope and spirit of the application being indicated by the following claims.

[0215] It should be understood that the application is not limited to the precise construction and combinations of the components and steps described above and shown in the accompanying drawings. The scope of the application is limited only by the claims that follow.

Claims

1. A method for generating heterogeneous files, characterized in that, include: Obtain binary code for multiple processor architectures; Before writing the binary code of any processor architecture into the heterogeneous file, it is determined whether the binary code of the first processor architecture corresponding to the processor architecture exists in the heterogeneous file; If so, the metadata of the binary code of the processor architecture is stored in the storage area of ​​the heterogeneous file. The binary code of the processor architecture is not written. The metadata includes the identification information of the processor architecture, the offset, and the memory size occupied by the binary code. Wherein, the offset of the binary code of the processor architecture and the memory size occupied are the same as the offset of the binary code of the first processor architecture, so that the processor architecture reuses the binary code of the first processor architecture; the offset is the length between the starting position of the storage area of ​​the heterogeneous file and the starting position of the area occupied by the binary code of the processor architecture in the storage area. The binary code of the first processor architecture and the metadata of the binary code are stored in the heterogeneous file.

2. The method according to claim 1, characterized in that, Determining whether the heterogeneous file contains binary code for the first processor architecture corresponding to the processor architecture includes: Obtain the first field and the second field of the binary code of the processor architecture. The first field is used to describe the basic information of the binary code, and the second field is used to describe the comment information of the binary code. Based on the first field and the second field, determine whether binary code of the first processor architecture exists in the heterogeneous file.

3. The method according to claim 2, characterized in that, The metadata also includes the hash value of the binary code; Determining whether binary code for the first processor architecture exists in the heterogeneous file based on the first field and the second field includes: Obtain the first and second fields of the binary code for at least one processor architecture stored in the heterogeneous file; If a processor architecture that meets the conditions exists in the at least one processor architecture, then it is determined that the binary code of the first processor architecture exists in the heterogeneous file; The conditions include: The first value in the first field of the binary code is the same as the first value in the first field of the binary code of the processor architecture. The first value is a value in the first field other than the third value, and the third value is a flag bit related to the processor architecture. The second value in the second field of the binary code is the same as the second value in the second field of the binary code of the processor architecture. The second value is a value in the second field other than the fourth value, which indicates the specific processor architecture environment that the binary code requires to run. The hash value of the binary code is the same as the hash value of the binary code of the processor architecture.

4. The method according to claim 2 or 3, characterized in that, The hash value of the binary code is determined based on the fields in the binary code other than the first and second fields.

5. The method according to any one of claims 1-4, characterized in that, Obtain binary code for various processor architectures, including: Obtain device codes for multiple processor architectures; For any given processor architecture, generate the binary code for that processor architecture based on the device code of that processor architecture.

6. The method according to any one of claims 1-5, characterized in that, The method further includes: Obtain the identification information of the target processor architecture; Based on the identification information of the target processor architecture, extract the first and second fields of the binary code of the target processor architecture; Based on the first and second fields of the binary code of the target processor architecture, determine whether the target processor architecture matches the target device; If so, it is determined that the target device can use the binary code of the target processor architecture to run the algorithm.

7. A device for generating heterogeneous files, characterized in that, include: The acquisition module is used to acquire binary code for various processor architectures; The storage module is used to determine whether binary code of a first processor architecture corresponding to the processor architecture exists in the heterogeneous file before writing binary code of any processor architecture into the heterogeneous file; If so, the metadata of the binary code of the processor architecture is stored in the storage area of ​​the heterogeneous file. The binary code of the processor architecture is not written. The metadata includes the identification information of the processor architecture, the offset, and the memory size occupied by the binary code. Wherein, the offset of the binary code of the processor architecture and the memory size occupied are the same as the offset of the binary code of the first processor architecture, so that the processor architecture reuses the binary code of the first processor architecture; the offset is the length between the starting position of the storage area of ​​the heterogeneous file and the starting position of the area occupied by the binary code of the processor architecture in the storage area. The binary code of the first processor architecture and the metadata of the binary code are stored in the heterogeneous file.

8. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 1 to 6.

Images