Distributed machine learning method and system based on gradient compression and quantization

Abstract

The invention discloses a distributed machine learning method and system based on gradient compression and quantization, and relates to the technical field of communication resource calculation. The method comprises the following steps: a server performs data compression processing on gradient information of a client, the gradient information of the client is preliminarily compressed by using discretization quantization processing, and the preliminarily compressed information is further compressed by using a coding method; distributing the compressed gradient information to a client; the client performs self-adaptive calculation and selection on the gradient threshold by using an edge calculation method to obtain updated gradient information; and the client uploads the updated information to the server. According to the method, when the server issues the gradient information, the gradient distributed by the server is compressed by adopting a discrete quantization and coding combined method, and then the data is further compressed by adopting a coding method on the basis of discrete quantization, so that efficient compression of the gradient is realized.

Description

Technical Field

[0001] The present invention relates to the field of communication resource computing technology, and in particular to a distributed machine learning method and system based on gradient compression and quantization. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] With the rapid development of the Internet and IoT technologies, the amount of data is growing exponentially. Traditional centralized machine learning methods are not only computationally expensive when processing such large-scale data, but also have hidden dangers in terms of data privacy and security. Distributed machine learning can effectively alleviate these problems by distributing data and computing tasks across multiple computing nodes. It allows data preprocessing and preliminary learning on local nodes, and then aggregates the learning results to the central node for integration, thereby improving learning efficiency and reducing the burden on the central server.

[0004] Although distributed machine learning has many advantages in theory, it faces significant challenges in practical applications, especially in environments with limited communication resources. Limited communication resources mean that communication bandwidth and latency limitations must be considered when transmitting data and model updates between nodes. In such environments, traditional distributed machine learning algorithms often result in huge communication overhead, thereby reducing overall learning efficiency. In distributed machine learning, communication overhead is mainly reflected in the process of transmitting model parameters, gradient information, update information, etc. between nodes. Due to the huge amount of data, the transmission of this information often consumes a lot of communication resources. Especially in wireless network environments, where communication bandwidth is limited and latency is high, the communication overhead problem is particularly prominent. Therefore, how to reduce communication overhead and improve communication efficiency has become an urgent problem to be solved in the field of distributed machine learning.

[0005] In recent years, some progress has been made in the study of the problem of limited communication resources. For example, the FedBCD algorithm reuses the statistics of the previous round for multiple iterations to reduce the number of communications. The Flex-VFL algorithm sets a threshold for the interval between communication rounds, allowing each party to perform different numbers of local iterations. However, these algorithms usually have certain requirements for the overall training architecture and are not universal in training scenarios. The Sparse Ternary Compression (STC) framework extends the existing top-k gradient dilution compression technology and introduces a new mechanism that enables downstream compression, normalization, and optimal Golomb encoding of weight updates. In addition, a new distributed quantized gradient method has been proposed, which has the characteristics of adaptive communication of quantized gradients, but this mechanism is difficult to determine the gradient of a constant threshold in practice. Some people have also proposed the Lazily Aggregated Gradient (LAG) algorithm, which can adaptively calculate the gradient and skip part of the gradient communication, thereby reducing the communication bandwidth and alleviating the pressure on the server. The basic principle is to detect slowly changing gradients and compress them, but LAG cannot be used to optimize convex problems.

[0006] In summary, how to overcome the defects of poor practicality, low compression efficiency and unstable gradient threshold in the gradient compression process has become an urgent problem to be solved in the prior art. Summary of the invention

[0007] In view of the shortcomings of the prior art, the purpose of the present invention is to provide a distributed machine learning method and system based on gradient compression and quantization. When the server sends gradient information, a method combining discrete quantization and encoding is used to compress the gradient distributed by the server. Then, on the basis of discrete quantization, an encoding method is used to further compress the data, thereby achieving efficient compression of the gradient.

[0008] In order to achieve the above object, the present invention is implemented through the following technical solutions: A first aspect of the present invention provides a distributed machine learning method based on gradient compression and quantization, comprising the following steps: The server obtains the client's gradient information and caches it; Performing data compression processing on the client gradient information, wherein the client gradient information is preliminarily compressed by using a discretization quantization process, and the preliminarily compressed information is further compressed by using an encoding method; Distribute the compressed gradient information to the client; After receiving the compressed gradient information, the client uses the edge computing method to adaptively calculate and select the gradient threshold to obtain the updated gradient information; The client uploads the updated information to the server.

[0009] Furthermore, the specific steps for the server to obtain the client gradient information and cache it are: Before distributing the gradient information to the client, first count the distribution of the gradient information of the corresponding client in the previous round in the local cache, and calculate the mean and variance to obtain the threshold range to be quantized.

[0010] Furthermore, the specific steps of performing data compression processing on the client gradient information include: Iterate the gradient information and quantize the gradient information before each iteration to achieve data compression; Huffman coding is used to further compress the amount of information of the gradient information after quantization to obtain the optimal variable-length code.

[0011] Furthermore, the specific steps for quantizing the gradient information are as follows: Obtain a threshold interval to be quantized, and divide the threshold interval into P sub-intervals on average to form P+1 endpoint values; Approximate each element in the gradient information to the nearest endpoint.

[0012] Furthermore, after each iteration of the gradient information, the server will re-record and count the original uncompressed gradient distribution of each client to achieve real-time dynamic update of the quantization interval and optimize the compression effect.

[0013] Furthermore, when the compressed gradient information is distributed to the client, the encoding values ​​of all elements in the gradient information are bit-joined to form a bit string, and the bit string and the gradient mapping code table are distributed to the corresponding client.

[0014] Furthermore, the specific steps of using the edge computing method to adaptively calculate and select the gradient threshold are as follows: The edge server first sends the current global model to all subordinate device nodes; Each device node then calculates the local parameter gradient and uploads it to the edge, where the edge server obtains the gradient sum through weighted summation; The trust threshold is set through an adaptive algorithm, and the trust threshold is used to evaluate the trust of all gradients to obtain a set of trustworthy communication nodes.

[0015] A second aspect of the present invention provides a distributed machine learning system based on gradient compression and quantization, comprising: An information acquisition module is configured to acquire client gradient information and cache it on the server; An information compression module is configured to perform data compression processing on the client gradient information, wherein the client gradient information is preliminarily compressed by using a discretization quantization process, and the preliminarily compressed information is further compressed by using an encoding method; An information distribution module is configured to distribute the compressed gradient information to the client; The local processing module is configured to adaptively calculate and select the gradient threshold using the edge computing method after the client receives the compressed gradient information, so as to obtain updated gradient information; The information feedback module is configured so that the client uploads the updated information to the server.

[0016] The third aspect of the present invention provides a medium on which a program is stored. When the program is executed by a processor, the steps in the distributed machine learning method based on gradient compression and quantization as described in the first aspect of the present invention are implemented.

[0017] The fourth aspect of the present invention provides a device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the distributed machine learning method based on gradient compression and quantization as described in the first aspect of the present invention are implemented.

[0018] One or more of the above technical solutions have the following beneficial effects: The present application discloses a distributed machine learning method and system based on gradient compression and quantization. When the server sends down the gradient information, the gradient distributed by the server is compressed by combining discrete quantization and coding. By discretizing the continuous gradient value into a limited number of levels, the amount of gradient data can be significantly reduced. This can reduce the space and time required for storing and transmitting the gradient, thereby reducing the communication burden of the server. After that, based on the discrete quantization, the data can be further compressed by using the coding method. After discrete quantization and coding processing, when distributing the gradient value, the server only needs to splice the coded values ​​of all elements in the gradient bit by bit to form a bit string, and transmit the string and the gradient mapping code table, thereby achieving efficient compression of the gradient.

[0019] This application adopts Huffman coding, which may perform optimal variable-length coding on the gradient information after discrete processing according to its frequency of occurrence. This coding method greatly shortens the overall coding length, thereby improving the efficiency of communication. At the same time, it is a lossless compression algorithm that can maintain the integrity of the data while compressing the data. That is, during the decoding process, the original gradient data can be fully restored without introducing any errors or distortions, which improves a certain fault tolerance rate.

[0020] In order to reduce the amount of data distributed by the server, the present application adopts an adaptive threshold gradient compression method when uploading gradient information to the server after the client has trained the model. This method can adaptively select which gradient information needs to be transmitted and which can be discarded or delayed for transmission, thereby reducing the amount of data that needs to be transmitted, ensuring the transmission of key gradient information while reducing the transmission of redundant information, thereby maintaining the convergence speed and accuracy of the model.

[0021] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0023] Figure 1 This is a flow chart of the distributed machine learning method based on gradient compression and quantization in the present invention. DETAILED DESCRIPTION

[0024] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0025] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "include" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or their combinations; Embodiment 1: At present, distributed machine learning has the following difficulties when compressing data for the problem of limited communication resources: 1. It is difficult to achieve efficient communication of algorithms during the gradient compression process: Some gradient compression methods will lead to partial loss of information. Although the communication is efficient, it will reduce the final model performance. At the same time, it is also necessary to consider whether the method used to compress the gradient will generate higher computing resources. 2. It is difficult to achieve accurate gradient quantization. Gradient quantization will introduce certain errors, which may affect the training effect and convergence speed of the model.

[0026] 3. It is difficult to choose the encoding. Selecting the correct encoding method can maximize the compression rate while ensuring data accuracy. The encoding and decoding process may introduce additional computing overhead, which may affect the real-time performance of gradient updates. The selected encoding method must have a certain degree of adaptability and flexibility, and be able to be adjusted and optimized according to different scenarios. At the same time, the encoding method should ensure good compatibility and efficiency at the decoding end.

[0027] Based on the above difficulties, the first embodiment of the present invention provides a distributed machine learning method based on gradient compression and quantization, such as Figure 1 As shown in the figure, it includes two parts: the server distributes the gradient to the client and the client uploads the gradient to the server. The cloud server aggregates the gradient information of the client, generates a gradient distribution table for n clients, and performs gradient quantization encoding to achieve gradient data compression. During the communication interaction process, the cloud server encodes the compressed gradient information and aggregates it through the edge server and sends it to the client. The client accumulates the local gradient information of the adaptive gradient screening. The specific process is to determine whether the node is trustworthy after training. If it is trusted, the gradient information is uploaded. Otherwise, the local accumulation will skip this round of communication. After the client model is trained, the trustworthy gradient information is uploaded to the cloud server through the edge server.

[0028] The specific steps include: Step 1: The server obtains the client gradient information and caches it.

[0029] Step 1.1: The server caches a corresponding gradient information locally for each client, as shown in the gradient distribution table in Figure 1.

[0030] Before distributing the gradient information to the client, first count the distribution of the gradient information of the corresponding client in the local cache and calculate the mean. and variance .

[0031] Step 1.2: The distribution of the gradient information obtained in the previous round is a normal distribution. Most of the data will be concentrated near the average value. The probability of data farther away from the average value is smaller. In order to remove some extreme gradient information and constrain the boundaries of the gradient value for easy quantization, the threshold interval of the gradient information to be distributed in the current round is set to 3σ according to the 3σ rule. , the elements in the gradient outside this interval are set to zero, and the threshold interval to be quantized is obtained.

[0032] Step 2: Perform data compression on the client gradient information. Discrete quantization is used to perform preliminary compression on the client gradient information, and the encoding method is used to further compress the preliminary compressed information.

[0033] Step 2.1: Iterate the gradient information and quantize the gradient information before each iteration to achieve data compression.

[0034] Step 2.1.1: Get the threshold interval to be quantified and divide the threshold interval into sub-intervals, forming endpoint values.

[0035] Step 2.1.2: Approximate each element in the gradient information to the nearest endpoint.

[0036] Step 2.1.3: After discrete quantization, all gradient values ​​in the threshold interval can be used The values ​​outside the interval are represented by 0.

[0037] Step 2.2: Use Huffman coding to further compress the amount of information of the quantized gradient information to obtain the optimal variable-length code.

[0038] In order to further compress the amount of information, this embodiment uses Huffman coding to process the possible common The values ​​are optimally encoded with variable length according to their frequency of occurrence. The specific process is as follows: 1. First, according to the 3σ rule and quantization step, all gradient values ​​have been mapped to a finite set of discrete values ​​(the set includes P+1 quantization interval endpoint values ​​and values ​​outside the interval (set to 0)).

[0039] 2. Secondly, count the frequency of occurrence of each quantized value, that is, the number of times each discretized gradient value appears in the current training round.

[0040] 3. Then, based on this frequency information, a Huffman tree is constructed; starting from the root node of the Huffman tree, binary codes are assigned according to the structure of the tree (usually 0 is assigned to the left branch and 1 is assigned to the right branch), and each leaf node (that is, the quantized gradient value) will eventually be assigned a binary code.

[0041] 4. Finally, the generated Huffman code is used to compress the quantized gradient. Each quantized value (including 0) will be replaced by the corresponding Huffman code.

[0042] Threshold interval , quantization interval number Taking 2 as an example, the gradient mapping code table is shown in Table 1.

[0043] Table 1. Gradient mapping code representation

[0044] After each round of discrete and encoded compression iteration of the gradient information, the server will re-record and count the uncompressed gradient distribution of each client before the next round of iteration to achieve real-time dynamic update of the quantization interval and optimize the compression effect.

[0045] Step 3: Distribute the compressed gradient information to the client.

[0046] After discrete quantization and encoding processing, when the compressed gradient information is distributed to the client as a global model, it is only necessary to splice the encoding values ​​of all elements in the gradient information bit by bit to form a bit string, and distribute the bit string and the gradient mapping code table to the corresponding client, thereby achieving efficient compression of the gradient and reducing the amount of data distributed by the server.

[0047] Step 4: After the client receives the compressed gradient information, the edge server corresponding to the client uses the edge computing method to train the local model, and obtains the updated gradient information by adaptively calculating and selecting the gradient threshold.

[0048] Step 4.1: Assume that a single edge server is connected to indivual The client performs gradient updates. Represents the collection of all device nodes under the edge server.

[0049] Step 4.2: In In each round of local iteration, the edge server first sends the current global model to all subordinate device nodes.

[0050] Step 4.3: Each device node then calculates the local parameter gradient And upload it to the edge, the edge server obtains the gradient sum through weighted summation .

[0051] The most important thing in this step is to determine whether the communication node is worth communicating with. Indicates that nodes worth communicating with are Indicates that the set of all client nodes , the sum of gradients can be expressed as: (1).

[0052] in, represents the sum of gradients, represents the sum of gradients of nodes that are not worth communicating with, The sum of gradients of nodes that are worth communicating with.

[0053] Step 4.4: Set the trust threshold through the adaptive algorithm, use the trust threshold to evaluate the trust of all gradients, and obtain a set of trustworthy communication nodes.

[0054] The adaptive threshold gradient compression used when uploading gradient information, the choice of threshold is very important. Due to the characteristics of distributed learning, the client data is usually not independent and identically distributed, and the amount of data may be unbalanced, which may cause the gradients of some clients to be significantly different from those of other clients, which will affect the choice of threshold. At the same time, the choice of threshold needs to weigh the accuracy and compression rate of the gradient. A threshold that is too high may cause too much loss of gradient information and affect the training effect of the model; a threshold that is too low may not achieve the expected compression effect. This embodiment determines the trust threshold through the following process: For nodes that are not worth communicating with, they are defined here as: (2).

[0055] Update rules for the client local model: After the client device updates the local model, it calculates the average gradient parameter: (3).

[0056] in, are model parameters, For the Clients, is the learning rate, is the total number of local model iterations, represents the gradient calculated in the previous round. For the Clients, , Total number of clients

[0057] Substituting equation (3) into equation (2) yields: (4).

[0058] in, Indicates the server's Round parameter, i.e. gradient Therefore, the The sum of the information of the round gradient is expressed as: (5).

[0059] in, Represents a device node exist The gradient of the round time, It is a collection of device points under a single edge server. .

[0060] From (5) and the mean inequality, we can get the following: (6).

[0061] According to the above analysis, if the client node is not a trustworthy node, that is, m∈ When , if the node meets the formula (7), then the updated parameters can definitely meet the gradient descent algorithm .

[0062] (7).

[0063] Next we need a scaling factor To measure the proportion of untrustworthy nodes and summary points in client nodes, as follows: (8).

[0064] A large proportional coefficient means that there are more untrustworthy nodes in this round of training, that is, more device nodes skip this round of communication, and vice versa. In actual applications, it is dynamically adjusted. In the initial stage, a smaller proportional coefficient can be selected. , ensuring that most nodes participate in communication and helping the model converge quickly; in the subsequent stages, as the model gradually stabilizes, , reduce communication volume and improve training efficiency.

[0065] Substituting formula (8) into formula (7) yields: (9).

[0066] Given that when the parameters are updated, It is not easy to obtain, but this parameter tends to be smooth during model training, so it is approximated as: (10).

[0067] In the above formula and D are set constant coefficients. For the convenience of calculation, this embodiment selects , Substituting formula (8) into formula (10) yields the client node gradient screening formula as follows: (11).

[0068] This formula takes into account the gradient information of the previous rounds and uses weighted average to make a smooth estimate of the gradients of the previous rounds, which helps to more accurately judge the gradient change trend. By introducing historical gradient information, the current gradient change can be estimated more smoothly, avoiding incorrect judgments due to abnormally large or small gradients in a certain round.

[0069] The principle of the above node gradient screening formula is based on the analysis of gradient size and change trend, and a certain threshold is set to determine whether the node needs to participate in communication. Specifically, when the node's gradient changes slightly and meets certain conditions, the node is considered to be an untrusted node and can skip this round of communication, thereby reducing communication overhead. When the node's gradient changes significantly, the node is considered to be a trusted node and must participate in communication to update the global model.

[0070] According to formula (11), the device node can be adaptively judged. When the node satisfies the formula, the current round of communication is skipped, the local gradient is accumulated, and the next round of training is continued. If the node does not satisfy the formula, it is judged as a trustworthy node, and the node participates in the current round of communication with the server. Otherwise, it is judged as an untrustworthy node. When the untrustworthy node is judged as a trustworthy node in the subsequent training process, it can be uploaded.

[0071] Step 5: The client uploads the updated information to the server.

[0072] Embodiment 2: Embodiment 2 of the present invention provides a distributed machine learning system based on gradient compression and quantization, including: An information acquisition module is configured to acquire client gradient information and cache it on the server; An information compression module is configured to perform data compression processing on the client gradient information, wherein the client gradient information is preliminarily compressed by using a discretization quantization process, and the preliminarily compressed information is further compressed by using an encoding method; An information distribution module is configured to distribute the compressed gradient information to the client; The local processing module is configured to adaptively calculate and select the gradient threshold using the edge computing method after the client receives the compressed gradient information, so as to obtain updated gradient information; The information feedback module is configured so that the client uploads the updated information to the server.

[0073] Embodiment three: Embodiment 3 of the present invention provides a medium on which a program is stored. When the program is executed by a processor, the steps in the distributed machine learning method based on gradient compression and quantization as described in Embodiment 1 of the present invention are implemented.

[0074] Embodiment 4: Embodiment 4 of the present invention provides a device, including a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, the steps in the distributed machine learning method based on gradient compression and quantization as described in Embodiment 1 of the present invention are implemented.

[0075] The steps involved in the above embodiments 2, 3 and 4 correspond to the method embodiment 1, and the specific implementation methods can refer to the relevant description part of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0076] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0077] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A distributed machine learning method based on gradient compression and quantization, characterized in that: The following steps are involved: The server obtains the client's gradient information and caches it; Performing data compression processing on the client gradient information, wherein the client gradient information is preliminarily compressed by using a discretization quantization process, and the preliminarily compressed information is further compressed by using an encoding method; Distribute the compressed gradient information to the client; After receiving the compressed gradient information, the client uses the edge computing method to adaptively calculate and select the gradient threshold to obtain the updated gradient information; The client uploads the updated information to the server.

2. The distributed machine learning method based on gradient compression and quantization according to claim 1, characterized in that: The specific steps for the server to obtain client gradient information and cache it are: Before distributing the gradient information to the client, first count the distribution of the gradient information of the corresponding client in the previous round in the local cache, and calculate the mean and variance to obtain the threshold range to be quantized.

3. The distributed machine learning method based on gradient compression and quantization according to claim 1, characterized in that: The specific steps of data compression processing for client gradient information include: Iterate the gradient information and quantize the gradient information before each iteration to achieve data compression; Huffman coding is used to further compress the amount of information of the gradient information after quantization to obtain the optimal variable-length code.

4. The distributed machine learning method based on gradient compression and quantization according to claim 3, characterized in that: The specific steps for quantifying gradient information are: Obtain a threshold interval to be quantized, and divide the threshold interval into P sub-intervals on average to form P+1 endpoint values; Approximate each element in the gradient information to the nearest endpoint.

5. The distributed machine learning method based on gradient compression and quantization according to claim 4, characterized in that: After each iteration of the gradient information, the server will re-record and count the original uncompressed gradient distribution of each client to achieve real-time dynamic update of the quantization interval and optimize the compression effect.

6. The distributed machine learning method based on gradient compression and quantization according to claim 1, characterized in that: When distributing the compressed gradient information to the client, the encoding values ​​of all elements in the gradient information are concatenated bit by bit to form a bit string, and the bit string and the gradient mapping code table are distributed to the corresponding client.

7. The distributed machine learning method based on gradient compression and quantization according to claim 1, characterized in that: The specific steps of using edge computing method to adaptively calculate and select gradient threshold are as follows: The edge server first sends the current global model to all subordinate device nodes; Each device node then calculates the local parameter gradient and uploads it to the edge, where the edge server obtains the gradient sum through weighted summation; The trust threshold is set through an adaptive algorithm, and the trust threshold is used to evaluate the trust of all gradients to obtain a set of trustworthy communication nodes.

8. A distributed machine learning system based on gradient compression and quantization, characterized in that: include: An information acquisition module is configured to acquire client gradient information and cache it on the server; An information compression module is configured to perform data compression processing on the client gradient information, wherein the client gradient information is preliminarily compressed by using a discretization quantization process, and the preliminarily compressed information is further compressed by using an encoding method; An information distribution module is configured to distribute the compressed gradient information to the client; The local processing module is configured to adaptively calculate and select the gradient threshold using the edge computing method after the client receives the compressed gradient information, so as to obtain updated gradient information; The information feedback module is configured so that the client uploads the updated information to the server.

9. A computer-readable storage medium, characterized in that: Multiple instructions are stored therein, and the instructions are suitable for being loaded by a processor of a terminal device and executing the distributed machine learning method based on gradient compression and quantization described in any one of claims 1-7.

10. A terminal device, characterized in that: It includes a processor and a computer-readable storage medium, the processor is used to implement each instruction; the computer-readable storage medium is used to store multiple instructions, and the instructions are suitable for being loaded by the processor and executing the distributed machine learning method based on gradient compression and quantization described in any one of claims 1-7.

Images