Method, device and equipment for open domain question answering implementation of large-scale language model
By acquiring the segmentation operator and the communication operator to generate parallel strategies, the problem of insufficient utilization of heterogeneous many-core devices in the existing technology is solved, and the efficient training of large-scale language models and the improvement of intelligent question answering are realized.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- 太初(无锡)电子科技有限公司
- Filing Date
- 2023-06-09
- Publication Date
- 2026-06-02
Smart Images
Figure CN116662509B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of deep learning technology, and in particular to a method, apparatus and device for implementing open-domain question answering for large-scale language models. Background Technology
[0002] As large-scale language models become increasingly complex, single-machine, single-GPU training is no longer sufficient to meet the training requirements of large-scale models. Therefore, multi-machine, multi-GPU parallel training has become a research hotspot in the field of deep learning. In multi-machine, multi-GPU parallel training, research on how to effectively utilize heterogeneous many-core devices and automated parallel strategies has received widespread attention.
[0003] Currently, existing parallel training methods typically utilize existing parallel training frameworks, such as TensorFlow and PyTorch, to achieve parallel training of large-scale models. However, most of these frameworks focus on optimizing for a single hardware platform and fail to effectively leverage the advantages of heterogeneous many-core devices. Furthermore, current parallel training frameworks require users to manually adjust parallel training strategies and parameters, and necessitate re-optimization for different hardware platforms, which is difficult for non-expert users and presents certain challenges and limitations in practical use. Summary of the Invention
[0004] This invention provides a method, apparatus, and device for implementing open-domain question answering for large-scale language models, which can improve the training efficiency of large-scale language models and the implementation efficiency of intelligent question answering.
[0005] According to one aspect of the present invention, an open-domain question-answering implementation method for a large-scale language model is provided, comprising:
[0006] Obtain multiple segmentation operators, and obtain multiple segmentation strategies corresponding to each segmentation operator, as well as communication operators corresponding to each segmentation strategy;
[0007] Based on each of the segmentation operators, the multiple segmentation strategies corresponding to each of the segmentation operators, and the communication operators corresponding to each of the segmentation strategies, multiple candidate parallel strategies are obtained;
[0008] The target parallel strategy is obtained from the multiple candidate parallel strategies using a preset strategy search algorithm;
[0009] Distributed parallel training of a large-scale language model is performed according to the target parallel strategy to obtain the trained target language model, and the target language model is used to implement open-domain question answering.
[0010] According to another aspect of the present invention, an open-domain question-answering implementation apparatus for a large-scale language model is provided, comprising:
[0011] The segmentation operator acquisition module is used to acquire multiple segmentation operators, acquire multiple segmentation strategies corresponding to each segmentation operator, and acquire communication operators corresponding to each segmentation strategy;
[0012] The candidate parallel strategy acquisition module is used to acquire multiple candidate parallel strategies based on each of the segmentation operators, multiple segmentation strategies corresponding to each of the segmentation operators, and communication operators corresponding to each of the segmentation strategies.
[0013] The target parallel strategy acquisition module is used to acquire the target parallel strategy from the multiple candidate parallel strategies through a preset strategy search algorithm.
[0014] The model training module is used to perform distributed parallel training of a large-scale language model according to the target parallel strategy, so as to obtain the trained target language model, and to use the target language model to implement open-domain question answering.
[0015] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:
[0016] At least one processor; and
[0017] A memory communicatively connected to the at least one processor; wherein,
[0018] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to execute the open-domain question-answering implementation method of the large-scale language model according to any embodiment of the present invention.
[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the open-domain question-answering method for a large-scale language model as described in any embodiment of the present invention.
[0020] The technical solution of this invention involves obtaining multiple segmentation operators, multiple segmentation strategies corresponding to each segmentation operator, and communication operators corresponding to each segmentation strategy. Then, based on each segmentation operator, each segmentation strategy, and each communication operator, multiple candidate parallel strategies are obtained. A target parallel strategy is then selected from these candidate parallel strategies using a preset strategy search algorithm. Finally, a large-scale language model is trained in a distributed parallel manner according to the target parallel strategy to obtain the trained target language model. This target language model is then used to implement open-domain question answering. By automatically generating candidate parallel strategies and selecting the optimal parallel strategy, the training efficiency of the large-scale language model and the implementation efficiency of intelligent question answering can be improved.
[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description
[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0023] Figure 1A This is a flowchart of an open-domain question-answering implementation method for a large-scale language model according to Embodiment 1 of the present invention;
[0024] Figure 1B This is a schematic diagram of the structure of a heterogeneous many-core device according to Embodiment 1 of the present invention;
[0025] Figure 1C This is a schematic diagram of a strategy search process provided in Embodiment 1 of the present invention;
[0026] Figure 1D This is a schematic diagram of an operator insertion according to Embodiment 1 of the present invention;
[0027] Figure 2 This is a schematic diagram of the structure of an open-domain question-answering implementation device for a large-scale language model according to Embodiment 2 of the present invention;
[0028] Figure 3 This is a schematic diagram of the structure of an electronic device that implements the open-domain question-answering method of a large-scale language model according to an embodiment of the present invention. Detailed Implementation
[0029] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0030] It should be noted that the terms "first," "second," "target," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0031] Example 1:
[0032] Figure 1A This is a flowchart illustrating an open-domain question-answering implementation method for a large-scale language model, as provided in Embodiment 1 of the present invention. This embodiment is applicable to situations where open-domain question answering is implemented using a large-scale language model trained in parallel on heterogeneous many-core devices. The method can be executed by an open-domain question-answering implementation device for a large-scale language model. This device can be implemented in hardware and / or software and can be configured in an electronic device, typically a computer or server. Figure 1A As shown, the method includes:
[0033] S110. Obtain multiple segmentation operators, and obtain multiple segmentation strategies corresponding to each segmentation operator, as well as communication operators corresponding to each segmentation strategy.
[0034] Here, operators can be building blocks of deep learning models used to implement specific data processing functions, such as convolutional layer operators and pooling layer operators. Segmentation operators can be operators with feature or data segmentation capabilities, such as convolutional layer operators. Communication operators can be operators used to implement data communication functions.
[0035] Optionally, the segmentation operator may include a row segmentation operator, a column segmentation operator, and / or a copy operator; wherein, different segmentation operators may correspond to different segmentation strategies. For example, a row segmentation operator may correspond to a row segmentation strategy, that is, to segment the input features by row; a column segmentation operator may correspond to a column segmentation strategy, that is, to segment the input features by column; and a copy operator may correspond to a copy strategy, that is, to copy the input features.
[0036] The communication operators may include reduction operators and / or collection operators. For example, a reduction operator could be an Allreduce operator, used to collect data processing results from various devices using preset rules (e.g., averaging, weighted summation, etc.). Furthermore, a collection operator could be an All gather operator, which shares the data processing results from different devices with all devices.
[0037] In this embodiment, for each component unit of the deep learning model, a corresponding segmentation operator type and a corresponding feasible segmentation strategy can be automatically assigned; secondly, a feasible communication operator can be inserted for each segmentation strategy. Typically, the segmentation operator type, segmentation strategy, and communication operator can be preset.
[0038] S120. Based on each of the segmentation operators, the multiple segmentation strategies corresponding to each of the segmentation operators, and the communication operators corresponding to each of the segmentation strategies, obtain multiple candidate parallel strategies.
[0039] In this embodiment, a mapping relationship between a segmentation operator, a segmentation strategy, and a communication operator can be considered as a candidate parallel strategy, thereby obtaining multiple candidate parallel strategies. Then, by considering different overheads, the optimal parallel strategy can be selected from the candidate parallel strategies.
[0040] The parallel strategy can be a strategy for parallel training of a deep learning model. In this embodiment, multiple heterogeneous many-core devices can be used to perform distributed parallel training of the deep learning model based on the finally selected parallel strategy. A heterogeneous many-core device can be a processor or system that integrates multiple processor cores with different architectures on a single chip.
[0041] S130. Obtain the target parallel strategy from the multiple candidate parallel strategies using a preset strategy search algorithm.
[0042] The preset strategy search algorithm can include greedy algorithms and / or genetic algorithms. Greedy algorithms construct the final parallel strategy by selecting the currently optimal partitioning strategy at each step, and are simple and effective. Genetic algorithms simulate the genetic and evolutionary processes in nature, searching for the optimal solution through operations such as selection, crossover, and mutation.
[0043] In this embodiment, a preset strategy search algorithm can be used to search all candidate parallel strategies to obtain the optimal parallel strategy as the target parallel strategy. Specifically, based on the characteristics of heterogeneous many-core devices, communication overhead, computational overhead, and memory overhead can be selected to construct a loss function; then, the preset strategy search algorithm can be used to select the target parallel strategy with the smallest loss value from all candidate parallel strategies according to the loss function.
[0044] S140. Perform distributed parallel training of a large-scale language model according to the target parallel strategy to obtain the trained target language model, and use the target language model to implement open-domain question answering.
[0045] Specifically, based on this parallel strategy, multiple heterogeneous many-core devices participating in parallel training can be used to perform distributed parallel training on an initial large-scale language model using pre-collected sample data until the trained target language model is obtained. The large-scale language model can be constructed based on deep learning algorithms. In this embodiment, the structure of the large-scale speech model is not specifically limited.
[0046] In distributed parallel training, each heterogeneous many-core device can process its own data and communicate with other devices to merge the results after computation. This process requires a target parallelism strategy to minimize communication and computational overhead while ensuring that memory usage does not exceed the device's own memory limits.
[0047] Furthermore, after obtaining the target language model, open-domain intelligent question answering can be implemented based on it. For example, the user's question text can be input into the target language model, and the answer text output by the target language model can be displayed to the user. The question text can be any document type or format.
[0048] Optionally, the number of heterogeneous many-core devices participating in parallel training can be determined based on the accuracy or speed requirements of the intelligent question answering system.
[0049] Understandably, the objective parallel strategy can also be applied to the parallel training of large-scale models in other fields; for example, object detection models, image classification models, or speech recognition models, which can greatly improve the efficiency and accuracy of object detection, image classification, speech recognition, and intelligent question answering.
[0050] The technical solution of this invention involves obtaining multiple segmentation operators, multiple segmentation strategies corresponding to each segmentation operator, and communication operators corresponding to each segmentation strategy. Then, based on each segmentation operator, each segmentation strategy, and each communication operator, multiple candidate parallel strategies are obtained. A target parallel strategy is then selected from these candidate parallel strategies using a preset strategy search algorithm. Finally, a large-scale language model is trained in a distributed parallel manner according to the target parallel strategy to obtain the trained target language model. This target language model is then used to implement open-domain question answering. By automatically generating candidate parallel strategies and selecting the optimal parallel strategy, the training efficiency of the large-scale language model and the implementation efficiency of intelligent question answering can be improved.
[0051] In another optional implementation of this embodiment, obtaining the target parallel strategy from the plurality of candidate parallel strategies through a preset strategy search algorithm may include:
[0052] Based on computational overhead, communication overhead, and memory overhead, obtain the loss function;
[0053] The target parallel strategy is obtained from the plurality of candidate parallel strategies by means of a preset strategy search algorithm and according to the loss function.
[0054] It should be noted that the structure of heterogeneous many-core devices can be as follows: Figure 1B As shown, compared to ordinary accelerator cards, heterogeneous many-core devices have interleaved memory segments and core-group-specific memory. The interleaved memory segments are allocated from the many-core memory of the accelerator card, forming contiguous memory addresses; while the core-group-specific memory segments are allocated from the many-core memory of the accelerator card, allowing specific many cores to access them at high speed. Therefore, heterogeneous many-core devices have certain advantages in communication, and communication factors need to be considered in policy search. Furthermore, unlike ordinary accelerator cards, a single heterogeneous many-core device has a 4-core group, so the computational factors of the core groups also need to be considered in policy search.
[0055] In this embodiment, the strategy search process can be as follows: Figure 1C As shown, the loss function used for parallel training can be set based on communication overhead, computational overhead, and memory overhead. For example, since communication overhead is often the bottleneck in distributed parallelism, and each device has limited memory, it is necessary to consider device memory usage to avoid memory overflow. Therefore, the ratio of total communication overhead to total computational overhead, and whether the memory usage ratio is less than a preset threshold, can be used as the loss function. Then, a preset policy search algorithm can be used to select the optimal target parallel policy from all candidate parallel policies based on this loss function.
[0056] The advantages of the above settings are that they can fully utilize the advantages of heterogeneous many-core devices, automatically select the best parallel strategy based on hardware configuration and model parameters, improve the efficiency of large-scale parallel training of models, shorten training time, and increase the iteration speed of models.
[0057] In another optional implementation of this embodiment, obtaining the target parallel strategy from the plurality of candidate parallel strategies according to the loss function using a preset strategy search algorithm may include:
[0058] By using a preset strategy search algorithm and based on the loss function, the loss value corresponding to each candidate parallel strategy is obtained;
[0059] If the minimum loss value corresponding to the current candidate parallel strategy is detected, then the current candidate parallel strategy is taken as the target parallel strategy.
[0060] In a specific example, a pre-defined strategy search algorithm can be used to calculate the ratio of communication overhead to computational overhead for each candidate parallel strategy, which serves as the loss value. Simultaneously, the memory usage ratio for each candidate parallel strategy can be obtained. Then, it is first determined whether the memory usage ratio for each candidate parallel strategy is less than a pre-defined threshold to obtain a certain number of initial candidate parallel strategies. Next, the loss values of all initially selected candidate parallel strategies are compared to select the initial candidate parallel strategy with the minimum loss value, which is then used as the final target parallel strategy.
[0061] In another optional implementation of this embodiment, performing distributed parallel training of a large-scale language model according to the target parallel strategy to obtain the trained target language model may include:
[0062] Based on the target parallel strategy, obtain the target segmentation operator, the target segmentation strategy, and the target communication operator;
[0063] The input features of the target segmentation operator are segmented according to the target segmentation strategy to obtain each segmentation feature, and each segmentation feature is sent to each training participating device;
[0064] The target communication operator is used to summarize the calculation results fed back by each of the training participating devices to obtain the total calculation result corresponding to the input feature.
[0065] In a specific example, the operator insertion can be illustrated as follows: Figure 1D As shown. For a convolutional layer operator, a column segmentation operator corresponding to the column segmentation logic can be inserted. This allows the input features to be segmented by column to obtain two segmented features: [0,1,2,3; 4,5,6,7]×[10,11,12,13] and [0,1,2,3; 4,5,6,7]×[14,15,16,17]. Then, the segmented features can be sent to a training device to calculate the convolution result. Finally, the calculation results from each training device can be aggregated and merged using the all-gather operator to obtain the total calculation result [14,98; 258,346].
[0066] Secondly, when inserting the row segmentation operator of the corresponding row segmentation logic, the input features can be segmented by row to obtain two segmentation features: [0,1; 4,5]×[10,14; 11,15] and [2,3; 6,7]×[12,16; 13,17]. Then, the segmentation features can be sent to the corresponding training devices, and the calculation results of each training device can be reduced by the all reduce operator. For example, the element values at the same position in the matrix are added together to obtain the final total calculation result [74,98; 258,346].
[0067] It should be noted that when inserting the copy operator corresponding to the copy logic, the input features can be copied and distributed to each training device, enabling each device to perform the same convolution operation. Finally, the calculation results from each training device can be aggregated using the current target communication operator to obtain the final overall calculation result.
[0068] Therefore, the parameters of each component of a large-scale language model can be adjusted through the above process to obtain the final trained language model.
[0069] The advantages of the above settings are that they can automatically perform parallel processing based on the characteristics of different models and the availability of hardware resources, reducing the workload of manual adjustments, lowering the difficulty of operation, and improving training efficiency and accuracy; secondly, they can support large-scale model training and meet the training needs of massive amounts of data and complex models.
[0070] Example 2:
[0071] Figure 2 This is a schematic diagram of the structure of an open-domain question-answering implementation device for a large-scale language model provided in Embodiment 2 of the present invention. Figure 2 As shown, the device may include: a segmentation operator acquisition module 210, a candidate parallel strategy acquisition module 220, a target parallel strategy acquisition module 230, and a model training module 240; wherein,
[0072] The segmentation operator acquisition module 210 is used to acquire multiple segmentation operators, acquire multiple segmentation strategies corresponding to each segmentation operator, and acquire communication operators corresponding to each segmentation strategy;
[0073] The candidate parallel strategy acquisition module 220 is used to acquire multiple candidate parallel strategies based on each of the segmentation operators, multiple segmentation strategies corresponding to each of the segmentation operators, and communication operators corresponding to each of the segmentation strategies.
[0074] The target parallel strategy acquisition module 230 is used to acquire the target parallel strategy from the multiple candidate parallel strategies through a preset strategy search algorithm.
[0075] The model training module 240 is used to perform distributed parallel training of a large-scale language model according to the target parallel strategy, so as to obtain the trained target language model and use the target language model to implement open-domain question answering.
[0076] The technical solution of this invention involves obtaining multiple segmentation operators, multiple segmentation strategies corresponding to each segmentation operator, and communication operators corresponding to each segmentation strategy. Then, based on each segmentation operator, each segmentation strategy, and each communication operator, multiple candidate parallel strategies are obtained. A target parallel strategy is then selected from these candidate parallel strategies using a preset strategy search algorithm. Finally, a large-scale language model is trained in a distributed parallel manner according to the target parallel strategy to obtain the trained target language model. This target language model is then used to implement open-domain question answering. By automatically generating candidate parallel strategies and selecting the optimal parallel strategy, the training efficiency of the large-scale language model and the implementation efficiency of intelligent question answering can be improved.
[0077] Optionally, the target parallel strategy acquisition module 230 includes:
[0078] The loss function acquisition unit is used to acquire the loss function based on computational overhead, communication overhead, and memory overhead.
[0079] The target parallel strategy acquisition unit is used to acquire the target parallel strategy from the plurality of candidate parallel strategies by means of a preset strategy search algorithm and according to the loss function.
[0080] Optionally, the target parallel strategy acquisition unit is specifically used to obtain the loss value corresponding to each candidate parallel strategy according to the loss function through a preset strategy search algorithm.
[0081] If the minimum loss value corresponding to the current candidate parallel strategy is detected, then the current candidate parallel strategy is taken as the target parallel strategy.
[0082] Optional, model training module 240, including:
[0083] The target segmentation operator acquisition unit is used to acquire the target segmentation operator, the target segmentation strategy, and the target communication operator according to the target parallel strategy.
[0084] The segmentation feature acquisition unit is used to segment the input features of the target segmentation operator according to the target segmentation strategy to obtain each segmentation feature, and send each segmentation feature to each training participating device;
[0085] The total calculation result acquisition unit is used to communicate and summarize the calculation results fed back by each of the training participating devices through the target communication operator, so as to obtain the total calculation result corresponding to the input feature.
[0086] Optionally, the preset strategy search algorithm includes a greedy algorithm and / or a genetic algorithm.
[0087] Optionally, the segmentation operator includes a row segmentation operator, a column segmentation operator, and / or a replication operator, and the communication operator includes a reduction operator and / or a collection operator.
[0088] The open-domain question-answering implementation device for large-scale language models provided in this embodiment of the invention can execute the open-domain question-answering implementation method for large-scale language models provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.
[0089] Example 3:
[0090] Figure 3 A schematic diagram of an electronic device 30 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.
[0091] like Figure 3 As shown, the electronic device 30 includes at least one processor 31 and a memory, such as a read-only memory (ROM) 32 or a random access memory (RAM) 33, communicatively connected to the at least one processor 31. The memory stores computer programs executable by the at least one processor. The processor 31 can perform various appropriate actions and processes based on the computer program stored in the ROM 32 or loaded from storage unit 38 into the RAM 33. The RAM 33 can also store various programs and data required for the operation of the electronic device 30. The processor 31, ROM 32, and RAM 33 are interconnected via a bus 34. An input / output (I / O) interface 35 is also connected to the bus 34.
[0092] Multiple components in electronic device 30 are connected to I / O interface 35, including: input unit 36, such as keyboard, mouse, etc.; output unit 37, such as various types of monitors, speakers, etc.; storage unit 38, such as disk, optical disk, etc.; and communication unit 39, such as network card, modem, wireless transceiver, etc. Communication unit 39 allows electronic device 30 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.
[0093] Processor 31 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 31 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 31 performs the various methods and processes described above, such as open-domain question-answering implementations of large-scale language models.
[0094] In some embodiments, the open-domain question-answering implementation method for a large-scale language model can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 38. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 30 via ROM 32 and / or communication unit 39. When the computer program is loaded into RAM 33 and executed by processor 31, one or more steps of the open-domain question-answering implementation method for a large-scale language model described above can be performed. Alternatively, in other embodiments, processor 31 can be configured to execute the open-domain question-answering implementation method for a large-scale language model by any other suitable means (e.g., by means of firmware).
[0095] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.
[0096] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.
[0097] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.
[0098] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).
[0099] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.
[0100] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.
[0101] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.
[0102] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.
Claims
1. A method for open domain question answering implementation of a large-scale language model, characterized in that, include: Obtain multiple segmentation operators, and obtain multiple segmentation strategies corresponding to each segmentation operator, as well as communication operators corresponding to each segmentation strategy; Based on each of the segmentation operators, the multiple segmentation strategies corresponding to each of the segmentation operators, and the communication operators corresponding to each of the segmentation strategies, multiple candidate parallel strategies are obtained; Based on computational overhead, communication overhead, and memory overhead, obtain the loss function; Using a preset strategy search algorithm, the ratio of communication overhead to computation overhead corresponding to each candidate parallel strategy is calculated based on the loss function and used as the loss value. The memory usage ratio corresponding to each candidate parallel strategy is also obtained. By determining whether the memory usage ratio corresponding to each candidate parallel strategy is less than a preset usage ratio threshold, preliminary candidate parallel strategies are obtained. The loss values corresponding to all preliminary candidate parallel strategies are compared to obtain the preliminary candidate parallel strategy with the minimum loss value, which is then used as the final target parallel strategy. Distributed parallel training of a large-scale language model is performed according to the target parallel strategy to obtain the trained target language model, and the target language model is used to implement open-domain question answering.
2. The method of claim 1, wherein, Distributed parallel training of a large-scale language model is performed according to the stated target parallel strategy to obtain the trained target language model, including: Based on the target parallel strategy, obtain the target segmentation operator, the target segmentation strategy, and the target communication operator; The input features of the target segmentation operator are segmented according to the target segmentation strategy to obtain each segmentation feature, and each segmentation feature is sent to each training participating device; The target communication operator is used to summarize the calculation results fed back by each of the training participating devices to obtain the total calculation result corresponding to the input feature.
3. The method of claim 1, wherein, The preset strategy search algorithm includes a greedy algorithm and / or a genetic algorithm.
4. The method according to claim 1 or 2, characterized in that, The segmentation operator includes a row segmentation operator, a column segmentation operator, and / or a replication operator, and the communication operator includes a reduction operator and / or a collection operator.
5. An open-domain question answering implementation apparatus of a large-scale language model, characterized by, include: The segmentation operator acquisition module is used to acquire multiple segmentation operators, acquire multiple segmentation strategies corresponding to each segmentation operator, and acquire communication operators corresponding to each segmentation strategy; The candidate parallel strategy acquisition module is used to acquire multiple candidate parallel strategies based on each of the segmentation operators, multiple segmentation strategies corresponding to each of the segmentation operators, and communication operators corresponding to each of the segmentation strategies. The target parallel strategy acquisition module is used to obtain the loss function based on computational overhead, communication overhead, and memory overhead. Using a preset strategy search algorithm, the ratio of communication overhead to computation overhead corresponding to each candidate parallel strategy is calculated based on the loss function and used as the loss value. The memory usage ratio corresponding to each candidate parallel strategy is also obtained. By determining whether the memory usage ratio corresponding to each candidate parallel strategy is less than a preset usage ratio threshold, preliminary candidate parallel strategies are obtained. The loss values corresponding to all preliminary candidate parallel strategies are compared to obtain the preliminary candidate parallel strategy with the minimum loss value, which is then used as the final target parallel strategy. The model training module is used to perform distributed parallel training of a large-scale language model according to the target parallel strategy, so as to obtain the trained target language model, and to use the target language model to implement open-domain question answering.
6. An electronic device, comprising: The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the open-domain question-answering implementation method of the large-scale language model according to any one of claims 1-4.
7. A computer readable storage medium characterized by The computer-readable storage medium stores computer instructions that cause a processor to execute and implement the open-domain question-answering method for a large-scale language model as described in any one of claims 1-4.