Model inference method, apparatus, device, medium, and program product

Abstract

The application provides a model reasoning method, which can be applied to the field of artificial intelligence. The model reasoning method comprises the following steps: obtaining reasoning information; inputting the reasoning information into a preset model comprising a plurality of network layers, to sequentially perform reasoning processing on the reasoning information according to a time sequence dependency relationship between the plurality of network layers, wherein each network layer in the plurality of network layers comprises a main thread and a plurality of slave threads, and the main thread and the plurality of slave threads are used for multi-dimensional parallel processing of data input into the network layer; for each network layer, if it is determined that the main thread receives processing states respectively fed back by the plurality of slave threads, the reasoning data output by the network layer is transmitted to a next network layer, so that the next network layer continues the reasoning processing; and if it is determined that the plurality of network layers all complete the reasoning processing, a reasoning result corresponding to the reasoning information is output. The application also provides a model reasoning device, equipment, storage medium and program product.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, and in particular to a model inference method, device, equipment, medium and program product. BACKGROUND

[0002] With the continuous development of large language models, the weight scale thereof also continues to grow. In order to cope with this growth, large language models usually adopt a distributed strategy, combining model parallelism and data parallelism and other technologies to achieve efficient training and inference. In related technologies, a mode of multi-process control is adopted for model inference, and each process independently runs in the inference process and is responsible for processing part of the computing task.

[0003] In the process of implementing the present application concept, the inventors have found that in related technologies, at least the following problems exist. Due to data isolation between processes, data cannot be shared between processes without communication. However, the process of model inference requires frequent communication between multiple processes, and the current process communication method (such as message passing) will bring about a large communication overhead, resulting in low model inference efficiency. SUMMARY

[0004] In view of the above problems, the present application provides a model inference method, device, equipment, medium and program product.

[0005] According to a first aspect of the present application, a model inference method is provided, comprising: obtaining inference information; inputting the inference information into a preset model comprising a plurality of network layers to sequentially perform inference processing on the inference information according to a time sequence dependency relationship between the plurality of network layers, wherein each network layer in the plurality of network layers comprises a main thread and a plurality of slave threads, and the main thread and the plurality of slave threads are used for multi-dimensional parallel processing of data input into the network layer; for each network layer, in a case where it is determined that the main thread receives processing states respectively fed back by the plurality of slave threads, passing inference data output by the network layer to a next network layer to enable the next network layer to continue inference processing; and in a case where it is determined that the plurality of network layers have all completed inference processing, outputting an inference result corresponding to the inference information.

[0006] According to an embodiment of the present application, the inputting the inference information into the preset model comprising a plurality of network layers to sequentially perform inference processing on the inference information according to a time sequence dependency relationship between the plurality of network layers comprises: inputting the inference information into the preset model to sequentially perform inference processing on the inference information using the plurality of network layers according to the time sequence dependency relationship; in a case where it is determined that an inference result output by a target network layer in the preset model is obtained, repeatedly performing the following operations until a preset number of iterations is reached: inputting an inference result output by a previous iteration task into the preset model to sequentially perform inference processing on the inference result using the plurality of network layers according to the time sequence dependency relationship to obtain an inference result output by a current iteration task.

[0007] According to an embodiment of the present application, the inputting the inference information into the preset model comprising a plurality of network layers to sequentially perform inference processing on the inference information according to a time sequence dependency relationship between the plurality of network layers comprises: inputting the inference information into the preset model to sequentially perform inference processing on the inference information using the plurality of network layers according to the time sequence dependency relationship; in a case where it is determined that an inference result output by a target network layer in the preset model is obtained, repeatedly performing the following operations until a preset number of iterations is reached: inputting an inference result output by a previous iteration task into the preset model to sequentially perform inference processing on the inference result using the plurality of network layers according to the time sequence dependency relationship to obtain an inference result output by a current iteration task.

[0008] According to an embodiment of the present application, the method further comprises: for each network layer, according to a model parameter of the preset model, splitting data input into the network layer into a plurality of sub-data blocks to be assigned to the master thread and the plurality of slave threads; performing parallel processing on the split sub-data blocks using the master thread and the plurality of slave threads; and combining results obtained by the master thread and each slave thread to obtain inference data output by the network layer.

[0009] According to an embodiment of the present application, the method further comprises: creating a thread lock for each of the stages in which the main thread processes the data input into the network layer; enabling the thread lock when it is determined that the main thread and the plurality of slave threads are processing the data at the current stage; releasing the thread lock of the current stage when the main thread receives the stage status feedback from each of the plurality of slave threads in the current stage, so that the main thread and the plurality of slave threads perform data processing in the next stage; determining the release state of the thread lock when it is determined that new data input into the network layer is obtained and the main thread and the plurality of slave threads have completed processing of the data in the plurality of stages; waiting for the thread lock to be released when it is determined that the thread lock is in the enabled state; and processing the obtained new data using the main thread and the plurality of slave threads when it is determined that the thread lock is in the released state.

[0010] According to an embodiment of the present application, when it is determined that the main thread receives the processing status feedback from each of the plurality of slave threads, the method further comprises: for each of the network layers, receiving the processing status feedback from each of the slave threads to the main thread based on the sub-communication domain corresponding to the network layer; determining the task execution status of the network layer based on the plurality of processing statuses; and when it is determined that the task execution status is execution complete, delivering the inference data output by the network layer to the next network layer using the main communication domain associated with the preset model.

[0011] According to an embodiment of the present application, the method further comprises: creating a network shared parameter corresponding to the preset model to construct a main communication domain for communication between the plurality of network layers using the network shared parameter; for each of the network layers, determining a thread corresponding to a first sub-data block in the network layer as a main thread, and determining a plurality of threads corresponding to a plurality of sub-data blocks other than the first sub-data block as slave threads; and creating a thread shared parameter based on the main thread to construct a sub-communication domain for communication between the main thread and the plurality of slave threads using the thread shared parameter.

[0012] A second aspect of this application provides a model inference apparatus, comprising: an information acquisition module for acquiring inference information; an inference processing module for, in response to a received model inference request, inputting the acquired inference information into a preset model composed of multiple network layers, and sequentially performing inference processing on the inference information according to the temporal dependencies between the multiple network layers, wherein each network layer includes a main thread and multiple slave threads, and the main thread and multiple slave threads are used to perform multi-dimensional parallel processing on the data input to the network layer; a data transmission module for, for each network layer, upon determining that the main thread has received the processing status feedback from each of the multiple slave threads, transmitting the inference data output by the network layer to the next network layer, so that the next network layer continues to perform inference processing; and a result output module for, upon determining that all of the multiple network layers have completed inference processing, outputting an inference result corresponding to the inference information.

[0013] A third aspect of this application provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0014] A fourth aspect of this application also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0015] The fifth aspect of this application also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method.

[0016] According to embodiments of this application, during model inference, a collaborative waiting mechanism between the main thread and slave threads in each network layer ensures the orderly execution of multiple network layer tasks with temporal dependencies, while also achieving efficient multidimensional parallel processing of the input network layer data. This mechanism not only guarantees the correctness of inference but also significantly improves overall performance. Furthermore, communication via a data-sharing mechanism between the main thread and slave threads enables rapid responses to the status feedback from each slave thread during data processing, further enhancing the efficiency of model inference. Attached Figure Description

[0017] The above-mentioned contents, other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0018] Figure 1 The illustrations depict application scenarios of model reasoning methods, apparatuses, devices, media, and program products according to embodiments of this application.

[0019] Figure 2 A flowchart illustrating a model reasoning method according to an embodiment of this application is shown schematically.

[0020] Figure 3 This illustration schematically shows a model inference flowchart in a model inference method according to an embodiment of the present application;

[0021] Figure 4 This illustration schematically shows a master-slave thread diagram of a preset model in a model inference method according to an embodiment of this application;

[0022] Figure 5 A schematic diagram illustrating the structure of a model inference apparatus according to an embodiment of this application is shown; and

[0023] Figure 6 A block diagram schematically illustrates an electronic device suitable for implementing a model inference method according to an embodiment of this application. Detailed Implementation

[0024] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0025] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0026] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0027] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0028] In the technical solution of this application, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse.

[0029] Embodiments of this application provide a model inference method, which includes: acquiring inference information; inputting the inference information into a preset model comprising multiple network layers, so as to perform inference processing on the inference information sequentially according to the temporal dependencies between the multiple network layers, wherein each of the multiple network layers includes a main thread and multiple slave threads, the main thread and multiple slave threads being used to perform multi-dimensional parallel processing on the data input to the network layer; for each network layer, when it is determined that the main thread has received the processing status feedback from the multiple slave threads, passing the inference data output by the network layer to the next network layer so that the next network layer continues to perform inference processing; when it is determined that the multiple network layers have completed the inference processing, outputting the inference result corresponding to the inference information.

[0030] Figure 1 The illustrations depict application scenarios of the model reasoning method, apparatus, device, medium, and program product according to embodiments of this application.

[0031] like Figure 1 As shown, application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0032] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0033] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0034] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0035] It should be noted that the model inference method provided in this application embodiment can generally be executed by server 105. Correspondingly, the model inference device provided in this application embodiment can generally be located in server 105. The model inference method provided in this application embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the model inference device provided in this application embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0036] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0037] The following will be based on Figure 1 The described scene, through Figures 2-4 The model reasoning method of the disclosed embodiments is described in detail.

[0038] Figure 2 A flowchart illustrating a model reasoning method according to an embodiment of this application is shown schematically.

[0039] like Figure 2 As shown, this embodiment includes operations S210 to S240.

[0040] In operation S210, inference information is obtained.

[0041] In operation S220, inference information is input into a preset model that includes multiple network layers, and inference information is processed sequentially according to the temporal dependencies between the multiple network layers. Each of the multiple network layers includes a main thread and multiple slave threads, which are used to perform multi-dimensional parallel processing on the data input to the network layers.

[0042] In operation S230, for each network layer, after determining that the main thread has received the processing status feedback from multiple slave threads, the inference data output by the network layer is passed to the next network layer so that the next network layer can continue to perform inference processing.

[0043] In operation S240, after determining that multiple network layers have completed inference processing, the inference result corresponding to the inference information is output.

[0044] According to embodiments of this application, as large language models continue to evolve, their weight scale also continues to grow. To cope with this growth, large language models typically employ a distributed strategy, combining techniques such as model parallelism and data parallelism to achieve efficient training and inference. Model parallelism involves splitting the large language model at the model level, dividing it by network layers (rows), and distributing different network layers to different devices for parallel computation. Data parallelism, on the other hand, involves splitting the data at the data level, dividing large tensors (such as feature data) by columns into multiple smaller tensors, and distributing them to different devices for parallel computation.

[0045] According to embodiments of this application, model parallelism involves splitting the network into layers, with temporal dependencies between these layers. The computation process from the first to the last network layer is linearly sequential. Data parallelism, on the other hand, splits large tensors in the model, and the resulting smaller tensors are distributed across multiple nodes for parallel computation, with these smaller tensors executing in parallel.

[0046] According to embodiments of this application, when performing inference on a large language model, data inference needs to be performed on each network layer and the tensors within each network layer in a model-parallel and data-parallel manner. After obtaining the inference information, the inference information is input into a preset model of the large language model to utilize multiple network layers in the large language model for inference processing. The inference information is typically text input provided by the user, such as a question, a paragraph, or a task description.

[0047] According to embodiments of this application, in the large language model, inference in each network layer is implemented through multi-threaded scheduling. Each network layer includes a main thread and multiple slave threads that work in conjunction with the main thread. During inference, the main thread and multiple slave threads in the network layer perform multi-dimensional parallel processing on the data input to the network layer.

[0048] According to embodiments of this application, during the collaborative data processing between the main thread and multiple slave threads, the main thread is responsible for managing and coordinating the work of the slave threads. Once a slave thread completes its task, it feeds back its processing status to the main thread based on the parameter sharing mechanism between threads. After receiving feedback from all slave threads, the main thread confirms that the processing task of the current network layer has been completed and can then proceed to the inference phase of the next network layer.

[0049] According to embodiments of this application, during the inference process of the main thread and slave threads, the generated data is transmitted via allreduce communication (global reduction communication). Once it is confirmed that both the main thread and slave threads in the network layer have completed data inference, the inferred data is passed to the next network layer via send-recv communication, so that the next network layer can continue inference processing. If it is determined that each network layer of the preset model has completed inference, the final inference result is output.

[0050] According to embodiments of this application, during model inference, a collaborative waiting mechanism between the main thread and slave threads in each network layer ensures the orderly execution of multiple network layer tasks with temporal dependencies, while also achieving efficient multidimensional parallel processing of the input network layer data. This mechanism not only guarantees the correctness of inference but also significantly improves overall performance. Furthermore, communication via a data-sharing mechanism between the main thread and slave threads enables rapid responses to the status feedback from each slave thread during data processing, further enhancing the efficiency of model inference.

[0051] According to an embodiment of this application, inference information is input into a preset model comprising multiple network layers, and inference processing is performed sequentially on the inference information according to the temporal dependencies between the multiple network layers. This includes: inputting inference information into the preset model, and performing inference processing sequentially on the inference information using multiple network layers according to the temporal dependencies; if it is determined that the inference result output by the target network layer in the preset model is obtained, the following operation is repeated until the number of iterations reaches a preset number: inputting the inference result output by the previous iteration task into the preset model, and performing inference processing sequentially on the inference result using multiple network layers according to the temporal dependencies to obtain the inference result output by the current iteration task.

[0052] According to embodiments of this application, during the large language model inference process, based on the temporal dependencies between multiple network layers, the overall model inference can be divided into two parts: the first inference and subsequent inferences. The overall temporal control of the model inference execution phase is implemented based on the inference characteristics of the KV-Cache model (Key-Value Cache). The length of the first inference is related to the input inference information, therefore the length of each first inference may be different. The length of each subsequent inference is 1, meaning that the output of the previous inference is used as input to continue the inference process.

[0053] According to embodiments of this application, the reasoning of the large language model is divided into two stages: the first reasoning (denoted as step 0) and subsequent reasoning (denoted as step N). When the number of iterations for the current task reaches a preset value, the reasoning of the large language model is stopped, and the reasoning result of the target network layer in the preset model is output. The target network layer is the last network layer in the preset model, i.e., the final step in the pipeline processing.

[0054] According to embodiments of this application, large language model inference is divided into first inference and subsequent inference, and timing control is performed based on a KV-Cache model, which significantly improves inference efficiency, reduces computational overhead, and optimizes memory usage. The first inference processes variable-length input, while subsequent inference has a fixed length of 1. The caching mechanism avoids redundant computation, making it suitable for long sequence generation tasks. This approach simplifies timing management and improves hardware resource utilization while ensuring correctness, making it suitable for high-efficiency large language model inference scenarios.

[0055] According to an embodiment of this application, inference information is input into a preset model to perform inference processing on the inference information sequentially using multiple network layers based on temporal dependencies. This includes: inputting the inference information into a first network layer in the preset model to perform multi-dimensional parallel processing on the inference information using the main thread and multiple slave threads in the first network layer, and outputting inference data; repeating the following operations according to temporal dependencies until the inference result output by the target network layer is obtained; inputting the inference data output by the previous network layer into the current network layer to perform multi-dimensional parallel processing on the inference data using the main thread and multiple slave threads in the current network layer, and obtaining the inference data output by the current network layer.

[0056] According to an embodiment of this application, in the first inference of the preset model, the inference information is input into the first network layer (i.e., the first network layer) of the preset model. The main thread and multiple slave threads in the first network layer perform multi-dimensional parallel processing on the inference information and output inference data. After the first network layer completes inference, the output inference data is passed to the next network layer according to the temporal dependencies between network layers. The next network layer only starts its inference thread to continue processing after receiving the inference data sent by the previous network layer. If the inference result output by the target network layer is obtained, the inference of the large language model is considered complete, and the inference result is output.

[0057] According to embodiments of this application, by inputting inference information into the first network layer of a preset model and utilizing a main thread and multiple slave threads for multi-dimensional parallel processing, inference efficiency can be significantly improved. By explicitly defining the temporal dependencies between network layers, data is ensured to be transmitted in an orderly manner between layers, reducing resource contention and data conflicts. The next network layer only starts its inference thread after receiving data from the previous network layer, guaranteeing the correctness and consistency of the inference process. This approach not only optimizes the utilization of computing resources but also improves overall inference performance, making it particularly suitable for pipelined processing of complex models.

[0058] like Figure 3 As shown, the inference process of the preset model includes operations S221 to S226.

[0059] In operation S221, the inference information is input into the first network layer of the preset model, and inference data is output. In operation S222, the inference data is input into the next network layer according to temporal dependencies, and inference data is output. In operation S223, it is determined whether the inference result output by the target network layer has been obtained. In operation S224, it is determined whether the preset number of iterations has been reached. In operation S225, the inference result is input into the first network layer of the preset model, and inference data is output. In operation S226, the inference result is output.

[0060] According to embodiments of this application, inference information is input into the first network layer of a preset model for processing. After generating inference data, it is passed to subsequent network layers for further processing based on temporal dependencies. Through multiple iterations and data transfer between network layers, the inference result is finally output at the target network layer. This process ensures the integrity and accuracy of the inference task while improving inference efficiency through the collaborative processing and iterative optimization of multiple network layers.

[0061] According to an embodiment of this application, the model inference method further includes: for each network layer, according to the model parameters of a preset model, splitting the data input to the network layer into multiple sub-data blocks, and correspondingly allocating them to the main thread and multiple slave threads; using the main thread and multiple slave threads to process the split sub-data blocks in parallel; merging the results obtained by the main thread and each slave thread to obtain the inference data output by the network layer.

[0062] According to embodiments of this application, input data is distributed to the main thread and slave threads based on the model parameters of a preset model. Specifically, the model parameters of each network layer can be divided into multiple sub-matrices. For example, if the matrix size of the model parameters is [M, N], it can be divided into k sub-matrices, each sub-matrix having a size of [M / k, N].

[0063] According to embodiments of this application, the data input to the network layer is split according to a matrix partitioning method; specifically, the data is divided into multiple sub-data blocks based on feature dimensions. Each slave thread is responsible for processing matrix multiplication operations between a sub-data block and its corresponding submatrix. After all slave threads have completed their calculations, the main thread concatenates the calculation results from each slave thread into complete output data, generating the inference result. Subsequently, the inference result is passed to the next network layer to ensure the correctness and consistency of the inference process. Through parallel computing and thread synchronization mechanisms, the model inference efficiency is significantly improved, making it particularly suitable for large language models and high-performance computing scenarios.

[0064] According to embodiments of this application, the model inference method further includes: creating a thread lock for each stage in multiple stages of the main thread processing data of the input network layer; enabling the thread lock when it is determined that the main thread and multiple slave threads are processing data in the current stage; releasing the thread lock for the current stage when the main thread receives stage status feedback from each of the multiple slave threads in the current stage, so that the main thread and multiple slave threads can proceed to the next stage of data processing; determining the release state of the thread lock when it is determined that new data of the input network layer has been obtained and the main thread and multiple slave threads have completed the processing of data in multiple stages; waiting for the thread lock to be released when it is determined that the thread lock is in the enabled state; and processing the obtained new data using the main thread and multiple slave threads when it is determined that the thread lock is in the released state.

[0065] According to embodiments of this application, since different threads may have different speeds during the loading and preprocessing stages of large language model weights (model parameters), a locking mechanism is used during weight loading and preprocessing to ensure that all threads complete their weight loading before continuing execution. This approach effectively avoids timing conflicts between threads and provides consistency guarantees for subsequent timing processing.

[0066] According to embodiments of this application, a thread lock is created for each of the multiple stages in which the main thread processes data from the input network layer. These multiple stages include a task initiation stage, a pre-task execution stage, a task preparation stage, and a post-task execution stage. The thread lock is activated when it is determined that the main thread and multiple slave threads are processing data in the current stage. Upon receiving stage status feedback from each of the multiple slave threads in the current stage, the thread lock for the current stage is released, allowing the main thread and multiple slave threads to proceed to the next stage of data processing.

[0067] According to an embodiment of this application, when a task is initiated, a thread lock is first created to ensure that the main thread waits for all slave threads to complete their states before continuing execution, thereby avoiding timing disorder when multiple tasks are submitted. Before task execution, it is checked whether there is any completed task information. If so, a thread lock is created, causing the main thread to wait for all slave threads to complete their states before continuing, and the lock is released after the slave threads complete their states.

[0068] According to embodiments of this application, during the task information construction and cache preparation phase, a thread lock is also created to ensure that the main thread waits for the slave threads to complete their states before proceeding to the next step, and releases the lock after the slave threads have completed their states. After the task execution is complete, a thread lock is created again, causing the main thread to wait for all slave threads to complete their states before ending the task, and releases the lock after the slave threads have completed their states. In this way, the correctness and consistency of the task execution sequence are ensured, while the timing of lock release is managed reasonably to avoid resource waste.

[0069] According to embodiments of this application, when processing data in the current input network layer, if new input data is received, the system first determines whether the current network layer is available, i.e., checks whether the thread lock has been released. If the thread lock has not been released, the system waits for the current network layer to complete processing of the existing data, and then, after the thread lock is released, it accesses the new data for processing. The next network layer then further processes the inference data output by the current network layer, ensuring that data is transmitted sequentially between network layers to avoid timing discrepancies.

[0070] According to embodiments of this application, by setting thread locks, not only can concurrent access from multiple requests be effectively managed, but resource contention and data conflicts can also be avoided, thereby improving the concurrency and stability of the inference process. Furthermore, the proper use of thread locks ensures the efficiency and correctness of model inference, enabling the preset model to handle multiple inference requests simultaneously, significantly improving system throughput and response speed.

[0071] According to an embodiment of this application, when it is determined that the main thread has received processing status feedback from multiple slave threads, the inference data output by the network layer is passed to the next network layer so that the next network layer can continue to perform inference processing. This includes: for each network layer, receiving the processing status feedback from each slave thread to the main thread based on the sub-communication domain corresponding to the network layer; determining the task execution status of the network layer based on multiple processing statuses; and when it is determined that the task execution status is completed, passing the inference data output by the network layer to the next network layer using the main communication domain associated with the preset model.

[0072] According to embodiments of this application, a sub-communication domain is created for each network layer to manage communication between slave threads and the main thread within that network layer. After completing its assigned task, each slave thread reports its processing status (e.g., "completed" or "error") to the main thread through the sub-communication domain. The main thread receives the processing statuses from all slave threads through the sub-communication domain. The main thread then aggregates these statuses to determine the task execution status of the network layer (e.g., "in execution," "completed," or "failed").

[0073] According to an embodiment of this application, if all slave threads are in the "complete" state, the task execution status of the network layer is determined to be "execution completed". If any slave thread is in the "error" state, the task execution status is determined to be "failed", and an error handling mechanism is triggered. The error handling mechanism can be configured to keep the current thread lock enabled and not release it temporarily. Multiple inferences are performed on the current data in the network layer (up to a preset number). If all inference results are "failed", processing of the data is stopped, and the current thread lock is released to allow processing of data from other inference requests. This approach ensures that erroneous data processing attempts are attempted while avoiding long-term resource occupation, thus improving system resource utilization and response efficiency.

[0074] According to embodiments of this application, data with a task execution status of "failed" can be marked as abnormal. When it is determined that the current network layer is idle and no new requests are waiting, the data marked as abnormal is allowed to be inferred again. For these abnormal data, the processing status of each slave thread can be further analyzed, and detailed information corresponding to the "failed" slave thread can be output so that relevant personnel can locate the problem and make modifications. This approach not only improves the fault tolerance of the system but also provides a clear direction for troubleshooting, while ensuring efficient use of resources.

[0075] According to embodiments of this application, when the task execution status is "execution completed," the main thread uses the main communication domain associated with the preset model to pass the inference data output by the current network layer to the next network layer. The main communication domain is responsible for ensuring reliable data transmission between network layers, avoiding data loss or conflicts. By utilizing sub-communication domains and the main communication domain for communication, the status feedback and data transmission of multi-threaded tasks can be effectively managed, improving the efficiency and stability of model inference.

[0076] According to an embodiment of this application, the model inference method further includes: creating network shared parameters corresponding to a preset model, so as to construct a main communication domain for communication between multiple network layers using the network shared parameters; for each network layer, determining the thread corresponding to the first sub-data block in the network layer as the main thread; and determining the multiple threads corresponding to the other multiple sub-data blocks besides the first sub-data block as slave threads; and creating thread shared parameters based on the main thread, so as to construct a sub-communication domain for communication between the main thread and multiple slave threads using the thread shared parameters.

[0077] According to embodiments of this application, such as Figure 4 As shown, based on the parallel characteristics of the model, assuming the preset model includes two network layers, namely the first network layer 410 and the second network layer 420. Each network layer includes two tensors, then four threads are used. Figure 4 As shown, the four threads in the preset model are numbered 0, 1, 2, and 3. A network shared parameter is created as the ID information for the main communication domain, and each thread has the same parameter information. The main communication domain is created using the network shared parameter. This main communication domain contains four threads, corresponding to thread information [0, 1, 2, 3], and is used for communication between network layers.

[0078] According to embodiments of this application, the thread corresponding to the first sub-data block (i.e., the first sub-data block) of each network layer is determined as the main thread, and the threads corresponding to other sub-data blocks in the same network layer are determined as slave threads. For example... Figure 4 As shown, for the first network layer 410, the thread numbered 0 in the first network layer 410 is determined as the master thread 411, and the thread numbered 1 is determined as the slave thread 412.

[0079] According to an embodiment of this application, when creating a sub-communication domain, a thread-shared parameter is created using the main thread, and the thread-shared parameter is accessed by assigning values ​​to other threads in the same sub-communication domain. The thread-shared parameter is used to create the resource communication domain. Each sub-communication domain contains two threads, corresponding to thread information [0, 1] and [2, 3], used for communication between feature vectors.

[0080] According to embodiments of this application, an inter-thread parameter sharing mechanism is employed to quickly create multiple communication domains by efficiently passing unique identifiers, thereby further improving communication efficiency. When handling timing issues between parallel and serial tasks, a cooperative waiting mechanism between the main thread and slave threads ensures both the orderly execution of serial tasks and the efficient processing of parallel tasks. This approach significantly improves overall performance while ensuring task correctness.

[0081] Based on the above-described model reasoning method, this application also provides a model reasoning apparatus. The following will combine... Figure 5 The device is described in detail.

[0082] Figure 5 A schematic block diagram of a model inference apparatus according to an embodiment of this application is shown.

[0083] like Figure 5 As shown, the model inference device 500 of this embodiment includes an information acquisition module 510, an inference processing module 520, a data transmission module 530, and a result output module 540.

[0084] The information acquisition module 510 is used to acquire reasoning information. In one embodiment, the information acquisition module 510 can be used to perform the operation S210 described above, which will not be repeated here.

[0085] The inference processing module 520 is used to input inference information into a preset model comprising multiple network layers, and to perform inference processing on the inference information sequentially according to the temporal dependencies between the multiple network layers. Each of the multiple network layers includes a main thread and multiple slave threads, which are used to perform multi-dimensional parallel processing on the data input to the network layers. In one embodiment, the inference processing module 520 can be used to execute the operation S220 described above, which will not be repeated here.

[0086] The data transmission module 530, for each network layer, after determining that the main thread has received processing status feedback from multiple slave threads, transmits the inference data output by the network layer to the next network layer, so that the next network layer can continue inference processing. In one embodiment, the data transmission module 530 can be used to perform the operation S230 described above, which will not be repeated here.

[0087] The result output module 540 is used to output the inference result corresponding to the inference information when it is determined that multiple network layers have completed the inference process. In one embodiment, the result output module 540 can be used to perform the operation S240 described above, which will not be repeated here.

[0088] According to an embodiment of this application, the reasoning processing module 520 includes an information reasoning submodule and an iterative reasoning submodule.

[0089] The information reasoning submodule is used to input reasoning information into a preset model, and to process the reasoning information sequentially using multiple network layers according to temporal dependencies.

[0090] The iterative inference submodule is used to repeatedly perform the following operation until the number of iterations reaches the preset number, provided that the inference result output by the target network layer in the preset model is obtained: input the inference result output by the previous iteration task into the preset model, and use multiple network layers to perform inference processing on the inference result in sequence according to the temporal dependency relationship to obtain the inference result output by the current iteration task.

[0091] According to an embodiment of this application, the information reasoning submodule includes a first reasoning unit and an iterative reasoning unit.

[0092] The first inference unit is used to input inference information into the first network layer in the preset model, so as to use the main thread and multiple slave threads in the first network layer to perform multi-dimensional parallel processing on the inference information and output inference data.

[0093] The iterative inference unit is used to repeatedly perform the following operations according to the temporal dependency until the inference result output by the target network layer is obtained: input the inference data output by the previous network layer into the current network layer, so as to use the main thread and multiple slave threads in the current network layer to perform multi-dimensional parallel processing on the inference data and obtain the inference data output by the current network layer.

[0094] According to embodiments of this application, the model inference device 500 further includes a data splitting module, a data processing module, and a result merging module.

[0095] The data splitting module is used to split the input data of each network layer into multiple sub-data blocks according to the model parameters of the preset model, and then allocate them to the main thread and the multiple slave threads mentioned above.

[0096] The data processing module is used to process the split sub-data blocks in parallel using the main thread and multiple slave threads.

[0097] The result merging module is used to merge the results processed by the main thread and each slave thread to obtain the inference data output by the network layer.

[0098] According to embodiments of this application, the model inference device 500 further includes a thread lock creation module, a thread lock enabling module, a thread lock release module, a state determination module, a wait-to-release module, and a new data processing module.

[0099] The thread lock creation module is used to create thread locks for each stage in the multiple stages of the main thread's processing of the input network layer data.

[0100] The thread lock enabling module is used to enable thread locks when it is determined that the main thread and multiple slave threads are processing data at the current stage.

[0101] The thread lock release module is used to release the thread lock of the current stage when the main thread receives the stage status feedback from multiple slave threads in the current stage, so that the main thread and multiple slave threads can proceed to the next stage of data processing.

[0102] The state determination module is used to determine the release state of the thread lock when it is determined that new data has been obtained from the input network layer and the main thread and multiple slave threads have completed the processing of the data in multiple stages.

[0103] The wait-to-release module is used to wait for the thread lock to be released when it is determined that the thread lock is in an enabled state.

[0104] The new data processing module is used to process newly acquired data using the main thread and multiple slave threads, provided that the thread lock is determined to be in a released state.

[0105] According to an embodiment of this application, the data transmission module 530 includes a status receiving submodule, a status determining submodule, and a data transmission submodule.

[0106] The status receiving submodule is used to receive the processing status fed back from each slave thread to the main thread for each network layer, based on the sub-communication domain corresponding to the network layer.

[0107] The state determination submodule is used to determine the task execution state of the network layer based on multiple processing states.

[0108] The data transfer submodule is used to transfer the inference data output by the network layer to the next network layer by utilizing the main communication domain associated with the preset model when the task execution status is determined to be completed.

[0109] According to embodiments of this application, the model inference device 500 further includes a parameter creation module, a main thread determination module, a slave thread determination module, and a communication construction module.

[0110] The parameter creation module is used to create network shared parameters corresponding to the preset model, so as to build a main communication domain for communication between multiple network layers using the network shared parameters.

[0111] The main thread determination module is used to determine the thread corresponding to the first sub-data block in each network layer as the main thread.

[0112] The thread determination module is used to determine the multiple threads corresponding to multiple sub-data blocks other than the first sub-data block as slave threads.

[0113] The communication building module is used to create thread-shared parameters based on the main thread, and to build sub-communication domains for communication between the main thread and multiple child threads using the thread-shared parameters.

[0114] According to embodiments of this application, any multiple modules among the information acquisition module 510, inference processing module 520, data transmission module 530, and result output module 540 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of this application, at least one of the information acquisition module 510, inference processing module 520, data transmission module 530, and result output module 540 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the information acquisition module 510, reasoning processing module 520, data transmission module 530, and result output module 540 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0115] Figure 6 A block diagram schematically illustrates an electronic device suitable for implementing a model inference method according to an embodiment of this application.

[0116] like Figure 6 As shown, an electronic device 600 according to an embodiment of this application includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage portion 608 into a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this application.

[0117] RAM 603 stores various programs and data required for the operation of electronic device 600. Processor 601, ROM 602, and RAM 603 are interconnected via bus 604. Processor 601 executes various operations of the method flow according to embodiments of this application by executing programs in ROM 602 and / or RAM 603. It should be noted that programs may also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 may also execute various operations of the method flow according to embodiments of this application by executing programs stored in one or more memories.

[0118] According to embodiments of this application, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to a bus 604. The electronic device 600 may also include one or more of the following components connected to the input / output (I / O) interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output (I / O) interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 610 as needed so that computer programs read from it can be installed into the storage section 608 as needed.

[0119] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0120] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this application, the computer-readable storage medium may include ROM 602 and / or RAM 603 and / or one or more memories other than ROM 602 and RAM 603 described above.

[0121] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code enables the computer system to implement the model inference method provided in the embodiments of this application.

[0122] When the computer program is executed by the processor 601, it performs the functions defined in the system / apparatus of this application embodiment. According to the embodiments of this application, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0123] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 609, and / or installed from the removable medium 611. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0124] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, it performs the functions defined in the system of this application embodiment. According to the embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0125] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0126] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0127] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

[0128] The embodiments of this application have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of this application. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Without departing from the scope of this application, those skilled in the art can make various substitutions and modifications, all of which should fall within the scope of this application.

Claims

1. A model reasoning method, characterized in that, The method includes: Obtain reasoning information; The inference information is input into a preset model comprising multiple network layers, and the inference information is sequentially processed according to the temporal dependencies between the multiple network layers. Each of the multiple network layers includes a main thread and multiple slave threads. The main thread and multiple slave threads are used to perform multi-dimensional parallel processing on the data input to the network layer. The main thread is also used to manage and coordinate the work of the slave threads. For each network layer, the data input to the network layer is divided into multiple sub-data blocks according to the model parameters of the preset model, and then allocated to the main thread and multiple slave threads accordingly. When the main thread receives the processing status feedback from each of the multiple slave threads through the sub-communication domain, and determines that the task execution status of the network layer is completed based on the multiple processing statuses, the inference data output by the network layer is passed to the next network layer through the main communication domain so that the next network layer can continue to perform inference processing. If it is determined that all of the network layers have completed the inference process, the inference result corresponding to the inference information is output. The method further includes: For each network layer, the thread corresponding to the first sub-data block in the network layer is determined as the main thread; and the threads corresponding to the other sub-data blocks besides the first sub-data block are determined as slave threads. If the task execution status is "execution failed", it is marked as "abnormal". When it is determined that the current network layer is in an idle state and no new requests are waiting, the data marked as "abnormal" is re-inferred. For the abnormal data, the processing status of each slave thread is analyzed and the detailed information corresponding to the failed slave thread is output.

2. The method according to claim 1, characterized in that, The step of inputting the inference information into a preset model comprising multiple network layers, and sequentially processing the inference information according to the temporal dependencies between the multiple network layers, includes: The inference information is input into the preset model so that, according to the temporal dependency, the inference information is processed sequentially using multiple network layers. If the inference result output by the target network layer in the preset model is obtained, repeat the following operation until the preset number of iterations is reached: The inference result output from the previous iteration task is input into the preset model, and the inference result is processed sequentially by multiple network layers according to the temporal dependency relationship to obtain the inference result output from the current iteration task.

3. The method according to claim 2, characterized in that, The step of inputting the inference information into the preset model, and then sequentially processing the inference information using multiple network layers according to the temporal dependency, includes: The inference information is input into the first network layer of the preset model, so as to utilize the main thread and multiple slave threads in the first network layer to perform multi-dimensional parallel processing on the inference information and output inference data. Based on the temporal dependencies, repeat the following operations until the inference result output by the target network layer is obtained: The inference data output from the previous network layer is input into the current network layer, so that the main thread and multiple slave threads in the current network layer can be used to perform multi-dimensional parallel processing on the inference data to obtain the inference data output by the current network layer.

4. The method according to any one of claims 1 to 3, characterized in that, The method further includes: For each of the network layers, the main thread and multiple slave threads are used to process the split sub-data blocks in parallel. The results obtained from the main thread and each of the slave threads are merged to obtain the inference data output by the network layer.

5. The method according to claim 4, characterized in that, The method further includes: In the multiple stages of the main thread processing the data input to the network layer, a thread lock is created for each stage; If it is determined that the main thread and the multiple slave threads are processing the data at the current stage, the thread lock is enabled; When the main thread receives the stage status feedback from each of the multiple slave threads in the current stage, it releases the thread lock for the current stage, so that the main thread and the multiple slave threads can proceed with the data processing for the next stage. When it is determined that new data has been obtained for input to the network layer, and the main thread and the multiple slave threads have completed the processing of the data in multiple stages, the release state of the thread lock is determined; If it is determined that the thread lock is in an enabled state, wait for the thread lock to be released; If it is determined that the thread lock is in a released state, the new data obtained is processed using the main thread and multiple slave threads.

6. The method according to claim 1, characterized in that, The step of passing the inference data output by the network layer to the next network layer after determining that the main thread has received the processing status feedback from each of the multiple slave threads, so that the next network layer can continue to perform inference processing, includes: For each network layer, based on the sub-communication domain corresponding to the network layer, the processing status fed back from each slave thread to the main thread is received; Based on the multiple processing states, the task execution state of the network layer is determined; If the task execution status is determined to be completed, the inference data output by the network layer is transmitted to the next network layer using the main communication domain associated with the preset model.

7. The method according to claim 6, characterized in that, The method further includes: Create network shared parameters corresponding to the preset model, and use the network shared parameters to construct a main communication domain for communication between multiple network layers; A thread-shared parameter is created based on the main thread to construct a sub-communication domain for communication between the main thread and multiple slave threads.

8. A model reasoning device, characterized in that, The device includes: The information acquisition module is used to acquire reasoning information; The inference processing module is used to respond to the received model inference request, input the acquired inference information into a preset model composed of multiple network layers, and perform inference processing on the inference information sequentially according to the temporal dependencies between the multiple network layers. The network layer includes a main thread and multiple slave threads. The main thread and multiple slave threads are used to perform multi-dimensional parallel processing on the data input to the network layer. The main thread is also used to manage and coordinate the work of the slave threads. The data transmission module is used to, for each of the network layers, transmit the inference data output by the network layer to the next network layer through the main communication domain when the main thread receives the processing status feedback from the multiple slave threads through the sub-communication domain, and the task execution status of the network layer is determined to be completed based on the multiple processing statuses, so that the next network layer can continue to perform inference processing. The result output module is used to output the inference result corresponding to the inference information when it is determined that all of the multiple network layers have completed the inference process; The device is further configured to, according to the model parameters of the preset model, split the data input to the network layer into multiple sub-data blocks, and allocate them accordingly to the main thread and the multiple slave threads; For each network layer, the thread corresponding to the first sub-data block in the network layer is determined as the main thread; and the threads corresponding to the other sub-data blocks besides the first sub-data block are determined as slave threads. If the task execution status is "execution failed", it is marked as "abnormal". When it is determined that the current network layer is in an idle state and no new requests are waiting, the data marked as "abnormal" is re-inferred. For the abnormal data, the processing status of each slave thread is analyzed and the detailed information corresponding to the failed slave thread is output.

9. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 7.

Images