Request processing method and device, readable medium, electronic equipment and program product

Abstract

The invention relates to a request processing method and device, a readable medium, electronic equipment and a program product, and the method comprises the steps: storing a reasoning request which carries to-be-reasoned data inputted by a user into a request queue after the reasoning request for a multi-modal machine learning model is received, the to-be-reasoned data comprises image data and / or audio data; when a preset parallel processing condition is met, feature processing is carried out on the image data and / or the audio data carried by the reasoning request in the request queue in parallel, target feature data are obtained, and the target feature data are used for model reasoning of the multi-mode machine learning model. Therefore, the data carried by the reasoning request can be concurrently processed in a batch processing mode, the feature processing time of the data is shortened, the data processing efficiency is effectively improved, and then the service throughput of the multi-mode machine learning model is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular, to a request processing method, device, readable medium, electronic device and program product. Background Art

[0002] Multimodal machine learning models can use machine learning models to perform multimodal tasks. In the multimodal machine learning model inference service, it is necessary to first perform feature processing on the data carried by the inference request, and then perform model inference. If there are many inference requests, the carried data will take a lot of time to perform feature processing, and the data processing efficiency is low, which in turn leads to low service throughput of the multimodal machine learning model. Summary of the invention

[0003] This summary is provided to introduce concepts in a brief form that will be described in detail in the detailed description below. This summary is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0004] In a first aspect, the present disclosure provides a request processing method, the method comprising: After receiving an inference request for a multimodal machine learning model, storing the inference request in a request queue, wherein the inference request carries data to be inferred input by a user, and the data to be inferred includes image data and / or audio data; When the preset parallel processing conditions are met, feature processing is performed in parallel on the image data and / or audio data carried by the inference request in the request queue to obtain target feature data, and the target feature data is used for model inference by the multimodal machine learning model.

[0005] In a second aspect, the present disclosure provides a request processing device, the device comprising: A receiving module, configured to store an inference request for a multimodal machine learning model into a request queue after receiving the inference request, wherein the inference request carries data to be inferred input by a user, and the data to be inferred includes image data and / or audio data; A processing module is used to perform feature processing on the image data and / or audio data carried by the inference request in the request queue in parallel when a preset parallel processing condition is met to obtain target feature data, and the target feature data is used for model inference by the multimodal machine learning model.

[0006] In a third aspect, the present disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of any one of the methods described in the first aspect.

[0007] In a fourth aspect, the present disclosure provides an electronic device, including: a storage device having a computer program stored thereon; A processing device is used to execute the computer program in the storage device to implement the steps of any method described in the first aspect above.

[0008] In a fifth aspect, the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of any one of the methods in the first aspect.

[0009] Through the above technical solution, after receiving the inference request for the multimodal machine learning model, the inference request is first stored in the request queue, and then when the preset parallel processing conditions are met, the image data and / or audio data carried by the inference request in the request queue are processed in parallel to obtain the target feature data. Therefore, the data carried by the inference request can be processed concurrently in batch processing, which shortens the feature processing time of the data, effectively improves the data processing efficiency, and thus improves the service throughput of the multimodal machine learning model.

[0010] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale. In the drawings: Figure 1 is a flowchart of a request processing method according to an exemplary embodiment of the present disclosure; Figure 2 is a question-answering schematic diagram of a question-answering model according to an exemplary embodiment of the present disclosure; Figure 3 is a process diagram of a request processing method according to an exemplary embodiment of the present disclosure; Figure 4 is a block diagram of a request processing device according to an exemplary embodiment of the present disclosure; Figure 5 It is a schematic structural diagram of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0012] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0013] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0014] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0015] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0016] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0017] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0018] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0019] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0020] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0021] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0022] At the same time, it is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0023] Multimodal machine learning models usually include two main GPU (Graphics Processing Unit) computing modules: feature processing module and inference computing module. The feature processing module performs feature processing on the data carried by the inference request in sequence according to the order in which the inference request is received. If there are many inference requests, the processing time of the feature processing module will increase linearly, that is, it will take a lot of time to perform feature processing on the data, and the data processing efficiency is low, which will lead to low service throughput of the multimodal machine learning model.

[0024] In view of this, the present disclosure provides a request processing method, device, readable medium, electronic device and program product to solve the above technical problems.

[0025] The embodiments of the present disclosure are further explained below with reference to the accompanying drawings.

[0026] Figure 1 A request processing method is shown according to an exemplary embodiment of the present disclosure, referring to Figure 1 , the request processing method includes: S101: After receiving an inference request for a multimodal machine learning model, the inference request is stored in a request queue.

[0027] The inference request carries the data to be inferred input by the user, and the data to be inferred includes image data and / or audio data.

[0028] It is worth noting that the multimodal machine learning model can be a multimodal large language model (MLLM) or other models that can perform multimodal tasks, such as a vision language large model (VLLM), etc., and the present disclosure does not impose any restrictions on this.

[0029] For example, in practical applications, the multimodal machine learning model can be a content generation model, a content retrieval model, a question-answering model, etc., and the present disclosure does not limit this. Taking the question-answering model as an example, Figure 2 As shown, the user can input image data and text questions on the smart object page, and the question-answering model performs feature processing on the image data, and then obtains the answer corresponding to the question through model reasoning and displays it to the user. Alternatively, the user can also input audio data and text questions on the smart object page, for example, requesting the model to analyze audio data. Of course, image data and audio data can also be input at the same time, and the present disclosure does not limit this.

[0030] Similarly, the content generation model can generate more content in response to the user's input content, for example, generating an image similar to the input image based on the input image data, and the content retrieval model can retrieve related content in response to the user's input content, and so on. The present disclosure does not limit this.

[0031] S102: When preset parallel processing conditions are met, feature processing is performed in parallel on the image data and / or audio data carried by the inference request in the request queue to obtain target feature data, and the target feature data is used for model inference in the multimodal machine learning model.

[0032] It should be understood that if the inference request includes image data, the target feature data includes image features, and if the inference request includes audio data, the target feature data includes audio features.

[0033] For example, when the parallel processing conditions are met, the image data and / or audio data carried by the inference request in the request queue are combined into a batch for processing, that is, parallel feature processing is achieved in a batch manner, thereby improving the efficiency of data feature processing.

[0034] With the above method, after receiving the inference request for the multimodal machine learning model, the inference request is first stored in the request queue, and then when the preset parallel processing conditions are met, the image data and / or audio data carried by the inference request in the request queue are processed in parallel to obtain the target feature data. In this way, the data carried by the inference request can be processed concurrently in batch processing, which shortens the feature processing time of the data, effectively improves the data processing efficiency, and thus improves the service throughput of the multimodal machine learning model.

[0035] like Figure 3 As shown, the entire multimodal machine learning model system may include a request receiver, a request manager, a feature processor, and a model processor. First, the client generates an inference request in response to the data to be inferred input by the user, and sends the inference request to the server, wherein the server can receive multiple inference requests generated by different clients, or multiple inference requests generated by the same client. Then, after the server receives the inference request through the request receiver, it stores it in the request queue in the request manager. The request manager then determines whether the preset parallel processing conditions are met. If not, parallel processing is temporarily not performed, and requests are continuously received. If it is met, batch parallel feature processing is performed on the data carried by the inference request in the request queue through the feature processor, and the obtained feature data is sent to the model processor for model inference, and finally the obtained model inference result is returned to the client.

[0036] Therefore, by introducing a request manager, batch parallel feature processing of image data and / or audio data carried by inference requests can be achieved, thereby improving the efficiency of data feature processing and thereby increasing the throughput of the model inference service.

[0037] In a possible manner, the preset parallel processing conditions include at least one of the following: the duration from the last parallel feature processing is greater than a preset duration threshold; the number of image data and / or audio data to be inferred in the request queue reaches a preset maximum number threshold.

[0038] For example, a time window, i.e., a preset time threshold, may be set to periodically perform feature processing on the image data and / or audio data carried by the inference requests in the request queue. Alternatively, when the number of image data and / or audio data carried by the inference requests in the request queue reaches a maximum number threshold, feature processing is performed on the image data and / or audio data carried by the inference requests in the request queue.

[0039] In this way, by setting a preset time threshold, the image data and / or audio data carried by the inference request in the request queue will wait for a preset time at most, thereby avoiding the user waiting too long for the model inference result, and improving the user experience. By setting a maximum number threshold, it is possible to avoid processing a large amount of data at a time, which leads to a long feature processing time, and thus avoid the user waiting too long for the model inference result, and improve the user experience.

[0040] It should be noted that the preset duration threshold and the maximum number threshold can be set according to the requirements, and the present disclosure does not limit this. In addition, the preset duration threshold and the maximum number threshold can also be adaptively adjusted in the actual application process.

[0041] In a possible manner, the method also includes: obtaining historical delay duration and / or historical quantity corresponding to historical reasoning requests, the historical delay duration is the difference between the expected processing duration and the actual processing duration of the historical reasoning requests, and the historical quantity is the quantity of image data and / or audio data carried by the historical reasoning requests; adjusting the maximum quantity threshold based on the historical delay duration and / or historical quantity, wherein the maximum quantity threshold is negatively correlated with the historical delay duration, and the maximum quantity threshold is positively correlated with the historical quantity.

[0042] For example, the expected processing time and actual processing time of historical reasoning requests can be obtained. The expected processing time can be set by the user or fed back by the user, and the present disclosure does not limit this. The historical delay time is then determined based on the difference between the expected processing time and the actual processing time of the historical reasoning request. If the historical delay time indicates that the request processing delay is large, it means that the user has been waiting for too long for the model reasoning results, and the maximum number threshold can be appropriately reduced. This can reduce the waiting time for triggering a parallel feature processing, and can also reduce the amount of data for a parallel feature processing, thereby reducing the data processing time, thereby reducing the time required to feedback the model reasoning results to the user, thereby improving the user experience.

[0043] For example, the historical number corresponding to the historical reasoning requests can be obtained. If the historical number corresponding to the historical reasoning requests is large, for example, the number of image data and / or audio data that require feature processing within the historical unit time is small, if the maximum number threshold is set large, it will take a long time to trigger a parallel feature processing, which will cause the user to wait too long for the model inference results. In this case, the maximum number threshold can be appropriately increased to shorten the waiting time for triggering a parallel feature processing and reduce the time the user waits for the model inference results, thereby improving the user experience.

[0044] The specific adjustment can be set according to actual needs, and the historical delay duration and historical quantity can be obtained at the same time to adjust the maximum number threshold, which is not limited by the present disclosure. It is worth noting that the historical delay duration and historical quantity corresponding to a period of historical requests can be set to adjust the maximum number threshold, such as within 1 hour, within 24 hours, etc., and the maximum number threshold can also be set to be automatically adjusted periodically, such as once a day, once a week, and the present disclosure does not limit this.

[0045] It should be noted that a request quantity threshold can also be set, so that when the number of requests in the request queue reaches the request quantity threshold, parallel feature processing is triggered, so that the target data of the same inference request can be processed in parallel in the same batch. The specific setting can be based on demand, and this disclosure does not impose any restrictions on this.

[0046] In a possible manner, the method also includes: obtaining a historical delay duration corresponding to a historical reasoning request, the historical delay duration being the difference between an expected processing duration and an actual processing duration of the historical reasoning request; and adjusting a preset duration threshold based on the historical delay duration, wherein the preset duration threshold is negatively correlated with the historical delay duration.

[0047] For example, if the historical delay duration indicates that the request processing delay is large, it means that the user has been waiting for too long for the model inference result, and the preset duration threshold can be appropriately reduced. This can reduce the waiting time for triggering a parallel feature processing, thereby reducing the time required to feedback the model inference result to the user, thereby improving the user experience. The specific adjustment can be set according to actual needs, and this disclosure does not limit this.

[0048] Similarly, the historical delay duration corresponding to historical requests over a period of time can be set to adjust the preset duration threshold, for example, within 1 hour, within 24 hours, and so on. The preset duration threshold can also be set to be automatically adjusted periodically, for example, once a day, once a week, and the present disclosure does not impose any restrictions on this.

[0049] In a possible manner, feature processing is performed on image data and / or audio data carried by inference requests in a request queue in parallel, including: when the number of target data to be inferred in the request queue is less than or equal to a preset maximum number threshold, all target data in the request queue are obtained, and feature processing is performed on all target data in parallel, wherein the target data is image data and / or audio data; when the number of target data to be inferred in the request queue is greater than the maximum number threshold, the maximum number threshold of first target data is obtained from the target data, and feature processing is performed on the first target data in parallel, wherein the first target data is stored in the request queue earlier than the second target data in the request queue except the first target data.

[0050] It should be noted that when the preset parallel processing conditions are met, parallel feature processing will be triggered. However, since the server will continue to receive inference requests and store them in the request queue, when parallel feature processing is performed, the number of target data to be inferred in the request queue may exceed the preset maximum number threshold.

[0051] For example, if the number of target data to be inferred in the request queue is less than or equal to the maximum number threshold, all target data in the request queue can be directly obtained, and then all target data can be processed in parallel. If the number of target data to be inferred in the request queue is greater than the maximum number threshold, the maximum number threshold of target data corresponding to the inference request stored earlier can be selected for parallel feature processing according to the storage time of the inference request in the request queue.

[0052] It should be noted that if the target data of the same inference request needs to be processed in parallel in the same batch, a request quantity threshold can also be set, or when obtaining the target data corresponding to the inference request, the total target data carried by the selected inference request does not exceed the maximum quantity threshold. For example, the maximum quantity threshold is 10, request 1 carries 4 target data, request 2 carries 5 target data, and request 3 carries 2 target data, then the target data carried by request 1 and request 2 are selected for parallel feature processing. The specific setting can be based on demand, and the present disclosure does not impose any restrictions on this.

[0053] In a possible manner, feature processing is performed in parallel on image data and / or audio data carried by inference requests in a request queue to obtain target feature data, including: acquiring image data and / or audio data carried by inference requests in the request queue in parallel to obtain first data; preprocessing the first data in parallel to obtain second data, wherein the preprocessing includes normalization processing; and feature extraction is performed in parallel on the second data to obtain target feature data.

[0054] For example, when the preset parallel processing conditions are met, feature processing can be performed in parallel on the image data and / or audio data carried by the inference request in the request queue. For example, when the inference request carries the storage path of the image data and / or audio data, the data can be downloaded first, and the downloaded data can also be preprocessed such as normalization, and feature extraction can be performed on the data. The specific setting can be based on the actual feature processing flow, and the present disclosure does not limit this. By performing feature processing on the image data and / or audio data carried by the inference request in the request queue in parallel, the processing time of feature processing can be shortened from O(N) to O(1), effectively improving the efficiency of data feature processing.

[0055] In a possible manner, each feature data in the target feature data carries a request identifier of a corresponding reasoning request. The method also includes: performing feature splicing according to the request identifier carried by each feature data in the target feature data to obtain feature data corresponding to each reasoning request, and the multimodal machine learning model is used to perform model reasoning according to the feature data corresponding to each reasoning request.

[0056] For example, Figure 3 As shown, after obtaining the target feature data, feature splicing is performed according to the request identifier carried by each feature data in the target feature data to obtain the feature data corresponding to each reasoning request, and then the feature data corresponding to each reasoning request is sent to the model processor, so that the multimodal machine learning model can perform model reasoning according to the feature data corresponding to each reasoning request.

[0057] For example, when an inference request is stored in a request queue, a request identifier corresponding to the inference request can be generated, and then the corresponding request identifier can be marked on the image data and / or audio data carried by the inference request. Correspondingly, the corresponding request identifier is also marked on the corresponding target feature data, thereby distinguishing the inference requests corresponding to different target feature data, thereby facilitating feature splicing according to the inference request.

[0058] By way of example, a mapping relationship may also be established between an inference request and the image data and / or audio data carried, and then the image data and / or audio data before feature processing may be determined through the target feature data, and then the corresponding inference request may be determined based on the mapping table. The present disclosure does not impose any restrictions on this.

[0059] It is worth noting that multimodal machine learning models can be processed in batches through continuous batching, that is, batch parallel model reasoning. This can avoid the situation when model reasoning requires waiting for the previous reasoning request to complete model reasoning and release resources before continuing the model reasoning of the next reasoning request because the number of feature data corresponding to different reasoning requests in the batch is different. By combining batch parallel feature processing and batch parallel model reasoning, the service throughput of model reasoning can be effectively improved.

[0060] Based on the same concept, the embodiment of the present disclosure also provides a request processing, such as Figure 4 As shown, the request processing device 400 may include: A receiving module 401 is used to store an inference request for a multimodal machine learning model into a request queue after receiving the inference request, wherein the inference request carries data to be inferred input by a user, and the data to be inferred includes image data and / or audio data; The processing module 402 is used to perform feature processing on the image data and / or audio data carried by the inference request in the request queue in parallel when the preset parallel processing conditions are met to obtain target feature data, and the target feature data is used for model inference by the multimodal machine learning model.

[0061] Optionally, the processing module 402 is used to: When the amount of target data to be inferred in the request queue is less than or equal to a preset maximum amount threshold, all the target data in the request queue are acquired, and feature processing is performed on all the target data in parallel, wherein the target data is image data and / or audio data; When the number of the target data to be inferred in the request queue is greater than the maximum number threshold, the first target data of the maximum number threshold are obtained from the target data, and feature processing is performed on the first target data in parallel, wherein the first target data is stored in the request queue earlier than the second target data in the request queue except the first target data.

[0062] Optionally, the preset parallel processing condition includes at least one of the following: The time since the last parallel feature processing is greater than the preset time threshold; The amount of image data and / or audio data to be inferred in the request queue reaches a preset maximum amount threshold.

[0063] Optionally, the request processing device 400 may further include a first adjustment module, wherein the first adjustment module is configured to: Obtaining a historical delay duration and / or a historical quantity corresponding to a historical reasoning request, wherein the historical delay duration is a difference between an expected processing duration and an actual processing duration of the historical reasoning request, and the historical quantity is a quantity of image data and / or audio data carried by the historical reasoning request; The maximum number threshold is adjusted based on the historical delay duration and / or the historical quantity, wherein the maximum number threshold is negatively correlated with the historical delay duration, and the maximum number threshold is positively correlated with the historical data volume.

[0064] Optionally, the request processing device 400 may further include a second adjustment module, wherein the second adjustment module is configured to: Obtaining a historical delay duration corresponding to a historical reasoning request, where the historical delay duration is a difference between an expected processing duration and an actual processing duration of the historical reasoning request; The preset duration threshold is adjusted based on the historical delay duration, wherein the preset duration threshold is negatively correlated with the historical delay duration.

[0065] Optionally, each feature data in the target feature data carries a request identifier of a corresponding reasoning request, and the request processing device 400 may further include a splicing module, which is used to: Feature splicing is performed according to the request identifier carried by each feature data in the target feature data to obtain feature data corresponding to each reasoning request, and the multimodal machine learning model is used to perform model reasoning according to the feature data corresponding to each reasoning request.

[0066] Optionally, the processing module 402 is used to: Acquire image data and / or audio data carried by the inference requests in the request queue in parallel to obtain first data; Preprocessing the first data in parallel to obtain second data, wherein the preprocessing includes normalization processing; In parallel, feature extraction is performed on the second data to obtain the target feature data.

[0067] Based on the same concept, an embodiment of the present disclosure further provides a computer-readable medium on which a computer program is stored, and when the program is executed by a processing device, the steps of any of the above request processing methods are implemented.

[0068] Based on the same concept, the present disclosure also provides an electronic device, which may include: a storage device having a computer program stored thereon; The processing device is used to execute the computer program in the storage device to implement the steps of any of the above request processing methods.

[0069] Based on the same concept, an embodiment of the present disclosure further provides a computer program product, including a computer program, which implements the steps of any of the above request processing methods when executed by a processor.

[0070] Reference below Figure 5 , which shows a schematic diagram of the structure of an electronic device 500 suitable for implementing the embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0071] like Figure 5As shown, the electronic device 500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 to a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device 500 are also stored. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0072] Typically, the following devices may be connected to the I / O interface 505: an input device 506 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 508 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 509. The communication device 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 5 The electronic device 500 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0073] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0074] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0075] In some embodiments, any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol) can be used for communication, and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0076] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0077] The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the electronic device, the electronic device: after receiving an inference request for a multimodal machine learning model, stores the inference request in a request queue, wherein the inference request carries the data to be inferred input by the user, and the data to be inferred includes image data and / or audio data; when the preset parallel processing conditions are met, feature processing is performed in parallel on the image data and / or audio data carried by the inference request in the request queue to obtain target feature data, and the target feature data is used for model inference of the multimodal machine learning model.

[0078] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0079] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0080] The modules involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a module does not, in some cases, limit the module itself.

[0081] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that may be used include, without limitation, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0082] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0083] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0084] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0085] Although the subject matter has been described in language specific to structural features and / or method logic actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims. Regarding the device in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.

Claims

1. A request processing method, characterized in that: The method comprises: After receiving an inference request for a multimodal machine learning model, storing the inference request in a request queue, wherein the inference request carries data to be inferred input by a user, and the data to be inferred includes image data and / or audio data; When the preset parallel processing conditions are met, feature processing is performed in parallel on the image data and / or audio data carried by the inference request in the request queue to obtain target feature data, and the target feature data is used for model inference by the multimodal machine learning model.

2. The request processing method according to claim 1, characterized in that: The parallel processing of the image data and / or audio data carried by the inference request in the request queue includes: When the amount of target data to be inferred in the request queue is less than or equal to a preset maximum amount threshold, all the target data in the request queue are acquired, and feature processing is performed on all the target data in parallel, wherein the target data is image data and / or audio data; When the number of the target data to be inferred in the request queue is greater than the maximum number threshold, the first target data of the maximum number threshold are obtained from the target data, and feature processing is performed on the first target data in parallel, wherein the first target data is stored in the request queue earlier than the second target data in the request queue except the first target data.

3. The request processing method according to claim 1 or 2, characterized in that: The preset parallel processing condition includes at least one of the following: The time since the last parallel feature processing is greater than the preset time threshold; The amount of image data and / or audio data to be inferred in the request queue reaches a preset maximum amount threshold.

4. The request processing method according to claim 3, characterized in that: The method further comprises: Obtaining a historical delay duration and / or a historical quantity corresponding to a historical reasoning request, wherein the historical delay duration is a difference between an expected processing duration and an actual processing duration of the historical reasoning request, and the historical quantity is a quantity of image data and / or audio data carried by the historical reasoning request; The maximum number threshold is adjusted based on the historical delay duration and / or the historical quantity, wherein the maximum number threshold is negatively correlated with the historical delay duration, and the maximum number threshold is positively correlated with the historical quantity.

5. The request processing method according to claim 3, characterized in that: The method further comprises: Obtaining a historical delay duration corresponding to a historical reasoning request, where the historical delay duration is a difference between an expected processing duration and an actual processing duration of the historical reasoning request; The preset duration threshold is adjusted based on the historical delay duration, wherein the preset duration threshold is negatively correlated with the historical delay duration.

6. The request processing method according to claim 1 or 2, characterized in that: Each feature data in the target feature data carries a request identifier of a corresponding inference request, and the method further includes: Feature splicing is performed according to the request identifier carried by each feature data in the target feature data to obtain feature data corresponding to each reasoning request, and the multimodal machine learning model is used to perform model reasoning according to the feature data corresponding to each reasoning request.

7. The request processing method according to claim 1 or 2, characterized in that: The parallel processing of the image data and / or audio data carried by the inference request in the request queue to obtain target feature data includes: Acquire image data and / or audio data carried by the inference requests in the request queue in parallel to obtain first data; Preprocessing the first data in parallel to obtain second data, wherein the preprocessing includes normalization processing; In parallel, feature extraction is performed on the second data to obtain the target feature data.

8. A request processing device, characterized in that: The device comprises: A receiving module, configured to store an inference request for a multimodal machine learning model into a request queue after receiving the inference request, wherein the inference request carries data to be inferred input by a user, and the data to be inferred includes image data and / or audio data; A processing module is used to perform feature processing on the image data and / or audio data carried by the inference request in the request queue in parallel when a preset parallel processing condition is met to obtain target feature data, and the target feature data is used for model inference by the multimodal machine learning model.

9. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processing device, the steps of the method according to any one of claims 1 to 7 are implemented.

10. An electronic device, characterized in that: include: a storage device having a computer program stored thereon; A processing device, configured to execute the computer program in the storage device to implement the steps of the method according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images