Voice control and instruction processing method, device, client, server and system

Abstract

The invention relates to the technical field of image formation, and particularly provides a voice control and instruction processing method, equipment, a client, a server and a system. According to the invention, through cooperation between the voice control module and the server, the voice control capability of the image forming device is realized. Specifically, after receiving a voice instruction, a voice control module sends the voice instruction to a server, and the server analyzes and processes the voice instruction through an instruction data set of a large language model and an image forming device so as to generate voice response information; therefore, after the voice control module receives the voice response information, the operation instruction corresponding to the voice response information can be executed according to the voice response information. According to the technical scheme provided by the embodiment of the invention, the dependence on software and hardware capabilities of the image forming equipment is relatively small, the flexibility is relatively high, full-function voice control on various image forming equipment can be realized, and the user experience of voice printing is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image forming, and in particular to a voice control and instruction processing method, device, client, server and system. BACKGROUND

[0002] With the wide development of artificial intelligence and Internet of Things (IoT) technologies, intelligent interaction (such as voice interaction) has become a core trend for device upgrades in various industries. In particular, in office scenarios, intelligent interaction upgrades for office devices such as printers have become a key direction for device research and development. For example, voice control requirements for image forming devices.

[0003] In related technologies, the software and hardware capabilities of image forming devices are different, and even have a large gap. If full-featured voice control of image forming devices is to be achieved, AI software that is adapted to each image forming device needs to be developed independently for each image forming device. The AI software is installed locally on the image forming device to give the image forming device voice control capabilities through the AI software. However, this personalized development scheme has poor flexibility and high cost, and even full-featured voice control cannot be achieved for a single image forming device, resulting in poor voice control effect.

[0004] It should be noted that the information disclosed in the background section of the present application is only intended to deepen the understanding of the general background of the present application, and should not be regarded as acknowledging or implying in any form that this information constitutes prior art known to those skilled in the art. SUMMARY

[0005] The present application is proposed in view of the above technical problems. The present application provides a voice control and instruction processing method, device, client, server and system.

[0006] According to one aspect of the present application, a voice control method is provided, applied to a voice control module, the voice control module being in communication connection with an image forming device, and the method comprising: receiving a voice instruction; sending the voice instruction to a server, the server being configured to analyze and process the voice instruction through a large language model and an instruction data set of the image forming device to generate voice response information; receiving the voice response information; executing a job instruction corresponding to the voice response information.

[0007] According to another aspect of the present application, a voice instruction processing method is provided, applied to a server, the method comprising: obtaining a voice instruction; analyzing the voice instruction by a large language model and an instruction data set of the image forming device to generate voice response information; sending the voice response information; the voice response information is used to instruct a voice control module to execute a job instruction.

[0008] According to another aspect of the present application, a voice control module is provided, which is communicatively connected with an image forming device, and comprises: a transceiving unit configured to receive a voice instruction; the transceiving unit is further configured to send the voice instruction to a server, and the server is configured to analyze the voice instruction by a large language model and an instruction data set of the image forming device to generate voice response information; the transceiving unit is further configured to receive the voice response information; a control unit configured to execute a job instruction corresponding to the voice response information.

[0009] According to another aspect of the present application, an image forming device is provided, which comprises a body and a voice control module provided by any one of the embodiments of the present application.

[0010] According to another aspect of the present application, a voice client is provided, which comprises a voice control module provided by any one of the embodiments of the present application; and the voice client is communicatively connected with the image forming device.

[0011] According to another aspect of the present application, a server is provided, which comprises: a transceiving unit configured to obtain a voice instruction; a processing unit configured to analyze the voice instruction by a large language model and an instruction data set of the image forming device to generate voice response information; the transceiving unit is further configured to send the voice response information; the voice response information is used to instruct a voice control module to execute a job instruction.

[0012] According to another aspect of the present application, a voice control system is provided, which comprises: an image forming device provided by any one of the embodiments of the present application; a server provided by any one of the embodiments of the present application.

[0013] According to another aspect of the present application, a voice control system is provided, which comprises: an image forming device; a voice client provided by any one of the embodiments of the present application; The server provided by any one of the embodiments of the present application.

[0014] According to another aspect of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored on the memory, the processor executing the computer program to implement the method of any one of the above embodiments.

[0015] According to another aspect of the present application, a computer readable storage medium is provided, having stored thereon a computer program / instructions, which, when executed by a processor, implement the method of any one of the above embodiments.

[0016] According to another aspect of the present application, a computer program product is provided, comprising computer program / instructions, which, when executed by a processor, implement the method of any one of the above embodiments.

[0017] As will be described in detail below, according to an embodiment of the present application, a voice control method, an instruction processing method, an equipment, a client, a server, a system, a storage medium and a program product. The present application realizes full-function voice control capability of the image forming equipment through the cooperation and matching between the voice control module and the server. Specifically, after receiving the voice instruction, the voice control module sends the voice instruction to the server, and the server analyzes and processes the voice instruction through the large language model and the instruction data set of the image forming equipment to generate voice response information; thus, after receiving the voice response information, the voice control module can execute the job instruction corresponding to the voice response information, and in this process, the server uses the model context protocol (MCP) processing capability of the large language model to understand the instruction data set of the image forming equipment, so as to convert the voice instruction into voice response information corresponding to the semantic, and issue it to the voice control module for execution. The dependence on the image forming equipment is small, which can be widely applied to voice control scenes of various image forming equipment with large differences in software and hardware, saving development cost, and being able to realize full-function voice control of the image forming equipment based on the instruction data set of the image forming equipment, and even in some cases, more functions can be realized beyond the existing capabilities of the image forming equipment. The present application has less dependence on the software and hardware capabilities of the image forming equipment, and has high flexibility, so as to realize full-function voice control of various image forming equipment.

[0018] It is to be understood that both the foregoing general description and the following detailed description are exemplary, and are intended to provide further explanation of the subject technology. BRIEF DESCRIPTION OF DRAWINGS

[0019] The above and other objects, features and advantages of the present application will become more apparent from the following detailed description thereof taken in conjunction with the accompanying drawings in which:

[0020] Figure 1 A structural block diagram of an image forming device is provided for the embodiments of the present application.

[0021] Figure 2 A structural block diagram of a voice client is provided for the embodiments of the present application.

[0022] Figure 3 A structural block diagram of a voice control system is provided for the embodiments of the present application.

[0023] Figure 4 A structural block diagram of a voice control system is provided for the embodiments of the present application.

[0024] Figure 5 A structural block diagram of a voice control system is provided for the embodiments of the present application.

[0025] Figure 6 An interaction flowchart of a voice control method is provided for the embodiments of the present application.

[0026] Figure 7 A flowchart of a voice control method is provided for the embodiments of the present application.

[0027] Figure 8 A flowchart of a voice instruction processing method is provided for the embodiments of the present application.

[0028] Figure 9 A processing logic diagram of a voice instruction processing method is provided for the embodiments of the present application.

[0029] Figure 10 A flowchart of a voice instruction processing method is provided for the embodiments of the present application.

[0030] Figure 11 An interaction flowchart of a voice control method is provided for the embodiments of the present application.

[0031] Figure 12 A structural block diagram of a voice control module is provided for the embodiments of the present application.

[0032] Figure 13 A structural block diagram of a server is provided for the embodiments of the present application.

[0033] Figure 14 A hardware block diagram of an electronic device is provided for embodiments of the present application.

[0034] Figure 15 A schematic diagram of a computer readable storage medium is provided for embodiments of the present application. DETAILED DESCRIPTION

[0035] For better understanding of the technical solutions of the present application, the embodiments of the present application are described in detail below with reference to the drawings.

[0036] It should be clear that the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0037] Unless otherwise specifically stated, the relative arrangement of the components and steps, numerical expressions, and numerical values set forth in the embodiments are not meant to limit the scope of the present application.

[0038] The terms used in the embodiments of the present application are only for the purpose of describing the specific embodiments, and are not intended to limit the present application. The singular forms "a", "an", and "the" used in the embodiments of the present application and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise.

[0039] It should be understood that the term "and / or" used herein only describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the three cases of A alone, A and B together, and B alone. In addition, the character " / " in this paper generally represents that the front and rear associated objects are a "or" relationship.

[0040] The present application can be applied to a voice control scene for an image forming device. For this scene, the present application embodiment provides a new design concept: the voice control module interacts with the server, and the server undertakes the core processing of the voice instruction. The processing idea is to split the capabilities of the image forming device into the smallest functional units (i.e., the smallest instruction units), convert the smallest instruction units into instruction functions, and combine the instruction functions and instruction function combinations to form an instruction data set to represent the capabilities supported by the image forming device. The Model Context Protocol processing capability in the large language model is used to understand the instruction data set of the image forming device, convert the voice instruction input by the user into voice response information corresponding to the semantic, and issue it to the voice control module for execution. Thus, through the cooperation between the voice control module and the server, voice control capabilities can be provided for various image forming devices, full-function control of the image forming device can be realized through voice instructions, and even in some cases, more functions can be realized beyond the existing capabilities of the image forming device. The present application has less dependence on the software and hardware capabilities of the image forming device and is highly flexible, and can thus realize full-function voice control of various image forming devices.

[0041] For ease of understanding, first introduce the system architecture of the present application embodiment.

[0042] For example, the present application provides a voice control method applied to a voice control module. The voice control module can be a software module, a hardware module, or a functional module combined with software and hardware, and there is no limitation. For example, the voice control module can be a controller or a software processing unit inside the controller. The voice control method and the voice control module are described in detail below.

[0043] In this application, the voice control module is communicatively connected to the image forming equipment. This application does not impose any particular limitation on the communication connection method between the voice control module and the image forming equipment; it can include, but is not limited to, one or more of wired and wireless connections. Wired connections can include, but are not limited to, one or more of the following: fiber optic cable connection, network cable connection, serial cable connection, such as a connection via Universal Serial Bus (USB). Wireless connections can include, but are not limited to, one or more of the following: Wireless Fidelity (Wi-Fi) connection, Bluetooth connection, ZigBee connection, Near Field Communication (NFC) connection, cellular network connection, local area network connection, infrared connection, etc., without exhaustive listing. It should be understood that, based on the actual application scenario (e.g., one or more of network environment, equipment deployment requirements, and transmission stability requirements), the above communication connection methods can be flexibly selected to ensure smooth data transmission between the voice control module and the image forming equipment, providing communication assurance for the efficient transmission of voice commands and the stable execution of printing tasks.

[0044] This application does not restrict the configuration relationship between the voice control module and the image forming apparatus. They can be configured independently, such as as an external voice client within the image forming apparatus, or as an electronic device with voice receiving capabilities, but are not limited to these; alternatively, the voice control module can also be part of the image forming apparatus. Based on this relationship, the structure of the voice control system can also vary.

[0045] In one embodiment, the voice control module may be part of the image forming apparatus. In this case, the voice control method provided in this application can be implemented through the voice interaction capabilities, communication capabilities, and processing capabilities inherent in the image forming apparatus itself. Therefore, the voice control method provided in this embodiment can be executed by the image forming apparatus.

[0046] This application provides an image forming apparatus, such as... Figure 1 As shown, the image forming apparatus 100 includes a main body 110 and a voice control module 120. The voice control module 120 is used to implement the voice control method provided in any embodiment of this application, as described below.

[0047] In practical scenarios, an image forming apparatus 100 typically includes an image forming control unit and an image forming unit. The image forming control unit (also called a controller) controls the entire image forming apparatus, while the image forming unit, under the control of the image forming control unit, forms an image on the incoming paper based on image data. For example, the image forming control unit can be a System on Chip (SoC). An SoC is a miniature system composed of multiple system components, configured to control the imaging processing operations of the image forming apparatus, such as performing linear correction, noise reduction, bad pixel removal, and detail enhancement on the image data to improve the quality of the output image. The image forming control unit also performs data transmission and reception, command transmission and reception, and processing operations related to the printing engine control of the image. For example, it transmits and receives data, print engine control commands, and status information through interface units (including but not limited to USB ports, wired network ports, wireless network ports, or other interfaces).

[0048] In another embodiment, the voice control module can be a separate device independent of the image forming device, such as a voice client.

[0049] This application also provides a voice client, such as Figure 2 As shown, the voice client 200 includes a voice control module 120; wherein the voice client 200 is communicatively connected to the image forming apparatus 100.

[0050] The voice client 200 is independent of the image forming apparatus 100. In this case, the voice client 200 can be used in combination with the image forming apparatus 100 to realize voice control of the image forming apparatus 100, or it can be used in combination with other devices to provide voice control capabilities for other devices.

[0051] It should be noted that the voice client 200 and the image forming device 100 are independent of each other, but they are communicatively connected. Based on this communication connection, they can be used in combination to reuse each other's capabilities and enrich their own capabilities. That is, this application also allows for partial capability reuse between the voice client 200 and the image forming device 100. For example, in one possible scenario, if the voice client 200 does not have the ability to communicate externally, then the voice client 200 cannot communicate directly with the server 300; in this case, one can refer to... Figure 3, based on the communication connection relationship (for example, USB connection) between the voice client 200 and the image forming device 100, the voice client 200 can multiplex the external communication capability (for example, external communication module, such as wifi communication, local area network communication, etc.) of the image forming device 100 to realize communication with the server 300. In other words, the voice client 200 can realize communication connection between the server 300 through the image forming device 100. In addition, the application embodiment has no special limitation on the external communication capability of the image forming device 100, which can be one or more of the foregoing communication connection modes, and details are omitted.

[0052] In addition, in the present application, the voice control module 120 or the voice client 200 can obtain the voice instruction issued by the user, and in some embodiments, the voice output interaction information is also involved. For this, the receiving / sending of these voice data can be realized by the voice transceiver module.

[0053] In one embodiment, the voice transceiver module and the voice control module are independently connected in communication, and the communication connection mode is not repeated here. In addition, the voice transceiver module can include a voice receiving module and / or a voice output module, which can be realized by a device that simultaneously has voice receiving and voice output functions, or can be realized by a combination of a voice receiving device and a voice output device.

[0054] In actual scenarios, the voice transceiver module can be realized by a voice transceiver module configured on the image forming device body, or can be realized by any device with voice receiving and / or voice output functions in related technologies, such as mobile phones, tablet computers, notebook computers, sound boxes, speakers, power amplifiers, earphones, smart home devices, etc., without being limited to the above. Figure 3 For example, the voice client 200 includes a voice control module 120, a voice receiving module 210, and a voice output module 220. The voice receiving module 210 is used to receive voice instructions and deliver the voice instructions to the voice control module 120 for processing. The voice processing module 120 can also be used to deliver the interaction information (to be described below) in the voice response information to the voice output module 220, and the voice output module 220 can output the interaction information in the form of voice. In addition, it should be understood that the voice transceiver module can further have other capabilities, such as audio optimization capability, in some embodiments, without being repeated.

[0055] The application also provides a voice instruction processing method and a server. The voice instruction processing method is executed on the server, which will be described in detail below.

[0056] Furthermore, this application does not limit the type or location of the server. For example, the server provided in this application can be a cloud server, and the cloud server can establish a cloud communication link with the voice control module or image forming device when necessary. Furthermore, as described below... Figure 11 As shown, during the process of establishing a cloud communication link, the cloud server can also perform authentication processing on the voice control module. If the authentication is successful, the cloud communication link will be established; otherwise, the cloud communication link will be refused.

[0057] For example, the server can also be a local server. The server can communicate with the voice control module or the image forming device through any of the communication connection methods described above, which will not be elaborated further.

[0058] This application also provides a voice control system. Based on the different relationships between the aforementioned voice processing module and the image forming device, this voice control system can also have various possible implementations.

[0059] In one embodiment, reference may be made to Figure 4 The voice control system 400 provided in this application may include an image forming apparatus 100 and a server 300. In this embodiment, the image forming apparatus 100 includes a body 110 and a voice control module 120, and the voice control module 120 is used to implement the voice control method provided in any embodiment of this application, which will not be described in detail.

[0060] In another embodiment, reference may be made to Figure 5 The voice control system 400 provided in this application may include: an image forming apparatus 100, a voice client 200, and a server 300. In this embodiment, the image forming apparatus 100 and the voice client 200 are communicatively connected. The voice client 200 includes a voice control module 120, and the voice control module 120 is used to implement the voice control method provided in any embodiment of this application, which will not be elaborated further.

[0061] It should be understood that, such as Figure 4 and Figure 5 The voice control system 400 shown takes one image forming device 100 and one voice client 200 as an example. In actual scenarios, one server can correspond to multiple image forming devices 100 and / or multiple voice clients 200. This application does not limit this.

[0062] The following details the voice control method and voice command processing method provided in this application. For ease of understanding, please refer to [the relevant documentation / reference]. Figure 6 , Figure 6 The interaction flow of a voice control method provided in this application illustrates the core interaction logic between the voice control module and the server. For example... Figure 6As shown, the method comprises: S602, the voice control module receives the voice instruction.

[0063] S604, the voice control module sends the voice instruction to the server.

[0064] S606, the server acquires the voice instruction.

[0065] S608, the server processes the voice instruction through a large language model and an instruction data set of the image forming device to generate voice response information.

[0066] S610, the server sends the voice response information to the voice control module. The voice response information is used to instruct the voice control module to execute the job instruction.

[0067] S612, the voice control module receives the voice response information.

[0068] S614, the voice control module executes the job instruction corresponding to the voice response information.

[0069] As Figure 6 shown in the interaction flow, in the present application, the voice control module is used to realize the human-computer interaction part with the user, for example, receiving the voice instruction of the user; and is used to execute the job instruction corresponding to the voice response information; while the core processing part of the voice instruction is completed by the server; and in the analysis process of the voice instruction, the capabilities of the large language model and the image processing device are comprehensively considered (the instruction data set can indicate this), thereby the analysis process has less dependence on the software and hardware capabilities of the image forming device, can rely on the data analysis capability and learning capability of the large language model, analyze and generate voice response information more suitable for the capabilities of the image forming device, and is beneficial to improve the voice control service experience.

[0070] Hereinafter, the present scheme is described from the voice control module and the server respectively.

[0071] The present application provides a voice control method, which is applied to a voice control module. Please refer to Figure 7 , Figure 7 for a flowchart of a voice control method provided by an embodiment of the present application. As Figure 7 shown, the method comprises: S702, receiving a voice instruction.

[0072] The voice instruction is a voice instruction issued by a user to an image forming device; the acquisition method of the voice instruction is referred to the foregoing, which can be transmitted to the voice control module after the voice instruction is received by the voice transceiver module, and is not described herein.

[0073] In addition, in an actual scenario, a wake-up mechanism can also be set for the voice control method provided in this application. Specifically, the voice control module can be woken up after receiving a wake-up word and trigger the implementation of the present solution. The wake-up word can be set by the user and is not limited, and details are not described herein. The recognition of the wake-up word can be implemented by the voice control module. For example, if the voice control module recognizes that the user has issued a wake-up word, a communication link can be established with the server for subsequent application. Based on the wake-up mechanism, the present solution can be triggered by irrelevant voice at will, and the waste of communication resources and processing resources can be avoided.

[0074] S704, sending the voice instruction to the server, the server being configured to analyze and process the voice instruction through a large language model and an instruction data set of the image forming device to generate voice response information.

[0075] Specifically, when sending the voice instruction to the server, if the voice control module has external communication capability, it can be connected with the server, and the voice instruction can be sent to the server directly based thereon; or the voice control module can also indirectly send the voice instruction to the server through the image forming device based on the communication connection between the voice control module and the image forming device. In this case, the voice control module sends the voice instruction to the image forming device and instructs the image forming device to forward it to the server, and details are not described herein.

[0076] The present application implements the core processing of the voice instruction based on the server. In order to meet the better processing effect of different image forming devices, the server implements specific analysis and processing through a large language model (LLM), and reasonably references the instruction data set of the image forming device in the analysis and processing process, so as to generate more suitable voice response information based on the capability range of the image forming device.

[0077] In the present application, the server adopts a large language model to implement the core processing of the voice instruction. The large language model at least has good semantic understanding ability, logical reasoning ability and text generation ability, so as to accurately capture the literal meaning and deep semantics between the input data (such as the instruction data set of the image forming device and the voice instruction), and perform reasonable and effective logical reasoning on the input data based on mathematical calculation, logical deduction, causal analysis and other methods, and finally generate voice response information conforming to the rules of natural language. The present application does not limit the type of large language model and the deployment relationship between the large language model and the server; for example, the large language model can be a self-developed model or can reuse a large language model in related technologies; for another example, the large language model can be deployed locally on the server or can be provided by a third party, which is described in detail in the server part below. The application uses an instruction data set of an image forming device to realize voice instruction analysis, which is used to indicate the capability of the image forming device.

[0078] Specifically, the instruction data set includes instruction functions supported by the image forming device and / or instruction function combinations, the instruction function is the smallest instruction unit that can be executed by the image forming device, and the instruction function combination includes at least two instruction functions. In this way, the instruction function (which can be denoted as Func, combined with numbers or other identifiers to represent different instruction functions) corresponds to the basic job capability that can be implemented by the image forming device, and the instruction function combination can implement a new job capability by combining multiple different basic job capabilities. For example, the instruction functions supported by the image forming device include function 1 and function 2, wherein function 1 can be used to implement the basic job capability of single-sided printing, and function 2 can be used to implement the basic job capability of color printing; based on this, the combination of function 1 and function 2 can be used to implement the color single-sided printing capability; the combination of two function 1s can be used to implement the double-sided printing effect. In this way, the instruction function combination means that more rich job capabilities can be implemented based on the basic job capabilities of the image forming device. Any instruction function or any instruction function combination in the instruction data set can correspond to an executable job instruction. In actual scenarios, the image forming device can complete the job task by splitting and combining the instruction functions corresponding to the job instruction.

[0079] In actual scenarios, the instruction data sets of different image forming devices can be the same or different, for example, if the instruction functions (i.e., basic job capabilities) of the image forming devices are the same, there can also be a case where the instruction function combinations are different. For the image forming device, the corresponding instruction functions are generally preset in advance based on its hardware and software capabilities, for example, preset before the image forming device is shipped or applied; in addition, the application also supports the case of adding part of the instruction functions under the device upgrade scenario. The instruction function combinations in the instruction data set are generally also preset in advance; and some embodiments of the application also support the case of extending the capability range of the image forming device (i.e., based on the instruction functions supported by the image forming device, adding part of the instruction function combinations to implement new functions) in actual application scenarios, and updating the instruction data set. For example, the new instruction function combination provided by the voice response information is registered and saved in the instruction data set, which is described in detail below.

[0080] In addition, the application does not have special restrictions on the storage form of the instruction data set, which can be maintained in the form of a table, such as forming an instruction function table (or a printer function table); or the relationship between the instruction functions and their corresponding capability rules can also be maintained in the form of key-value pairs; or it can also be stored in any way such as technical documents, code information, etc., which is not described in detail.

[0081] It should be understood that for the voice control module, only the voice instruction needs to be sent to the server, and the voice response information of the feedback thereof is received. How the server side specifically parses the voice instruction and generates the voice response information is actually imperceptible to the voice control module, and is not described in detail.

[0082] Based on this, the application can split the capabilities of the image forming device into the smallest functional units (i.e., the smallest instruction units), so as to understand the instruction data set of the image forming device based on the model context protocol (Model Context Protocol) processing capability of the large language model, and convert the voice instruction into voice response information corresponding to the semantics, and issue it to the voice control module for execution. Thus, full function control of the printer can be realized through voice commands.

[0083] S706, receiving the voice response information.

[0084] It can be understood that the voice response information is used to respond to the voice instruction, and the content of the voice response instruction is different based on the content indicated by the voice instruction.

[0085] For example, the voice response information related to the embodiments of the application can include but is not limited to at least one of the following: execution information, interaction information.

[0086] In an embodiment, the voice response information includes execution information, which is used to indicate how the image forming device executes the job instruction. For example, the execution information can include but is not limited to: instruction function and / or instruction function combination. Further, in a possible embodiment, the execution information can further include job parameters, which are used to indicate specific execution parameters of the job instruction. For example, if the voice instruction indicates to print a file A, the voice response information can be specifically the execution information corresponding to the job task of printing job A, i.e., how to execute the job task, and the instruction function matched with the voice instruction is printing, and the job parameter can be used to indicate the number of copies of the printed file A. For another example, if the voice instruction indicates to print the file A in color, the voice response information can be specifically the execution information corresponding to the job task of color printing job A, i.e., how to execute the job task, at this time, the execution information can include the instruction function or instruction function combination corresponding to the execution of the color printing task, at this time, the execution information further includes the job parameter, which can be used to indicate the color required for printing the file A or the consumable information corresponding to the color; not described in detail.

[0087] In one embodiment, the voice response information can include interaction information, which is used to implement human-computer interaction with the user. For example, if the voice instruction is intended to inquire whether the image forming device supports a scanning task, the voice response information can only include interaction information to intelligently respond to the user's inquiry. In this embodiment, the interaction information can be response information including support for scanning or not.

[0088] In another embodiment, the voice response information can include execution information and interaction information. For example, in the process of executing a job task based on the execution information, the user can also be informed of the current execution progress or instructed to cooperate based on the interaction information. Details will be described below in conjunction with embodiments.

[0089] S708, executing the job instruction corresponding to the voice response information.

[0090] For example, if the voice response information includes execution information, executing the job instruction corresponding to the voice response information at least includes executing the job instruction corresponding to the instruction function and / or instruction function combination. In one embodiment, if the execution information further includes a job parameter, the step of executing the job instruction corresponding to the voice response information includes executing the job instruction corresponding to the instruction function and / or instruction function combination based on the job parameter.

[0091] For example, if the voice response information includes interaction information, executing the job instruction corresponding to the voice response information includes voice outputting the interaction information.

[0092] It should be understood that the above two cases can be used alone or in combination. If used in combination, i.e., the voice response information includes execution information and interaction information, the above two implementation manners are executed respectively. The execution timing is not limited, and can be executed simultaneously or sequentially. In some embodiments, the voice response information can also indicate the execution timing relationship of multiple information, which will be described below.

[0093] In the embodiments of the present application, the voice control module has at least two execution modes: execution by the voice control module itself or execution by the image forming device in communication with the voice control module. In specific scenarios, the specific execution mode is different based on the relationship between the voice control module and the image forming device. If the voice control module is part of the image forming device, such as the scenario shown in the foregoing Figure 1 If the voice control module and the image forming device are independently arranged and in communication, if the voice control module does not have direct execution capability, it can instruct other devices in communication therewith to execute the job task to complete the job task. Figure 3The illustrated scenario is exemplary. If the voice response information includes execution information, the voice control module can send the job instruction corresponding to the voice response information to the image forming device connected in communication, so that the image forming device executes the job instruction to complete the job task. For example, if the voice response information includes interaction information, the voice control module can instruct the voice output module to voice output the interaction information.

[0094] The interaction information involved in the embodiments of the present application can include, but is not limited to, one or more of the following: first information, second information, third information, fourth information, and fifth information. The first information is used for human-computer interaction to confirm whether the user accepts the backup function combination (described below); the second information is used for voice prompting that the image forming device does not support the current job task; the third information is used for voice prompting that the target file indicated by the voice instruction does not exist; the fourth information is used for human-computer interaction to prompt the user to cooperate during the job process; and the fifth information is used for voice prompting the job status. It should be understood that this is only used for exemplary description, and the interaction information is intelligently analyzed and determined by the large language model for the voice instruction in the actual scenario, and is not limited to the cases listed here, and is not described in detail. The following will be described in detail in combination with Figure 8 Specific description.

[0095] In the present application, the backup function combination is a backup instruction function combination determined by the server when the instruction data set does not match the instruction function and the instruction function combination matching the voice instruction. In the actual scenario, it generally occurs when the image forming device does not support part of the basic job capability required by the voice instruction (i.e., the image forming device does not have a certain instruction function). In this case, the instruction function not supported by the image forming device can be flexibly changed or omitted to obtain the backup function combination. For example, the voice instruction indicates color double-sided printing of file A, and based on the instruction data set of the image forming device, the instruction function does not include color printing function. In this case, a backup function combination for completing black-and-white double-sided printing of file A can be generated. In other words, the job capability that can be achieved by the backup function combination has at least a certain correlation with the voice instruction. When the server does not match the instruction function and the instruction function combination matching the voice instruction, a backup solution (i.e., a backup function combination) that can be achieved by the image forming device can be provided for the user.

[0096] In the scenario involving the backup function combination, the first information is included in the voice response information, and the first information is used for human-computer interaction to confirm whether the user accepts the backup function combination. That is, the voice control module can voice output the first information, for example, the first information can be: the current printer does not have the color double-sided printing function, do you accept the black-and-white double-sided printing file A? In this way, the user can further respond and interact with the first information. For example, the user can issue a new voice instruction (which can be parsed by the server according to the foregoing process) to indicate whether to accept the backup scheme, or the user can perform a touch operation on the touch panel (on the body of the image forming device or the touch panel of a remote control device such as a mobile phone) to indicate whether to accept the backup scheme. In this way, if the user accepts, the voice control module can execute the job instruction corresponding to the backup function combination; if not, the current task can be ended or further human-computer interaction with the user can be performed, for example, the second information is voice output to prompt that the current image forming device does not support color double-sided printing. Whether to execute the backup function combination can be selected or determined by the user through the output of the interaction information.

[0097] In addition, in the embodiments of the present application, the backup function combination can be one or more, that is, the present application supports outputting multiple backup function combinations for the user to select. For example, if the voice instruction indicates color double-sided printing, there is no matching item in the instruction data set of the image forming device, at this time, multiple backup function combinations can be generated based on the instruction data set, for example, color + single-sided printing, black and white + double-sided printing, black and white + single-sided printing; in this way, the first information can output the above-mentioned multiple backup function combinations for the user to select or reject.

[0098] In the present application, the third information is used for voice prompting that the target file indicated by the voice instruction does not exist. That is, the third information is feedback information to the user in the case that the target file indicated by the voice instruction cannot be found, so that the job task cannot be executed. In the present application, before the voice instruction is executed or the target instruction is parsed, the following processing can also be included: detecting whether the target file exists; if it exists, parsing the voice instruction or executing the job instruction corresponding to the voice response information; if it does not exist, the third information can be voice output. It should be understood that in the actual scenario, further interaction can also be made on the third information. For example, the user can issue a new voice instruction or touch information on the third information, in this way, the voice control module can further obtain the user operation information (such as the new voice instruction or touch information) on the interaction information, and then can continue to refer to the technical solutions provided in the present application for subsequent interaction, which will not be described herein.

[0099] In addition, detecting whether the target file exists can be realized by the voice control module, or can be realized by the image forming device as instructed by the voice control module, or can be realized by the server (for example,Figure 8 The specific detection manner is not limited, for example, a target file can be matched and found in the storage data based on the file identification (for example, the file name) of the target file; for another example, it can be judged whether the target file exists in the file range corresponding to the authority of the voice instruction initiator based on the matching relationship between the authority of the target file and the voice instruction initiator, without exhaustive enumeration and elaboration.

[0100] In the present application, the fourth information is used for human-computer interaction during the operation process to prompt the user to cooperate. For example, in the process of double-sided printing task, the fourth information can be used to instruct the user to manually turn over the current paper to realize the double-sided printing task based on the single-sided printing mechanism. For another example, the fourth information can be used to instruct the user to preheat the image forming device, replace consumables, and the like, without exhaustive enumeration.

[0101] In the present application, the fifth information is used for voice prompting of the operation state. For example, the fifth information can be output before the execution of the operation task to voice prompt the start of the operation; for another example, the fifth information can be output after the execution of the operation task to voice prompt the completion of the operation; for another example, the fifth information can also be output during the operation process to voice prompt the current operation progress, for example, which step is executed or which page is printed, without exhaustive enumeration.

[0102] In addition, in an embodiment of the present application, the method further comprises: receiving a new voice instruction for the interaction information; or, receiving touch control information for the interaction information; and sending the new voice instruction or the touch control information to the server. Thus, the server can process these user operation information (i.e., the new voice instruction or the touch control information) based on the LLM to regenerate new voice response information and feed back to the voice control module, so that the next round of interaction processing can be performed based on the new voice response information. For example, refer to the foregoing, without elaboration.

[0103] The present application also supports secondary combination processing of the instruction data set of the image forming device. In the process of generating the voice response information, in addition to the instruction function or instruction function combination possessed by the instruction data set, the large language model can also understand the instruction function knowledge based on itself, and on the basis of the instruction data set supported by the image forming device, it can perform secondary combination to form a new instruction function combination. The new instruction function combination can be a perfect solution to the voice instruction, or it can be a backup function combination generated under the premise of no perfect solution, without limitation. Based on this, the present application can further realize new capabilities beyond the current operation capability of the image forming device on the basis of the operation capability supported by the image forming device, and realize new functions that the image forming device currently does not have.

[0104] For example, refer to Figure 9, if the instruction data set of the image forming device includes: function 1, function 2, function 3 and function combination 1, wherein function 1 can be used to implement single-sided printing, function 2 can be used to implement color printing, function 3 can be used to implement double-sided printing, and function combination 1 is a combination of function 1 and function 2, used to implement single-sided color printing. In this case, if the voice instruction indicates double-sided color printing, but the instruction data set does not contain this combination, at this time, the present application can also provide function combination 2 of function 2 and function 3 based on its instruction data set to implement double-sided color printing job. In this way, the present application can realize the double-sided color printing capability based on the voice control method described in the present application for image forming devices that do not originally have double-sided color printing capability, that is, the present application can realize voice control of new functions beyond the current capability of the image forming device.

[0105] For example, if the instruction data set of the image forming device includes: function 1, function 2 and function combination 1, wherein function 1 can be used to implement single-sided printing, function 2 can be used to implement color printing, and function combination 1 is a combination of function 1 and function 2, used to implement single-sided color printing. In this case, if the voice instruction indicates double-sided color printing, but the instruction data set does not contain this combination, at this time, the present application can also provide function combination 3 based on its instruction data set to implement double-sided color printing job. At this time, function combination 3 is composed of two function combinations 1, and between the two function combinations 1 there is also a fourth information, which is used to indicate that the user manually flips the page. At this time, the voice response information includes: function combination 1 + fourth information + function combination 1. The present application can realize the double-sided color printing capability based on voice control and user cooperation for image forming devices that do not originally have double-sided color printing capability, that is, the present application can realize voice control of new functions beyond the current capability of the image forming device.

[0106] It should be understood that the foregoing two examples are only exemplary. It should be understood that in the present application, the new capability beyond the current capability of the image forming device is still based on the instruction functions supported by the image forming device, and the secondary combination is carried out on the basis of these instruction functions and / or instruction function combinations, and if necessary, the user can manually cooperate to provide the image forming device with more abundant job capabilities and improve the job range.

[0107] In the present application, if the execution information carried in the voice response information (such as the standby function combination) does not belong to the instruction data set of the image forming device, it is a new function combination scheme generated by the large language model, in this case, the method further includes: adding the new function combination scheme to the instruction data set. In actual scenarios, this processing can be actively implemented by the voice control module, or it can also be passively executed in response to the registration request initiated by the server for the new function combination, and there is no limitation.

[0108] In an example embodiment, the voice response information includes execution information, and the execution information includes a backup function combination, which is not included in the existing combinations in the instruction data set. In this case, the voice control module can further perform the following processing: adding the backup function combination to the instruction data set; or, in response to receiving a registration request for the instruction function combination, adding the backup function combination to the instruction data set.

[0109] Based on this, the backup function combination is added to the instruction data set, and in a subsequent application scenario, when the same or similar voice instruction is received, the backup function combination can be directly used to implement the related job capability. In this way, the application further enriches the capability range of the image forming device in the voice control process, and widens the capability space of the image forming device.

[0110] In the embodiment of the application, the voice response information can carry one or more information, and the information includes execution information and / or interaction information. For example, the voice response information can include multiple execution information, or the voice response information includes multiple interaction information, or the voice response information includes at least one execution information and at least one interaction information.

[0111] If the voice response information includes multiple information, the multiple information can be executed in sequence according to a preset order or an order indicated by the server, or can be executed simultaneously. The specific execution order can be set based on the scene.

[0112] In an example embodiment, if the voice response information includes multiple information, executing the job instruction corresponding to the voice response information includes: executing the job instruction corresponding to the multiple information in sequence according to the order of the multiple information in the voice response information; or, if the voice response information further includes an execution order between the multiple information, executing the job instruction corresponding to the multiple information in sequence according to the execution order. In one implementation of the embodiment, the arrangement order of the multiple information is directly used as the execution order for execution in sequence; in another implementation, an information in the voice response information fed back by the server (i.e., the execution order) is used for indication. The application does not limit the form of the execution order (or the indication information of the execution order), for example, one or more of the following ways can be used to indicate the execution order between the multiple information: table, sequence, tree structure, flowchart, etc.

[0113] In one embodiment of the present application, the voice control module can further interact with the server to synchronize the instruction data set of the image forming device connected with itself to the server. In a specific scenario, the voice control module can actively synchronize (i.e. send) the instruction data set to the server, for example, periodically synchronize or periodically synchronize, or actively synchronize when there is an update, or actively report the instruction data set of the image forming device after establishing the communication link (e.g. as shown in Figure 9 Alternatively, the server can also actively request the instruction data set from the voice control module.

[0114] In an exemplary embodiment, for the voice control module, the method further comprises: receiving an acquisition request from the server, the acquisition request being used to request to acquire the instruction data set supported by the image forming device; and sending the instruction data set to the server.

[0115] It should be understood that if the voice control module is deployed in the image forming device, the voice control module can directly acquire the instruction data set of the image forming device, which is not described in detail. If the voice control module and the image forming device are independently arranged and communicatively connected, there can be two ways: the voice control module can send a first instruction to the image forming device to instruct the image forming device to send the instruction data set to the server, for example, the scenario as shown in the foregoing Figure 3 Alternatively, the voice control module can interact with the image forming device to acquire the instruction data set of the image forming device, so that the voice control module can directly send the instruction data set to the image forming device, which is not described in detail.

[0116] The present application does not have special restrictions on the format of data interaction between the voice control module and the server. For example, the voice instruction can be in one or more of text format, audio format, etc.; the voice response information (including execution information and / or interaction information) can also be in one or more of text format, audio format, etc. In addition, the format of the voice instruction sent by S704 and the voice response information received by S706 has no relationship, both of which can be the same, for example, both are in audio format; or they can be different, for example, S704 can send voice instructions in audio format, and S706 can receive voice response information in text format. It is not exhaustive.

[0117] Therefore, in some embodiments of the present application, the voice control module can further include the following processing: pre-processing the voice instruction before S704; and / or, data processing the voice response information before S708. In this way, through the above-mentioned pre-processing and / or data processing, the input data / output data of the voice control module and the server can be better adapted, and the scheme implementation can be guaranteed.

[0118] Specifically, the preprocessing can include but is not limited to at least one of the following: text conversion, audio optimization, etc. without being exhaustive. Among them, text conversion means converting the voice instruction from an audio format to a text format to obtain instruction text corresponding to the voice instruction. In this case, S704 is: sending the instruction text to the server. In actual scenarios, the text conversion processing can have multiple possible implementation manners.

[0119] In an embodiment, the voice control module (or can be through the image forming device) can call a third-party voice recognition service to recognize the voice instruction to obtain the instruction text. At this time, the voice control module (or can be through the image forming device) can call a service interface of the third-party voice recognition service, input the voice instruction through the service interface, and obtain the instruction text fed back by the third-party voice recognition service.

[0120] In an embodiment, the voice control module (or can be through the image forming device) can call a voice recognition model to recognize the voice instruction to obtain the instruction text. Among them, the voice recognition model is deployed locally on the image forming device or provided by a third party. The voice recognition model can be a large language model or any pre-trained machine learning model, the input data of which is audio data, the output data of which is text data, which is used to convert audio data into text data and output; the model structure of the voice recognition model is not limited in the present application, and mature models in related technologies can also be reused in actual scenarios, which will not be described here.

[0121] In an embodiment, the voice control module (or can be through the image forming device) utilizes a preset text conversion algorithm to process the voice instruction to obtain the instruction text. The text conversion algorithm can be a self-defined algorithm, or a mature algorithm technology in related technologies can be reused, which will not be described here.

[0122] It should be understood that the text conversion processing of the voice instruction to obtain the instruction text corresponding to the voice instruction can include one or more of the above manners, that is, the above several text conversion processing manners can be used alone or in combination. If used in combination, the instruction texts obtained by the multiple processing manners can be fused to obtain the instruction text that can be output by S704. Among them, the fusion processing manner is not limited, which can be realized by a machine learning type fusion model, or by a way of merging the same information and deleting the difference information, without special limitation or exhaustive.

[0123] Specifically, the data processing after S706 can include but is not limited to at least one of the following: voice synthesis, audio optimization, etc. without being exhaustive. Among them, voice synthesis means generating voice data corresponding to the voice response information.

[0124] Here, mainly involves the case that the voice response information includes interactive information, and the interactive information is non-voice data (for example, text data). If the interactive information is interactive voice, the voice output interactive information includes: playing the interactive voice. Or, if the interactive information is interactive text, the voice output interactive information includes: performing voice synthesis processing on the interactive text to obtain interactive voice, and playing the interactive voice. In actual scenarios, the voice synthesis processing can have multiple possible implementation manners.

[0125] In an embodiment, the voice control module (or can be through the image forming device) calls a third-party voice synthesis service to perform voice synthesis processing on the interactive text to obtain the interactive voice. At this time, the voice control module (or can be through the image forming device) can call a service interface of the third-party voice synthesis service, input the interactive text through the service interface, and obtain the interactive voice fed back by the third-party voice synthesis service.

[0126] In an embodiment, the voice control module (or can be through the image forming device) can call a voice synthesis model to perform voice synthesis processing on the interactive text to obtain the interactive voice; wherein the voice synthesis model is deployed locally on the image forming device or provided by a third party. The voice synthesis model can be a large language model or any pre-trained machine learning model, the input data of which is text data, and the output data of which is voice data (i.e. a kind of audio data), for generating voice data corresponding to the text data and outputting; the model structure of the voice synthesis model is not limited in the present application, and mature models in related technologies can also be reused in actual scenarios, which will not be described herein.

[0127] In an embodiment, the voice control module (or can be through the image forming device) can also utilize a preset audio synthesis algorithm to perform voice synthesis processing on the interactive text to obtain the interactive voice. The audio synthesis algorithm can be a self-defined algorithm, or a mature algorithm technology in related technologies can be reused, which will not be described herein.

[0128] It should be understood that the voice synthesis processing on the interactive text to obtain the interactive voice corresponding to the interactive information can include one or more of the above manners, i.e. the above several voice synthesis processing manners can be used alone or in combination. If used in combination, the interactive voice obtained by multiple processing manners can be fused or filtered to obtain the interactive voice for voice output. For example, one with higher voice quality can be filtered for voice output, without particular limitation or exhaustive enumeration.

[0129] In summary, the present application provides a voice control method, in which the voice control module can cooperate with the server to realize full-function voice control of various image forming devices.

[0130] The application provides a voice instruction processing method, which is applied to a server. Figure 10 , Figure 10 A flowchart of a voice instruction processing method provided by an embodiment of the application is shown in FIG. 1. Figure 10 As shown in FIG. 1, the method comprises the following steps. S1002, obtaining a voice instruction.

[0131] Specifically, the voice instruction sent directly or indirectly (through an image forming device, for example, a printer) by a voice control module can be received. In addition, the voice instruction can be in one or more of audio format or text format, which is not limited. Figure 3

[0132] S1004, performing analysis processing on the voice instruction by a large language model and an instruction data set of the image forming device, to generate voice response information.

[0133] As described above, the MCP processing capability of the large language model can be used to understand the instruction data set of the image forming device, and the voice instruction can be converted into voice response information corresponding to the semantics, and then sent to the voice control module for execution.

[0134] S1006, sending the voice response information; the voice response information is used to instruct the voice control module to execute a job instruction.

[0135] In an exemplary embodiment, the voice response information can comprise execution information; the execution information comprises an instruction function and / or an instruction function combination; wherein the instruction data set comprises instruction functions and instruction function combinations supported by the image forming device, the instruction function is the smallest instruction unit that can be executed by the image forming device, and the instruction function combination comprises at least two instruction functions. In addition, in some embodiments, the execution information can further comprise an execution parameter. Details are described above and will not be repeated here.

[0136] In an exemplary embodiment, the voice response information comprises interaction information. The interaction information can comprise one or more of the following: first information, second information, third information, fourth information, and fifth information. Details are described above and will not be repeated here. Figure 8

[0137] In addition, the voice response information can comprise execution information and interaction information. In addition, the voice response information can comprise a plurality of information, and the voice response information can comprise an execution order of the plurality of information; or the execution order of the plurality of information can be indicated by the order of the plurality of information in the voice response information. Details are described above and will not be repeated here.

[0138] ​​In the present application, the parsing of the voice instruction by the large language model and the instruction data set of the image forming device can include: obtaining the instruction data set of the image forming device in communication connection with the voice control module; calling the large language model, and inputting the instruction data set and the voice instruction as the input of the large language model to obtain the voice response information output by the large language model.

[0139] In the present application, the large language model can be deployed on a local server, or can be deployed on a third party, for example, a large language model provided by a third party service can be used to realize the parsing of the voice instruction. In one possible embodiment, the server can call the service interface of the third party service (which is used to provide the large language model service), input the voice instruction through the service interface, and receive the feedback voice response information.

[0140] In addition, the present application does not have special restrictions on the input data type and output data type of the large language model. Specifically, the input data of the large language model can include but is not limited to at least one of audio data and text data; the output data of the large language model can include but is not limited to at least one of audio data and text data. In addition, in the present application, the input data type and the output data type of the large language model are not related, and they can be the same or different, and can be customized or set based on actual scene requirements or model structure.

[0141] In actual scenarios, the large language model can only support partial types of data input and / or output, for example, the large language model can only support text input and text output. In this case, there can be a situation that the external interaction data of the server (the received voice instruction and / or the issued voice response information) does not match the data type of the large language model. For example, the large language model only supports text input, while the voice instruction received by the server is in audio format. For the foregoing data type mismatch, in the present application, data adjustment can be performed before and / or after data parsing by the large language model, so that the external interaction data of the server matches the data type of the large language model.

[0142] Exemplarily, based on the different types of input data supported by the large language model, the parsing of the voice instruction by the large language model and the instruction data set of the image forming device can include the following several use scenarios.

[0143] In one scenario, if the voice instruction is audio data and the large language model only supports text input, the instruction text corresponding to the voice instruction is obtained, and the instruction text is parsed by the large language model and the instruction data set to generate the voice response information.

[0144] In this scenario, the voice control module can send the received voice instruction in the form of audio data to the server, and the server needs to perform text conversion processing on the voice instruction before parsing it using a large model. The instruction text corresponding to the voice instruction is used as input for the large language model to generate voice response information. In addition, the server obtaining the instruction text corresponding to the voice instruction can also include one or more of the following three processes: calling a third-party speech recognition service to recognize the voice instruction and obtain the instruction text; calling a speech recognition model to recognize the voice instruction and obtain the instruction text, wherein the speech recognition model is deployed locally on the image forming device or provided by a third party; and processing the voice instruction using a preset text conversion algorithm to obtain the instruction text. Reference is not repeated in the foregoing. In addition, it is additionally pointed out that in this scenario, if the LLM model is provided by a third party, the provider of the LLM and the provider of the model or service used for text conversion can be the same or different.

[0145] Alternatively, in another scenario, if the voice instruction is audio data and the large language model supports audio input, the voice instruction is directly parsed and processed by the large language model and the instruction data set to generate voice response information.

[0146] In this scenario, the large language model supports audio input, so the voice instruction (audio data type) can be directly input into the large language model. In this case, in addition to being used to parse and process the voice instruction to generate voice response information, the large language model is also used for semantic understanding and recognition of the voice instruction, for example, it can be used to implement text conversion processing to simplify the complexity and difficulty of subsequent parsing processing.

[0147] In an exemplary embodiment, the large language model supports audio input, and the large language model includes a first model and a second model. The first model is used to perform voice recognition on the input audio data and output text data corresponding to the audio data. The second model is used to process the input text data and output voice response information. For example, the first model can be an automatic speech recognition (ASR) model, and the second model can be an LLM model. The output data of the ASR model (i.e., the instruction text) is the input data of the LLM model, which is equivalent to adding a part of the ASR model before the LLM to implement data preprocessing.

[0148] Alternatively, in another scenario, if the voice instruction is text data, the instruction text is directly parsed and processed by the large language model and the instruction data set to generate voice response information.

[0149] In this scenario, the voice instruction is text data, that is, the voice control module converts the voice instruction into text in advance, and sends the converted text data to the server; in this way, the server does not need additional processing, and can directly analyze and parse the voice instruction in the form of text data. As for how the voice control module realizes text conversion, there is no limitation here, and reference is made to the foregoing description.

[0150] In the embodiments of the present application, the interaction information can include one or more of the following: interaction voice, interaction text. Based on the different types of output data supported by the large language model, there can be the following several cases.

[0151] In one scenario, the large language model can only support text data, at this time, the voice response information (including execution information and / or interaction information) is text data. At this time, the server can directly send the voice response information in the form of text data to the voice control module, and the voice control module performs voice synthesis and output processing on the interaction information, reference is made to the foregoing description. Alternatively, before sending the voice response information, the method further includes: if the text data in the voice response information is converted into voice data. For example, the interaction information can include interaction text (i.e. interaction information in the form of text data), the server can also convert the interaction text into interaction voice. That is, the server is also used for voice synthesis processing.

[0152] In addition, the server converts the interaction text into interaction voice, which can also include one or more of the following three processes: calling a third-party voice synthesis service to perform voice synthesis processing on the interaction text to obtain the interaction voice; calling a voice synthesis model to perform voice synthesis processing on the interaction text to obtain the interaction voice, the voice synthesis model being deployed locally on the image forming device or being provided by a third party; using a preset audio synthesis algorithm to perform voice synthesis processing on the interaction text to obtain the interaction voice. Reference is made to the foregoing description. It is additionally pointed out that in this scenario, if the LLM model is provided by a third party, the provider of the LLM and the provider of the model or service used for voice synthesis can be the same or different.

[0153] Alternatively, in another scenario, the large language model supports audio output; further, the large language model can only support audio output, or can support audio output and text output, which is not limited. In this case, in addition to being used for analyzing and processing the voice instruction to generate voice response information, the large language model is also used for voice synthesis of the text data in the voice response information, for example, it can be used to convert the interaction text into interaction voice, and the final output data can be: execution information (text data) and interaction voice (audio data).

[0154] In an example embodiment, the large language model supports audio output, and the large language model includes a second model and a third model; the second model is configured to process input text data and output response information; and the third model is configured to generate audio from input text data and output audio data. For example, the second model is an LLM model, and the third model is a text-to-sound (TTS) model. The output data of the LLM model (e.g., interactive text) is input data of the TTS model, which is equivalent to adding a TTS model after the LLM to realize data format adjustment.

[0155] For ease of understanding, please refer to Figure 9 In this embodiment, the server corresponds to three third-party services, namely, a third-party ASR service, a third-party LLM service, and a third-party TTS service. Thus, after receiving the voice instruction (audio data), the server can call the third-party ASR service to perform voice recognition (i.e., text conversion processing) on the voice instruction to obtain instruction text corresponding to the voice instruction; then, the server calls the third-party LLM service to perform analysis processing on the instruction text to generate voice response information in the form of text data; then, if there is interactive text, the server can also call the third-party TTS service to perform voice synthesis processing on the interactive information in the voice response information (this step is an optional step, please refer to Figure 9 ). Finally, the processed voice response information is fed back to the voice control module.

[0156] In the embodiment, the server can also receive the instruction data set of the image forming device sent by the voice control module; or the server can actively request the voice control module to feed back the instruction data set. In a possible embodiment, the server can send a request for acquisition to the voice control module, and the request for acquisition is used to request to acquire the instruction data set supported by the image forming device; and receive the instruction data set. Please refer to the foregoing description.

[0157] It should be understood that the large language model in the present application generates voice response information based on the instruction data set of the image forming device. In specific implementation, within the scope of the instruction data set of the image forming device, a proper instruction function or instruction function combination can be matched; but if there is no matching instruction function and instruction function combination, the large language model can also generate a backup function combination based on the instruction data set of the image forming device.

[0158] The backup function combination is a backup instruction function combination determined by the server when there is no instruction function and instruction function combination matching the voice instruction in the instruction data set. The number thereof can be one or more. Please refer to the foregoing description.

[0159] In this case, the server calls the large language model and inputs the instruction data set and the voice instruction into the large language model as input of the large language model, obtains voice response information output by the large language model, and can include the following processing: matching between the voice instruction and the instruction data set based on the instruction function knowledge; if there is an instruction function and / or instruction function combination in the instruction data set that matches the voice instruction, generating voice response information based on the instruction function and / or instruction function combination; if there is no instruction function and instruction function combination in the instruction data set that matches the voice instruction, generating voice response information based on the instruction function knowledge, the instruction data set, and the voice instruction; wherein the voice response information includes: a backup function combination and first information; the first information is used for human-computer interaction to confirm whether the user accepts the backup function combination.

[0160] The instruction function knowledge includes instruction functions and capability boundaries of the instruction functions. In this application, for the large language model, the instruction function knowledge can be a special knowledge base in the image forming field. The large language model can understand each instruction function or instruction function combination in the instruction data set based on the instruction function knowledge to determine what capability each instruction function or instruction function combination can achieve. Based on this, the matching instruction function and / or instruction function combination for the voice instruction can be determined; and in the case of no matching scheme, a backup scheme (i.e., a backup function combination, refer to the foregoing and Figure 9 ).

[0161] The instruction function knowledge can be obtained through the image forming device knowledge base in related technologies, for example, the instruction function knowledge can be involved in the code file or instruction manual of the image forming device. In addition, the instruction function knowledge includes but is not limited to the instruction data set of the image forming device, for example, the instruction function knowledge can be a collection of all instruction data sets of the image forming device in the field or the enterprise or the type (dimension is not limited, not exhaustively listed).

[0162] In addition, as described above, in the voice control scene, multiple rounds of human-computer interaction with the user can be involved, for example, after the user feeds back to the first information, the voice control module feeds back the user operation information of the user to the server. The server also needs to further analyze and process it.

[0163] In an exemplary embodiment, if the voice response information includes interaction information; after the server sends the voice response information, the method further includes: obtaining user operation information for the interaction information; the user operation information includes: a new voice instruction or touch information; based on the user operation information, a new voice response information is regenerated and sent.

[0164] The application does not limit the number of interaction rounds corresponding to one voice task, which is determined based on the actual scene and the user operation information.

[0165] In addition, in the embodiments of the present application, the large language model can perform targeted analysis processing based on the current voice instruction, or perform comprehensive analysis processing based on the multi-round voice interaction after wake-up to obtain voice response information.

[0166] In a possible embodiment, the voice instruction is analyzed and processed by the large language model and the instruction data set of the image forming device, including: the large language model analyzes and processes the voice instruction based on the historical interaction information of the current task and the instruction data set to generate voice response information.

[0167] In specific implementation, the large language model is configured to analyze and process the voice instruction based on the historical interaction record of the current task and the instruction data set to generate voice response information. At this time, the historical interaction record of the current task and the instruction data set can both be prompt information or reference information for the large language model to analyze and process, without the need for server input. The large language model can automatically use the historical interaction record of the current task as reference information to generate voice response information for the current round of feedback. Alternatively, in another implementation, the server can also input the historical interaction record of the current task as part of the input data of the large language model, so that the large language model can perform comprehensive analysis processing in combination with the historical interaction record.

[0168] In addition, in some embodiments of the present application, while analyzing and processing the voice instruction, the server can also timely notify the voice control module to perform device preheating processing in advance if it is detected that the voice instruction is used to instruct a print job, etc., as shown in Figure 8 This can realize multi-flow parallel processing, greatly shorten the overall time consumption of the task from initiation to execution, and improve office efficiency.

[0169] In summary, it is explained how the embodiments of the present application realize full-featured voice control of the image forming device.

[0170] Now, the interaction process between the voice control module (taking a voice client as an example) and the server (taking a cloud server as an example) will be briefly described. Figure 11 As shown in Figure 11 The method comprises the following steps: S1, the voice client establishes a cloud communication link with the cloud server. Referring to the foregoing, the voice client sends a connection request to the cloud server, the cloud server authenticates the voice client, and if the authentication is passed, the cloud communication link is established.

[0171] S2, the voice client sends an instruction data set to the cloud server. That is, the voice client reports the capability information of the image forming device (including but not limited to the instruction data set) to the cloud server. As described above, the cloud server can actively request the voice client to report the capability information, or the voice client can actively report the capability information after establishing the communication link.

[0172] S3, the voice client receives a voice instruction.

[0173] S4, the voice client sends the voice instruction to the cloud server.

[0174] S5, the cloud server calls a third-party ASR service to obtain an instruction text corresponding to the voice instruction. That is, the text conversion process described above.

[0175] S6, the cloud server calls a third-party LLM service to analyze the instruction text and generate voice response information. Specifically, as shown in FIG. 6, the voice response information includes execution information, that is, an instruction function or a combination of instruction functions, and further includes job parameters. Figure 11

[0176] S7, the cloud server sends the voice instruction information to the voice client.

[0177] S8, the voice client receives the voice instruction information and executes a job instruction corresponding to the execution information.

[0178] S9, the voice client reports an execution result to the cloud server. At this time, the execution result can be used as further interaction information.

[0179] S10, the cloud server calls a third-party LLM service to analyze the execution result to determine whether the job task is completed and obtain new voice response information. The voice response information includes interaction information, such as the fifth information described above.

[0180] S11, the cloud server calls a third-party TTS service to perform voice synthesis on the interaction information.

[0181] S12, the cloud server sends the interaction information (audio data) to the voice client.

[0182] S13, the voice client plays the interaction information.

[0183] ​In the process, there is no need to preset the printer capability words in the cloud server, no need to enumerate the capability word setting; also no need to rely on the specific capability of the image forming device, the dependence on the capability of the image forming device is small, and the printer firmware can be widely applicable, and different versions and different functions are applicable. In addition, in the present application, based on the bidirectional semantic understanding capability of the large language model, the software and hardware capabilities of the image forming device can be understood from the function layer, the voice instruction can be understood, and the voice instruction can be accurately parsed in combination with the capability of the image forming device to generate voice response information more suitable for its needs. The intelligent adaptability of the present application is good, relying on the analysis capability of the large language model, the software and hardware capabilities of the image forming device can be automatically adapted, the feasibility of the task is ensured, and the task failure problem caused by the mismatch between the parameter setting and the device capability is reduced.

[0184] In addition, for the user, the operation of the present application is convenient, the user does not need to master complex printing setting operation, and only needs to initiate the printing demand through the natural language voice instruction, which effectively reduces the operation threshold, and is especially suitable for users who are not familiar with the printer or the driving end operation. In addition, the present application also guides through the interaction information, detects whether the file exists and feeds back the detection result in time, so that the user can master the printing task progress in real time, improve the use experience, and is also beneficial to reduce the response time and processing time.

[0185] In addition, part of the steps in the present application, such as instruction parsing processing, image forming device preheating, instruction data set acquisition, etc., can be processed in parallel, which can greatly shorten the overall time consumption of the task from initiation to execution, and improve the office efficiency.

[0186] To sum up, the technical scheme (including the voice control method and / or the voice instruction processing method) provided in the present application realizes full-function voice control capability of the image forming device through the cooperation between the voice control module and the server. Specifically, after receiving the voice instruction, the voice control module sends the voice instruction to the server, and the server analyzes and processes the voice instruction through the large language model and the instruction data set of the image forming device to generate voice response information. Therefore, after receiving the voice response information, the voice control module can execute the job instruction corresponding to the voice response information, and in this process, the server uses the MCP processing capability of the large language model to understand the instruction data set of the image forming device, so as to convert the voice instruction into voice response information with corresponding semantics and issue the voice control module for execution. The dependence on the image forming device is small, and the voice control scheme can be widely applied to voice control scenes of various image forming devices with large differences in software and hardware, saving development cost and enabling full-function control of the image forming device based on the instruction data set of the image forming device. Even in some cases, more functions can be realized beyond the existing capabilities of the image forming device. The present scheme has small dependence on the software and hardware capabilities of the image forming device and high flexibility, and can realize full-function voice control of various image forming devices accordingly.

[0187] Correspondingly, the present application also provides a voice control module capable of being communicatively connected with an image forming device. Figure 12 A structural block diagram of a voice control module provided in an embodiment of the present application is shown in Figure 12 As shown in the figure, the voice control module 120 includes: The transceiver unit 121 is configured to receive a voice instruction. The transceiver unit 121 is also configured to send the voice instruction to a server, and the server is configured to analyze and process the voice instruction through a large language model and an instruction data set of an image forming device to generate voice response information. The transceiver unit 121 is also configured to receive voice response information. The control unit 122 is configured to execute a job instruction corresponding to the voice response information.

[0188] In an exemplary embodiment, the voice response information includes execution information, and the execution information includes an instruction function and / or an instruction function combination. The control unit 122 is specifically configured to execute a job instruction corresponding to the instruction function and / or the instruction function combination. The instruction data set includes an instruction function and / or an instruction function combination supported by the image forming device. The instruction function is a minimum instruction unit that can be executed by the image forming device, and the instruction function combination includes at least two instruction functions.

[0189] In an example embodiment, the control unit 122 is specifically configured to execute a job instruction corresponding to the instruction function and / or the instruction function combination based on the job parameter.

[0190] In an example embodiment, the voice response information comprises interaction information, and the control unit 122 is specifically configured to output the interaction information via voice.

[0191] In an example embodiment, the interaction information comprises one or more of the following: first information for human-computer interaction to confirm whether the user accepts a backup function combination, the backup function combination being a backup instruction function combination determined by the server when there is no instruction function and instruction function combination matching the voice instruction in the instruction data set; second information for voice prompting that the image forming device does not support the current job task; third information for voice prompting that the target file indicated by the voice instruction does not exist; fourth information for human-computer interaction to prompt the user to perform a cooperative operation during the job process; fifth information for voice prompting of a job state.

[0192] In an example embodiment, the transceiver unit 121 is further configured to: receive a new voice instruction for the interaction information; or receive touch control information for the interaction information; send the new voice instruction or the touch control information to the server.

[0193] In an example embodiment, the control unit 122 is further configured to add the backup function combination to the instruction data set; or in response to receiving a registration request for the instruction function combination, add the backup function combination to the instruction data set.

[0194] In an example embodiment, if the voice response information comprises multiple information, the control unit 122 is specifically configured to execute the job instructions corresponding to the multiple information in sequence according to the order of the multiple information in the voice response information; or if the voice response information further comprises an execution order between the multiple information, execute the job instructions corresponding to the multiple information in sequence according to the execution order.

[0195] In an example embodiment, the transceiver unit 121 is further configured to receive an acquisition request from the server, the acquisition request being used to request acquisition of the instruction data set supported by the image forming device; and send the instruction data set to the server.

[0196] The parts not described in detail are referred to the foregoing, and will not be repeated.

[0197] Correspondingly, the application further provides a voice control module capable of being communicatively connected with the image forming device. Figure 13 A structural block diagram of a server provided by an embodiment of the application is shown in Figure 13 The server 300 includes: A transceiver unit 310 configured to obtain a voice instruction; A processing unit 320 configured to perform parsing processing on the voice instruction by a large language model and an instruction data set of the image forming device to generate voice response information; The transceiver unit 310 is further configured to send the voice response information; the voice response information is used to instruct the voice control module to execute a job instruction.

[0198] In an exemplary embodiment, the voice response information includes execution information; the execution information includes an instruction function and / or an instruction function combination; wherein the instruction data set includes instruction functions and instruction function combinations supported by the image forming device, the instruction function is the smallest instruction unit that can be executed by the image forming device, and the instruction function combination includes at least two instruction functions.

[0199] In an exemplary embodiment, the voice response information includes interaction information.

[0200] In an exemplary embodiment, the processing unit 320 is specifically configured to: obtain the instruction data set of the image forming device communicatively connected with the voice control module; call the large language model, and input the instruction data set and the voice instruction as inputs of the large language model to obtain the voice response information output by the large language model.

[0201] In an exemplary embodiment, the processing unit 320 is specifically configured to: match between the voice instruction and the instruction data set based on instruction function knowledge; wherein the instruction function knowledge includes instruction functions and capability boundaries of the instruction functions; if the instruction data set has instruction functions and / or instruction function combinations matching the voice instruction, generate the voice response information based on the instruction functions and / or the instruction function combinations; if the instruction data set does not have instruction functions and instruction function combinations matching the voice instruction, generate the voice response information based on the instruction function knowledge, the instruction data set, and the voice instruction; wherein the voice response information includes a backup function combination and first information; the first information is used for human-computer interaction to confirm whether the user accepts the backup function combination.

[0202] In an exemplary embodiment, if the voice response information includes interaction information; after the voice response information is sent, the transceiver unit 310 is further configured to obtain user operation information for the interaction information; the user operation information includes a new voice instruction or touch information; and the processing unit 320 is further configured to regenerate new voice response information based on the user operation information and send the new voice response information.

[0203] In an example embodiment, the processing unit 320 is specifically configured to: if the voice instruction is audio data and the large language model only supports text input, obtain instruction text corresponding to the voice instruction, and perform parsing processing on the instruction text through the large language model and the instruction data set to generate the voice response information; or, if the voice instruction is audio data and the large language model supports audio input, directly perform parsing processing on the voice instruction through the large language model and the instruction data set to generate the voice response information; or, if the voice instruction is text data, directly perform parsing processing on the instruction text through the large language model and the instruction data set to generate the voice response information.

[0204] In an example embodiment, the large language model supports audio input, and the large language model includes a first model and a second model; the first model is configured to perform voice recognition on input audio data and output text data corresponding to the audio data; and the second model is configured to process input text data and output voice response information.

[0205] In an example embodiment, the processing unit 320 is specifically configured to: based on the historical interaction information of the current task and the instruction data set, the large language model performs parsing processing on the voice instruction to generate the voice response information.

[0206] In an example embodiment, the interaction information includes one or more of the following: interaction voice and interaction text; the large language model supports audio output; the large language model includes a second model and a third model; the second model is configured to process input text data and output response information; and the third model is configured to perform audio generation on input text data and output audio data. Alternatively, the large language model supports text output; and before the voice response information is sent, the method further includes: converting the interaction text in the voice response information into interaction voice.

[0207] The parts not described in detail are referred to the foregoing, and will not be repeated.

[0208] Figure 14 A hardware block diagram of an electronic device is provided for the embodiments of the present application. The electronic device 1400 according to the embodiments of the present application at least includes a memory, a processor, and a computer program stored on the memory, and the processor executes the computer program to implement the method described in any of the above embodiments.

[0209] Figure 14The electronic device 1400 shown specifically includes a central processing unit (CPU) 1401, a graphics processing unit (GPU) 1402, and a memory 1403. These units are connected to each other through a bus 1404. The central processing unit (CPU) 1401 and / or the graphics processing unit (GPU) 1402 can be used as the above-mentioned processor, and the memory 1403 can be used as the above-mentioned memory storing computer readable instructions. In addition, the electronic device 1400 can also include a communication unit 1405, a storage unit 1406, an output unit 1407, an input unit 1408, and an external device 1409, which are also connected to the bus 1404.

[0210] Figure 15 A schematic diagram of a computer readable storage medium provided by an embodiment of the present application is shown. The computer readable storage medium according to the embodiment of the present application has computer programs / instructions (including but not limited to computer readable instructions) stored thereon. Specifically, as shown in the figure, the computer readable instructions 1501 are stored on the computer readable storage medium 1500. The computer programs / instructions are executed by a processor to implement the method described in any of the preceding embodiments of the present application. The computer readable storage medium includes but is not limited to, for example, volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and / or cache memory, etc. The non-volatile memory may, for example, include read-only memory (ROM), hard disk, flash memory, optical disc, magnetic disc, etc. Figure 15

[0211] The present application further provides a computer program product, which includes computer programs / instructions, and the computer programs / instructions are executed by a processor to implement the method described in any of the preceding embodiments of the present application.

[0212] The basic principles of the present application are described above in combination with specific embodiments, but it should be noted that the advantages, advantages, effects, etc. mentioned in the present application are only examples and are not limiting, and these advantages, advantages, effects, etc. cannot be considered as the must-have of each embodiment of the present application. In addition, the above-mentioned specific details are only for the purpose of example and for the purpose of understanding, and are not limiting, and the above-mentioned details do not limit the present application to the must-use of the above-mentioned specific details to realize.

[0213] ​The block diagrams of the devices, apparatuses, equipment, systems referred to in this application are merely illustrative examples and are not intended to require or imply that the connection, arrangement, configuration must be as shown in the block diagrams. These devices, apparatuses, equipment, systems can be connected, arranged, configured in any manner as will be appreciated by those skilled in the art. Words such as "include," "contain," "have," etc. are open-ended words that are to be interpreted to mean "including but not limited to" and are to be used interchangeably. The words "or" and "and" as used herein are to be interpreted as the word "and / or" and are to be used interchangeably unless the context clearly indicates otherwise. The word "such as" as used herein is to be interpreted as the phrase "such as but not limited to" and is to be used interchangeably.

[0214] Also, as used herein, the term "or" when used in a list of two or more items indicates the separate lists such that, for example, the phrase "at least one of A, B, or C" means A or B or C or AB or AC or BC or ABC (i.e., A and B and C). Also, the phrase "example of" as used herein means "example, but not the only example of."

[0215] It is also important to note that the systems and methods of the present application can be embodied in a variety of forms without departing from the spirit or essential characteristics thereof. Likewise, the applications is equally applicable to any embodiment or of the systems and methods of the present application.

[0216] Various changes, modifications and improvements in the herein described technologies can be made without departing from the teachings of the teachings defined by the appended claims. Moreover, the scope of the claims of this application is not limited to the specific aspects described above. The specific aspects are described above for illustrative purposes, but one of ordinary skill in the art will appreciate that other aspects can be employed and that the generic principles defined herein can be applied to other aspects without departing from the scope of the application. Accordingly, the appended claims encompass all such aspects and modifications as falling within the scope of the application.

[0217] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other aspects without departing from the scope of the application. Thus, the present application is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0218] The foregoing description has been presented for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of the application to the forms disclosed herein. Although various example aspects and embodiments have been discussed above, those of ordinary skill in the art will appreciate a variety of modifications, alternatives, permutations, additions, and sub-combinations of the described aspects and embodiments.

Claims

1. A voice control method applied to a voice control module, the voice control module being communicatively connected with an image forming apparatus, the method comprising: receiving a voice input from a user; determining whether the voice input is a voice command; and if the voice input is determined to be the voice command, controlling the image forming apparatus based on the voice command. The method comprises: receiving a voice instruction; sending the voice instruction to a server, the server being configured to analyze the voice instruction by a large language model and an instruction data set of an image forming device to generate voice response information; receiving the voice response information; executing a job instruction corresponding to the voice response information.

2. The method of claim 1, wherein, The voice response information comprises execution information. The execution information comprises an instruction function and / or an instruction function combination. The execution of the job instruction corresponding to the voice response information comprises: executing the job instruction corresponding to the instruction function and / or the instruction function combination. The instruction data set comprises instruction functions and / or instruction function combinations supported by the image forming device, the instruction function being a minimum instruction unit executable by the image forming device, and the instruction function combination comprising at least two instruction functions.

3. The method of claim 2, wherein, The execution information further comprises a job parameter, and the execution of the job instruction corresponding to the voice response information comprises: executing the job instruction corresponding to the instruction function and / or the instruction function combination based on the job parameter.

4. The method according to any one of claims 1 to 3, characterized in that, The voice response information comprises interaction information. The execution of the job instruction corresponding to the voice response information comprises: voice outputting the interaction information.

5. The method of claim 4, wherein, The interaction information comprises one or more of the following: first information for human-computer interaction to confirm whether a user accepts a backup function combination; the backup function combination being a backup instruction function combination determined by the server when the instruction data set does not comprise an instruction function and an instruction function combination matching the voice instruction; second information for voice prompting that the image forming device does not support a current job task; third information for voice prompting that a target file indicated by the voice instruction does not exist; fourth information for human-computer interaction to prompt a user to perform a cooperative operation during a job process; fifth information for voice prompting a job status.

6. The method of claim 4, wherein, The method further comprises: receiving a new voice instruction for the interaction information; or receiving touch control information for the interaction information; sending the new voice instruction or the touch control information to the server.

7. The method of claim 5, wherein, The method further comprises: adding the backup function combination to the instruction data set; or adding the backup function combination to the instruction data set in response to receiving a registration request for the instruction function combination. If the voice response information comprises multiple information, the execution of the job instruction corresponding to the voice response information comprises:

8. The method of claim 1, wherein, executing the job instruction corresponding to the multiple information in sequence according to an order of the multiple information in the voice response information; or if the voice response information further comprises an execution order between the multiple information, executing the job instruction corresponding to the multiple information in sequence according to the execution order. The method further comprises: receiving an acquisition request from the server, the acquisition request being configured to request to acquire an instruction data set supported by an image forming device; 9. The method of claim 1, wherein, sending the instruction data set to the server. The method comprises: acquiring a voice instruction; 10.A voice instruction processing method applied to a server, and the method comprises the steps of: ​ ​ The voice instruction is analyzed by a large language model and an instruction data set of the image forming device to generate voice response information; The voice response information is sent; the voice response information is used to instruct the voice control module to execute the job instruction.

11. The method of claim 10, wherein, The voice response information includes execution information. The execution information includes an instruction function and / or an instruction function combination. The instruction data set includes instruction functions and instruction function combinations supported by the image forming device; the instruction function is the smallest instruction unit that can be executed by the image forming device; and the instruction function combination includes at least two instruction functions.

12. The method of claim 10, wherein, The voice response information includes interaction information.

13. The method according to any one of claims 10-12, characterized in that, The voice instruction is analyzed by a large language model and an instruction data set of the image forming device to generate voice response information; The instruction data set of the image forming device that is communicatively connected to the voice control module is obtained; The large language model is called, and the instruction data set and the voice instruction are input into the large language model to obtain the voice response information output by the large language model.

14. The method of claim 13, wherein, The large language model is called, and the instruction data set and the voice instruction are input into the large language model to obtain the voice response information output by the large language model. Based on instruction function knowledge, the voice instruction and the instruction data set are matched; the instruction function knowledge includes instruction functions and the capability boundaries of the instruction functions; If the instruction data set includes instruction functions and / or instruction function combinations that match the voice instruction, the voice response information is generated based on the instruction functions and / or instruction function combinations; If the instruction data set does not include instruction functions and / or instruction function combinations that match the voice instruction, the voice response information is generated based on the instruction function knowledge, the instruction data set, and the voice instruction; the voice response information includes a backup function combination and first information; and the first information is used for human-computer interaction to confirm whether the user accepts the backup function combination.

15. The method of claim 12, wherein, If the voice response information includes the interaction information, after the voice response information is sent, the method further includes: User operation information for the interaction information is obtained; the user operation information includes a new voice instruction or touch information; Based on the user operation information, new voice response information is regenerated and sent.

16. The method according to any one of claims 10-12, characterized in that, The voice instruction is analyzed by a large language model and an instruction data set of the image forming device to generate voice response information; If the voice instruction is audio data and the large language model only supports text input, instruction text corresponding to the voice instruction is obtained, and the instruction text is analyzed by the large language model and the instruction data set to generate the voice response information; Or, If the voice instruction is audio data and the large language model supports audio input, the voice instruction is directly analyzed by the large language model and the instruction data set to generate the voice response information; Or, If the voice instruction is text data, the instruction text is directly parsed by the large language model and the instruction data set to generate the voice response information.

17. The method of claim 16, wherein, The large language model supports audio input, and the large language model includes a first model and a second model. The first model is used for voice recognition of input audio data and outputs text data corresponding to the audio data. The second model is used for processing input text data and outputting the voice response information.

18. The method according to any one of claims 10-12, characterized by, The parsing processing of the voice instruction by the large language model and the instruction data set of the image forming device includes: The large language model parses the voice instruction based on the historical interaction information of the current task and the instruction data set to generate the voice response information.

19. The method of claim 12, wherein, The interaction information includes one or more of the following: interaction voice, interaction text; The large language model supports audio output; the large language model includes a second model and a third model; wherein the second model is used for processing input text data and outputting the response information; the third model is used for audio generation of input text data and outputs audio data; Or, The large language model supports text output; before sending the voice response information, the method further includes converting the interaction text in the voice response information into interaction voice.

20. A voice control module, communicatively connected to an image forming device, characterized in that, Including: The transceiving unit is configured to receive a voice instruction; The transceiving unit is further configured to send the voice instruction to a server, and the server is configured to parse the voice instruction by a large language model and an instruction data set of an image forming device to generate voice response information; The transceiving unit is further configured to receive the voice response information; The control unit is configured to execute a job instruction corresponding to the voice response information.

21. An image forming apparatus, characterized by comprising: Including: The ontology and the voice control module of claim 20.

22. A voice client, characterized by Including: The voice control module of claim 20; wherein the voice client is in communication connection with the image forming device.

23. A server, characterized by Including: The transceiving unit is configured to receive a voice instruction; The processing unit is configured to parse the voice instruction by a large language model and an instruction data set of an image forming device to generate voice response information; The transceiving unit is further configured to send the voice response information; and the voice response information is used to instruct a voice control module to execute a job instruction.

24. A voice control system, characterized by Including: The image forming device of claim 21; The server of claim 23.

25. A voice control system, characterized by Including: The image forming device; The voice client of claim 22; The server of claim 23.

26. An electronic device comprising a memory, a processor, and a computer program stored on the memory, wherein the computer program comprises instructions that when executed by the processor cause the processor to perform the method of any one of claims 1-25. The processor executes the computer program to implement the method of any one of claims 1-19.

27. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the method of any one of claims 1-19.

Images