Control method and device, storage medium and electronic equipment

Abstract

The application provides a control method, device, storage medium and electronic equipment, wherein the method comprises: in response to a labeling operation on an explanatory picture, determining a labeled picture corresponding to the labeling operation, the explanatory picture being used to describe control explanation information of the device; based on a large language model, performing intent understanding on the labeling operation in the labeled picture to obtain instruction text corresponding to the labeling operation; and based on a control instruction corresponding to the instruction text, performing device control. The method, device, storage medium and electronic equipment provided by the application can realize device control by only performing a labeling operation on a picture, without the need to refer to the picture to find and position an entity operation component, thereby greatly saving the time required for device control, avoiding operation failure caused by complex operation, and optimizing user experience.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology, and in particular to a control method, device, storage medium, and electronic device. Background Technology

[0002] With the development of artificial intelligence technology, intelligent question-answering systems have emerged and are widely used in scenarios where user manuals for various devices are consulted.

[0003] Users can input questions about device usage into the intelligent Q&A system via voice or text. The system can then answer the questions and present detailed answers to users in a combination of text and images.

[0004] However, for some users who are not familiar with the device itself, even if they receive answers in the form of a combination of text and images, they still need to spend some time finding the location indicated by the image on the device to perform the corresponding control operations, which is time-consuming and laborious. Summary of the Invention

[0005] This invention provides a control method, device, storage medium, and electronic device to overcome the time-consuming and labor-intensive drawbacks of manual control in related technologies.

[0006] This invention provides a control method, comprising:

[0007] In response to a labeling operation on an explanatory image, a labeling image corresponding to the labeling operation is determined, wherein the explanatory image is used to describe control instruction information of the device;

[0008] Based on a large language model, the intent of the annotation operations in the annotated image is understood to obtain the instruction text corresponding to the annotation operations.

[0009] The device is controlled based on the control instructions corresponding to the instruction text.

[0010] According to a control method provided by the present invention, the step of performing intent understanding on annotation operations in the annotated image based on a large language model to obtain the instruction text corresponding to the annotation operations includes:

[0011] Based on the large language model, combined with the labeled images, and explanatory text and / or question text, the intent of the labeling operation is understood to obtain the instruction text corresponding to the labeling operation;

[0012] The explanatory text is text corresponding to the explanatory image and used to describe the control instructions for the device.

[0013] The question text refers to the text that forms a question-and-answer pair with the explanatory image in a question-and-answer scenario.

[0014] According to a control method provided by the present invention, the step of performing intent understanding on the annotation operation based on the large language model, combined with the annotated image, and explanatory text and / or question text, to obtain the instruction text corresponding to the annotation operation includes:

[0015] The labeled image, the explanatory text and / or the question text are combined with the prompt template to obtain an intent understanding prompt, which is used to instruct the large language model to perform intent understanding;

[0016] The intent-understanding prompt is input into the large language model to obtain the instruction text corresponding to the annotation operation output by the large language model.

[0017] According to a control method provided by the present invention, the prompt template includes intent description text for the operation form of the annotation operation.

[0018] According to a control method provided by the present invention, before determining the annotation image corresponding to the annotation operation, the method further includes:

[0019] Obtain the question text, and determine the relevant knowledge information that matches the question text from the candidate knowledge information of the device;

[0020] Based on the relevant knowledge information, the explanatory image and the explanatory text are generated and displayed.

[0021] According to a control method provided by the present invention, the step of controlling the device based on the control command corresponding to the instruction text further includes:

[0022] From each candidate standard instruction, determine the target standard instruction that matches the instruction text;

[0023] The target standard instruction is semantically understood to obtain the control instruction corresponding to the instruction text.

[0024] The present invention also provides a control device, comprising:

[0025] An operation response unit is used to respond to an annotation operation on an image and determine the operation information of the annotation operation, wherein the image is used to describe the control instructions of the device;

[0026] The intent understanding unit is used to understand the intent of the labeled operation based on a large language model, combined with the operation information and the image, and obtain the instruction text corresponding to the labeled operation.

[0027] The instruction control unit is used to control the device based on the control instructions corresponding to the instruction text.

[0028] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the control method described above.

[0029] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the control method as described above.

[0030] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the control method as described above.

[0031] The control method, device, storage medium, and electronic device provided by this invention utilize a large language model to understand the intent of annotation operations based on annotated images, thereby obtaining the instruction text corresponding to the annotation operations. Device control is then performed based on the control instructions corresponding to the instruction text. In this process, users only need to perform annotation operations on the image to achieve device control, without having to search for and locate the actual operation parts by referring to the image. This greatly saves the time required for device control, avoids operation failures due to operational complexity, and optimizes the user experience. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in this invention or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 This is a diagram of the display interface of a car assistant in related technologies.

[0034] Figure 2 This is one of the flowcharts illustrating the control method provided by the present invention.

[0035] Figure 3 These are the labeled images provided by this invention.

[0036] Figure 4 This is the second flowchart of the control method provided by the present invention.

[0037] Figure 5 This is a schematic diagram of the control device provided by the present invention.

[0038] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0040] With the development of artificial intelligence technology, intelligent question-answering systems have emerged and are widely used in scenarios involving searching user manuals for various devices. Users can input their questions about device usage into the intelligent question-answering system via voice or text, and the system can answer the questions and display detailed answers to users in a combination of text and images.

[0041] For example, in the automotive field, an intelligent question-and-answer system could be a car assistant. This assistant could integrate voice recognition and human-computer interaction technologies to help drivers and passengers resolve car-related issues. For instance, if a front passenger wants to adjust the seat but is unsure how, they can ask the car assistant, "How do I adjust the seat?" The assistant can then provide guidance such as... Figure 1 The document includes detailed text and illustrations explaining seat adjustments. This illustrated approach effectively guides passengers through seat adjustments and other operations.

[0042] However, despite the relatively comprehensive information provision capabilities of car assistants, some issues remain regarding their interaction methods and user experience. Particularly for users unfamiliar with the car's internal structure and operating procedures, even with detailed explanations and clear images, users still need to spend time comparing and locating the corresponding components or controls within the vehicle during actual operation. This step not only increases the complexity of the operation but may also lead to failure due to misjudgment or omission.

[0043] Therefore, how to optimize the interaction process to quickly meet user needs remains an urgent problem to be solved.

[0044] Figure 2 This is one of the flowcharts illustrating the control method provided by the present invention, such as... Figure 2 As shown, this control method can be applied to various intelligent question-answering systems, as well as other systems that require assistance in user operation. The method includes:

[0045] Step 210: In response to the annotation operation on the explanatory image, determine the annotation image corresponding to the annotation operation, wherein the explanatory image is used to describe the control instruction information of the device.

[0046] The "device" here refers to the device that needs to be controlled, such as a vehicle, large machinery, a smartphone, a tablet, or a smart home appliance. This embodiment of the invention does not specifically limit this. The executing entity of this embodiment, i.e., the system communicating with the device, can specifically be a system installed inside the device, such as a car assistant in a vehicle.

[0047] For equipment, there may be images used to describe the control information of the equipment, i.e., explanatory images. The control information here refers to the relevant explanatory information for implementing equipment control, such as the content of an instruction document or information on various knowledge points broken down from a manual. Explanatory images are visual descriptions of the control information, such as schematic diagrams of structures mentioned in the control information, or illustrations of operating positions mentioned in the control information.

[0048] To facilitate user control of the device, explanatory images describing the device's control instructions can be displayed. After viewing these images, users can annotate them, thus inputting their control intentions into the system through these annotations.

[0049] Here, the annotation operation refers to the use of explanatory images, specifically highlighting specific content within them. This specific content refers to elements in the explanatory image that are relevant to the user's intended control over the device. Annotation operations can manifest as selecting, clicking, pointing with arrows, or underlining specific content within the explanatory image.

[0050] The system can receive annotation operations on explanatory images and respond to these operations. Specifically, the response includes identifying the corresponding annotation image, which is an image overlaid with the operation information of the annotation operation on top of the explanatory image.

[0051] Here, regarding the annotation operation, the operation information may include the position of the annotation operation on the explanatory image and the operation form of the annotation operation. This embodiment of the invention does not specifically limit this.

[0052] It is understandable that the position of the annotation operation on the explanatory image in the operation information can reflect the specific content marked by the annotation operation, that is, it can reflect the module or object that needs to be controlled through the annotation operation; the operation form of the annotation operation can be circle selection, click, arrow pointing, underline, etc., and different operation forms can correspond to different operation intentions.

[0053] Step 220: Based on a large language model, perform intent understanding on the annotation operations in the annotated image to obtain the instruction text corresponding to the annotation operations.

[0054] Specifically, after obtaining the labeled images, they can be input into a large language model. Here, a large language model (LLM) is a natural language processing (NLP) model with a massive number of parameters. The number of parameters and / or the complexity of the model structure exceed a preset threshold. During training, the model processes large-scale text data and possesses the ability to understand and generate natural language. For example, large language models can include the Spark large model, etc.

[0055] Furthermore, the large language model here can be a general-domain large language model, or a large language model obtained by fine-tuning data from the device control domain. For example, the large language model can be obtained by fine-tuning a pre-trained large language model using various sample labeled images and corresponding labeled instruction texts; this embodiment of the invention does not specifically limit this.

[0056] In this embodiment of the invention, the large language model is a large language model capable of processing data of multiple modalities, which specifically include text modality and image modality. Therefore, for annotated images input into the large language model, the large language model can understand the intent of the annotation operation based on the information in the image modality, thereby representing the control intent reflected by the user's annotation operation on the explanatory image in the text modality of natural language, i.e., obtaining the instruction text. Accordingly, the system can obtain the instruction text corresponding to the annotation operation output by the large language model; for example, the instruction text could be "The user wants to adjust the seat leg rest."

[0057] Step 230: Perform device control based on the control instructions corresponding to the instruction text.

[0058] Specifically, after obtaining the instruction text, the instruction text can be converted into control instructions that the device can support, and the device can be controlled based on the control instructions, thereby completing the device control that conforms to the control intent reflected by the user's annotation operation.

[0059] In the method provided in this embodiment of the invention, a large language model performs intent understanding on the annotation operation based on the annotated image, thereby obtaining the instruction text corresponding to the annotation operation, and controlling the device based on the control instructions corresponding to the instruction text. In this process, the user only needs to perform the annotation operation on the image to achieve device control, without having to look up and locate the physical operation parts in the image, which greatly saves the time required for device control, avoids operation failure due to operation complexity, and optimizes the user experience.

[0060] Based on the above embodiments, step 220, which involves performing intent understanding on the annotation operations in the annotated image based on a large language model to obtain the instruction text corresponding to the annotation operations, includes:

[0061] Based on the large language model, combined with the labeled images, and explanatory text and / or question text, the intent of the labeling operation is understood to obtain the instruction text corresponding to the labeling operation;

[0062] The explanatory text is text corresponding to the explanatory image and used to describe the control instructions for the device.

[0063] The question text refers to the text that forms a question-and-answer pair with the explanatory image in a question-and-answer scenario.

[0064] Specifically, when performing intent understanding on annotation operations based on large-scale language models, in addition to using annotated images, explanatory text and / or question text can also be used to assist in intent understanding.

[0065] The explanatory text is the text that matches the explanatory images. The explanatory text is a textual description of the control instructions, and the control instructions described in the explanatory text are consistent with those described in the explanatory images. That is, the explanatory text and explanatory images can be text and images that describe the control instructions of the device in a combined text and image format.

[0066] In a question-and-answer scenario, the question text, along with the explanatory image, forms a question-and-answer pair. That is, in a question-and-answer context, the intelligent question-and-answer system can answer the user's question text and output an explanatory image. Here, the question text is the question in the question-and-answer pair, and the explanatory image is the answer. Additionally, some intelligent question-and-answer systems can output both explanatory text and an explanatory image for the user's question text, thus providing a more visually appealing answer.

[0067] Understandably, when performing intent understanding for labeled operations, the labeled images, along with explanatory text and / or question text, can be input into a large language model. This allows the large language model to reference control information in the explanatory text and / or user needs reflected in the question text when performing intent understanding for labeled operations, thus ensuring the richness of the information used for intent understanding and improving its reliability and accuracy.

[0068] Based on any of the above embodiments, in step 220, the step of performing intent understanding on the annotation operation based on the large language model, combined with the annotated image, and explanatory text and / or question text, to obtain the instruction text corresponding to the annotation operation, includes:

[0069] The labeled image, the explanatory text and / or the question text are combined with the prompt template to obtain an intent understanding prompt, which is used to instruct the large language model to perform intent understanding;

[0070] The intent-understanding prompt is input into the large language model to obtain the instruction text corresponding to the annotation operation output by the large language model.

[0071] Specifically, a prompt template is a pre-constructed prompt (Promot) used to achieve intent understanding through a large language model. The prompt template, in the form of a natural language model, instructs the large language model to perform intent understanding on the labeled operation and output the instruction text. For example, a prompt template could be: "You are a professional user intent analysis expert. Based on the following text description and the annotations in the image, please analyze what operation the user wants to perform on the device (provide a clear predicted instruction, such as: 'The user wants to close the car window')." Text description: "Quoting explanatory text," Image: "Applying labeled image."

[0072] After obtaining the labeled images, explanatory text, and / or question text, these images and text can be combined with the prompt template to form an intent-understanding prompt. The resulting intent-understanding prompt, which serves as the input to the large-scale speech model, can encompass the instructions reflected in the prompt template, as well as the labeled images, explanatory text, and / or question text.

[0073] By inputting intent-understanding prompts into a large language model, the large language model can understand the intent of the annotation operation according to the requirements of the prompt template in the intent-understanding prompt, based on the annotated image, explanatory text and / or question text in the intent-understanding prompt, thereby generating instruction text that conforms to the output format required by the prompt template.

[0074] Based on any of the above embodiments, the prompt template includes intent description text for the operation form of the annotation operation.

[0075] Specifically, annotation operations can take various forms, such as selection, clicking, pointing with an arrow, and underlining. Different forms of operations correspond to different operational intentions. To help large language models better understand the operational intentions reflected by different forms of operations, descriptive text describing the intentions of different forms of operations can be added to the prompt template.

[0076] Here, for any operation form, the intent description text for that operation form is used to describe the user's operation intent reflected by that operation form. For example, the intent description text for circling could be "The large circle on the image is the module the user wants to operate on," and the intent description text for clicking could be "The dot on the image is the specific action the user wants to perform."

[0077] In this embodiment of the invention, intent description texts of different operation forms are added to the prompt template, so that when a large language model performs intent understanding on labeled operations, it can more accurately understand the operation intent reflected by the operation form of the labeled operation, thereby more reasonably and reliably inferring the control intent reflected by the labeled operation itself, and thus ensuring the accuracy of intent understanding.

[0078] For example, Figure 3 These are the labeled images provided by this invention. Figure 3 The red ink marks represent the operation information superimposed on the explanatory image to indicate the operation.

[0079] Based on this, the labeled images, explanatory text, and prompt templates can be combined to obtain the following intention-based prompts:

[0080] You are a professional user intent analysis expert. Based on the text description below and the annotations in the image, please analyze what kind of operation the user wants to perform on the device (provide a clear predicted instruction, such as: "The user wants to close the car window"). The image is "referenced image".

[0081] Image caption:

[0082] The large red circle in the image represents the module the user wants to interact with, and the red dots represent the specific actions the user wants to perform.

[0083] Text description:

[0084] 1. Move the seat forward, backward, or backward; 2. Adjust the seat tilt; 3. Raise or lower the seat height; 4. Adjust the seat back tilt; 5. Move the buttons up or down to adjust the seat position.

[0085] Based on any of the above embodiments, before step 210, the method further includes:

[0086] Obtain the question text, and determine the relevant knowledge information that matches the question text from the candidate knowledge information of the device;

[0087] Based on the relevant knowledge information, the explanatory image and the explanatory text are generated and displayed.

[0088] Specifically, a knowledge-based question-and-answer session can be conducted before equipment control is implemented.

[0089] That is, the question text can be obtained first. The question text here can be the text directly entered by the user to ask the question, or it can be the text obtained by transcribing the user's voice of the question. This embodiment of the invention does not make specific limitations on this.

[0090] After obtaining the question text, it can be matched against each of the pre-acquired candidate knowledge information for the device. Here, for a single device, there can be multiple candidate knowledge information points. These candidate knowledge points are information used to instruct on the use of the device, such as knowledge points compiled from the user manual. Specifically, when matching the question text with each candidate knowledge information point, the correlation between the vector representation of the question text and the vector representation of each candidate knowledge information point can be calculated. Based on this correlation, relevant knowledge information matching the question text can be determined. For example, candidate knowledge information with a correlation higher than a preset threshold can be used as relevant knowledge information matching the question text, or the correlation can be arranged in descending order, and the candidate knowledge information corresponding to the top N correlations can be used as relevant knowledge information matching the question text. This embodiment of the invention does not specifically limit this approach.

[0091] Once the relevant knowledge information is obtained, an answer to the question text can be generated based on that knowledge information. This answer can include explanatory images and explanatory text. Furthermore, the generation of the answer to the question text can be achieved through a question-answering model, or the question text and relevant knowledge information can be input together into a large language model, which will then refer to the relevant knowledge information to generate explanatory images and explanatory text corresponding to the question text.

[0092] Based on any of the above embodiments, before step 230, the method further includes:

[0093] From the candidate standard instructions, determine the target standard instruction that matches the instruction text;

[0094] The target standard instruction is semantically understood to obtain the control instruction corresponding to the instruction text.

[0095] Specifically, after obtaining the instruction text, it can be matched with the pre-acquired candidate standard instructions. These candidate standard instructions are standardized instruction texts pre-set for the device. Compared to the text instructions output by a large language model, candidate standard instructions are more standardized at the instruction level and easier for the semantic understanding system to understand. Therefore, after obtaining the instruction text, the candidate standard instruction that matches the instruction text can be determined from among the candidate standard instructions; this is denoted as the target standard instruction.

[0096] Furthermore, when matching the instruction text with each candidate standard instruction, the correlation between the vector representation of the instruction text and the vector representation of each candidate standard instruction can be calculated. Based on this correlation, the target standard instruction that matches the instruction text can be determined. For example, candidate standard instructions with a correlation higher than a preset threshold can be used as the target standard instruction that matches the instruction text, or the candidate standard instruction with the highest correlation can be used as the target standard instruction that matches the instruction text. This embodiment of the invention does not specifically limit this approach.

[0097] After obtaining the target standard instruction, it can be input into the semantic understanding model for semantic parsing, thereby obtaining the control instructions that the device can execute, which correspond to the target standard instruction, output by the semantic understanding model.

[0098] Based on any of the above embodiments Figure 4 This is the second flowchart illustrating the control method provided by the present invention, as shown below. Figure 4 As shown, the method includes:

[0099] For the equipment to be controlled, the user manual can be pre-compiled, and the knowledge points involved in the manual can be arranged as candidate knowledge information, such as candidate knowledge information 1, 2, 3, etc. in the diagram. Vector encoding can be performed on each candidate knowledge information to obtain the vector representation of each candidate knowledge information, thereby constructing a vector knowledge base.

[0100] Furthermore, for semantic understanding systems, a set of candidate standard instructions supported by the semantic understanding system can be obtained, and each candidate standard instruction can be vector-encoded to obtain a vector representation of each candidate standard instruction, thereby constructing a standard instruction vector library.

[0101] First, the system can receive the question text input by the user. Then, the question text can be vectorized to obtain its vector representation. This vector representation is then input into the retrieval model, which retrieves relevant knowledge information matching the question text from a vector knowledge base. This relevant knowledge information, along with the question text, is then input into the large-scale inference and response model, which outputs the answer, specifically including an explanatory image and explanatory text. Here, the large-scale inference and response model is a large-scale language model used in the question scenario.

[0102] Subsequently, explanatory images and text can be displayed. Furthermore, user annotations on the explanatory images can be received, resulting in an annotated image. The annotated image and explanatory text can then be input into a multimodal large-scale model. This multimodal large-scale model is a large-scale language model capable of processing data from multiple modalities. Based on the annotated image and explanatory text, the multimodal large-scale model can understand the intent of the annotation actions and output instruction text.

[0103] The instruction text can be input into the retrieval model, which then retrieves candidate standard instructions that match the instruction text from the standard instruction vector library as target standard instructions. These target standard instructions are then input into the semantic understanding system to obtain control instructions, which are then sent to the device, thereby achieving device control.

[0104] The control device provided by the present invention is described below. The control device described below can be referred to in correspondence with the control method described above.

[0105] Figure 5 This is a schematic diagram of the control device provided by the present invention. Figure 5 As shown, the device includes:

[0106] The operation response unit 510 is configured to respond to an annotation operation on an image and determine the operation information of the annotation operation, wherein the image is used to describe the control instructions of the device;

[0107] The intent understanding unit 520 is used to understand the intent of the labeled operation based on a large language model, combined with the operation information and the image, and obtain the instruction text corresponding to the labeled operation.

[0108] The instruction control unit 530 is used to control the device based on the control instructions corresponding to the instruction text.

[0109] In the device provided in this embodiment of the invention, a large language model performs intent understanding on the annotation operation based on the annotated image, thereby obtaining the instruction text corresponding to the annotation operation, and controlling the device based on the control instructions corresponding to the instruction text. In this process, the user only needs to perform the annotation operation on the image to achieve device control, without having to look up and locate the physical operation parts in the image, which greatly saves the time required for device control, avoids operation failure due to operation complexity, and optimizes the user experience.

[0110] Based on any of the above embodiments, the intent understanding unit is specifically used for:

[0111] Based on the large language model, combined with the labeled images, and explanatory text and / or question text, the intent of the labeling operation is understood to obtain the instruction text corresponding to the labeling operation;

[0112] The explanatory text is text corresponding to the explanatory image and used to describe the control instructions for the device.

[0113] The question text refers to the text that forms a question-and-answer pair with the explanatory image in a question-and-answer scenario.

[0114] Based on any of the above embodiments, the intent understanding unit is specifically used for:

[0115] The labeled image, the explanatory text and / or the question text are combined with the prompt template to obtain an intent understanding prompt, which is used to instruct the large language model to perform intent understanding;

[0116] The intent-understanding prompt is input into the large language model to obtain the instruction text corresponding to the annotation operation output by the large language model.

[0117] Based on any of the above embodiments, the prompt template includes intent description text for the operation form of the annotation operation.

[0118] Based on any of the above embodiments, the device further includes a question-and-answer unit, used for:

[0119] Obtain the question text, and determine the relevant knowledge information that matches the question text from the candidate knowledge information of the device;

[0120] Based on the relevant knowledge information, the explanatory image and the explanatory text are generated and displayed.

[0121] Based on any of the above embodiments, the device further includes an instruction conversion unit, used for:

[0122] From the candidate standard instructions, determine the target standard instruction that matches the instruction text;

[0123] The target standard instruction is semantically understood to obtain the control instruction corresponding to the instruction text.

[0124] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include: a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communications interface 620, and the memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a control method, which includes:

[0125] In response to a labeling operation on an explanatory image, a labeling image corresponding to the labeling operation is determined, wherein the explanatory image is used to describe control instruction information of the device;

[0126] Based on a large language model, the intent of the annotation operations in the annotated image is understood to obtain the instruction text corresponding to the annotation operations.

[0127] The device is controlled based on the control instructions corresponding to the instruction text.

[0128] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to related technologies, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0129] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program, the computer program being able to be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer being able to execute the control methods provided by the above methods, the method comprising:

[0130] In response to a labeling operation on an explanatory image, a labeling image corresponding to the labeling operation is determined, wherein the explanatory image is used to describe control instruction information of the device;

[0131] Based on a large language model, the intent of the annotation operations in the annotated image is understood to obtain the instruction text corresponding to the annotation operations.

[0132] The device is controlled based on the control instructions corresponding to the instruction text.

[0133] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the control methods provided by the methods described above, the method comprising:

[0134] In response to a labeling operation on an explanatory image, a labeling image corresponding to the labeling operation is determined, wherein the explanatory image is used to describe control instruction information of the device;

[0135] Based on a large language model, the intent of the annotation operations in the annotated image is understood to obtain the instruction text corresponding to the annotation operations.

[0136] The device is controlled based on the control instructions corresponding to the instruction text.

[0137] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0138] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0139] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A control method, characterized in that, include: In response to a labeling operation on an explanatory image, a labeling image corresponding to the labeling operation is determined, wherein the explanatory image is used to describe control instruction information of the device; Based on a large-scale language model, combined with the labeled images, explanatory text, and / or question text, the intent of the labeled operations in the labeled images is understood to obtain the instruction text corresponding to the labeled operations; the explanatory text is the text corresponding to the explanatory image that describes the control instructions of the device, and the question text is the text that forms a question-answer pair with the explanatory image in a question-answering scenario; Based on the control instructions corresponding to the instruction text, the device is controlled; Before determining the labeled image corresponding to the labeling operation, the process also includes: Obtain the question text, and determine the relevant knowledge information that matches the question text from the candidate knowledge information of the device; Based on the relevant knowledge information, the explanatory image and the explanatory text are generated and displayed.

2. The control method according to claim 1, characterized in that, Based on the large language model, combined with the labeled images, and explanatory text and / or question text, the intent understanding of the labeled operations in the labeled images is performed to obtain the instruction text corresponding to the labeled operations, including: The labeled image, the explanatory text and / or the question text are combined with the prompt template to obtain an intent understanding prompt, which is used to instruct the large language model to perform intent understanding; The intent-understanding prompt is input into the large language model to obtain the instruction text corresponding to the annotation operation output by the large language model.

3. The control method according to claim 2, characterized in that, The prompt template includes an intentional description text for the operation form of the annotation operation.

4. The control method according to any one of claims 1 to 3, characterized in that, The process of controlling the device based on the control command corresponding to the instruction text also includes, prior to: From each candidate standard instruction, determine the target standard instruction that matches the instruction text; The target standard instruction is semantically understood to obtain the control instruction corresponding to the instruction text.

5. A control device, characterized in that, include: An operation response unit is configured to respond to a labeling operation on an explanatory image and determine the labeling image corresponding to the labeling operation, wherein the explanatory image is used to describe control instruction information of the device; The intent understanding unit is used to understand the intent of the labeled operation in the labeled image based on a large language model, combined with the labeled image, explanatory text and / or question text, to obtain the instruction text corresponding to the labeled operation; the explanatory text is text corresponding to the explanatory image that describes the control instructions of the device, and the question text is text that forms a question-answer pair with the explanatory image in a question-answering scenario; The instruction control unit is used to control the device based on the control instructions corresponding to the instruction text; The operation response unit is also used for: Obtain the question text, and determine the relevant knowledge information that matches the question text from the candidate knowledge information of the device; Based on the relevant knowledge information, the explanatory image and the explanatory text are generated and displayed.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the control method as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the control method as described in any one of claims 1 to 4.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the control method as described in any one of claims 1 to 4.

Images