Control method and device, storage medium and electronic equipment

Abstract

The invention provides a control method and device, a storage medium and electronic equipment, and the method comprises the steps: responding to an annotation operation on an instruction picture, determining an annotation picture corresponding to the annotation operation, and enabling the instruction picture to be used for describing the control instruction information of the equipment; on the basis of a large language model, intention understanding is carried out on annotation operation in the annotation picture, and an instruction text corresponding to the annotation operation is obtained; and performing equipment control based on a control instruction corresponding to the instruction text. According to the method, the device, the storage medium and the electronic equipment provided by the invention, a user only needs to execute the labeling operation on the picture to realize equipment control, and does not need to look up and position an entity operation component according to the picture, so that the time required for equipment control is greatly saved, the condition of operation failure caused by complex operation is avoided, and the user experience is optimized.

Description

Technical Field

[0001] The present invention relates to the field of human-computer interaction technology, and in particular to a control method, device, storage medium and electronic equipment. Background Art

[0002] With the development of artificial intelligence technology, intelligent question-answering systems have emerged and have been widely used in query scenarios of user manuals of various types of equipment.

[0003] Users can input questions about the use of the device into the intelligent question-and-answer system in the form of voice or text. The intelligent question-and-answer system can answer the questions and show the users detailed answers in the form of a combination of text and pictures.

[0004] However, for some users who are not familiar with the device itself, even if they get the answer in the form of a combination of text and pictures, they still need to spend some time looking for the location indicated by the picture on the device to perform the corresponding control operation, which is time-consuming and laborious. Summary of the invention

[0005] The present invention provides a control method, a device, a storage medium and an electronic device, which are used to solve the defects of manual control in the related art that is time-consuming and labor-intensive.

[0006] The present invention provides a control method, comprising: In response to a marking operation on the explanation picture, determining a marking picture corresponding to the marking operation, wherein the explanation picture is used to describe control explanation information of the device; Based on a large language model, the intention of the annotation operation in the annotated image is understood to obtain the instruction text corresponding to the annotation operation; The device is controlled based on the control instruction corresponding to the instruction text.

[0007] According to a control method provided by the present invention, the intention of the annotation operation in the annotated picture is understood based on a large language model to obtain the instruction text corresponding to the annotation operation, including: Based on the large language model, combined with the annotated image, and the description text and / or the question text, the intention of the annotation operation is understood to obtain the instruction text corresponding to the annotation operation; The description text is a text corresponding to the description picture and used to describe the control description information of the device; The question text is the text that forms a question-answer pair with the explanatory image in a question-answer scenario.

[0008] According to a control method provided by the present invention, the intention of the annotation operation is understood based on the large language model, combined with the annotated picture, and the description text and / or the question text, to obtain the instruction text corresponding to the annotation operation, including: Combining the annotated image, the description text and / or the question text with a prompt template to obtain an intent understanding prompt, wherein the prompt template is used to instruct the large language model to perform intent understanding; The intention understanding prompt is input into the large language model to obtain the instruction text corresponding to the annotation operation output by the large language model.

[0009] According to a control method provided by the present invention, the prompt template includes a text describing the intention of the operation form of the annotation operation.

[0010] According to a control method provided by the present invention, the step of determining the annotated image corresponding to the annotating operation further includes: Acquire the question text, and determine relevant knowledge information matching the question text from each candidate knowledge information of the device; Based on the relevant knowledge information, the explanatory image and the explanatory text are generated and displayed.

[0011] According to a control method provided by the present invention, the device control is performed based on the control instruction corresponding to the instruction text, and the method also includes: Determine a target standard instruction matching the instruction text from each candidate standard instruction; The target standard instruction is semantically understood to obtain a control instruction corresponding to the instruction text.

[0012] The present invention also provides a control device, comprising: an operation response unit, configured to determine operation information of the marking operation in response to a marking operation on the picture, wherein the picture is used to describe control description information of the device; An intention understanding unit, configured to understand the intention of the annotation operation based on a large language model in combination with the operation information and the image, and obtain an instruction text corresponding to the annotation operation; The instruction control unit is used to control the device based on the control instruction corresponding to the instruction text.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the control methods described above when executing the program.

[0014] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the control method described above is implemented.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the control methods described above.

[0016] The control method, device, storage medium and electronic device provided by the present invention use a large language model to understand the intention of the marked operation based on the marked picture, thereby obtaining the instruction text corresponding to the marked operation, and controlling the device based on the control instruction corresponding to the instruction text. In this process, the user only needs to perform the marked operation on the picture to achieve device control, and there is no need to find and locate the physical operating components by comparing the picture, which greatly saves the time required for device control, avoids the operation failure due to complex operation, and optimizes the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 This is a display interface diagram of a car assistant in the related art.

[0019] Figure 2 It is one of the flow charts of the control method provided by the present invention.

[0020] Figure 3 It is the labeled picture provided by the present invention.

[0021] Figure 4 This is the second flow chart of the control method provided by the present invention.

[0022] Figure 5 It is a structural schematic diagram of the control device provided by the present invention.

[0023] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] With the development of artificial intelligence technology, intelligent question-answering systems have emerged and are widely used in the query scenarios of user manuals of various devices. Users can input questions about the use of the device into the intelligent question-answering system in the form of voice or text. The intelligent question-answering system can answer the questions and show the detailed answers to users in the form of a combination of text and pictures.

[0026] For example, in the field of vehicles, the intelligent question-answering system can be a car assistant, which can integrate voice recognition and human-computer interaction technology to help drivers and passengers solve car problems. For example, if the front passenger wants to adjust the seat but is not sure how to do it, the front passenger can ask the car assistant "how to adjust the seat" and the car assistant can give a correct answer. Figure 1 The detailed seat adjustment instructions and pictures are shown in the figure. This illustrated answer method can effectively guide passengers to complete seat adjustment and other operations.

[0027] However, although the car assistant is quite complete in providing information, there are still some issues in its interaction mode and user experience that cannot be ignored. Especially for users who are not familiar with the internal structure and operation process of the car, even if the car assistant provides detailed instructions and clear pictures to answer questions, in the actual positioning and operation process, users still need to spend a certain amount of time to compare and find the corresponding parts or control devices in the car. This step not only increases the complexity of the operation, but may also lead to operation failure due to misjudgment or omission.

[0028] Therefore, how to optimize the interaction process to quickly meet user needs remains a problem that needs to be solved urgently.

[0029] Figure 2 It is one of the flow charts of the control method provided by the present invention, such as Figure 2 As shown, the control method can be applied to various intelligent question-answering systems, and can also be applied to other systems that need to assist users in operating. The method includes: Step 210: In response to a marking operation on an explanation picture, a marking picture corresponding to the marking operation is determined, where the explanation picture is used to describe control explanation information of a device.

[0030] The device here refers to the device that needs to be controlled, such as a vehicle, a large machine, a smart phone, a tablet computer, a smart home appliance, etc., which is not specifically limited in the embodiment of the present invention. The execution subject of the embodiment of the present invention, that is, the system that communicates with the device, can be a system set inside the device, such as a car assistant on a vehicle.

[0031] For a device, there may be a picture used to describe the control description information of the device, i.e., an explanation picture. The control description information here refers to the relevant explanation information for implementing device control, such as the content of an explanation document, or the various knowledge points separated from the manual. The explanation picture is a graphical description of the control description information, such as a schematic diagram of the structure mentioned in the control description information, a diagram of the operation position mentioned in the control description information, etc.

[0032] In order to facilitate the user to control the device, the above-mentioned description picture describing the control information of the device can be displayed. After viewing the description picture, the user can mark the description picture, thereby inputting his control intention for the device to the system in the form of a marking operation.

[0033] Here, the annotation operation refers to the description picture, specifically, the operation of highlighting specific content in the description picture. The specific content referred to here is the content in the description picture that is associated with the user's control intention for the device. The annotation operation can be specifically embodied in circling, clicking, pointing with an arrow, underlining, etc., for specific content in the description picture.

[0034] The system can receive the annotation operation on the explanation picture and respond to the annotation operation. The specific response method includes determining the annotation picture corresponding to the annotation operation, and the annotation picture is a picture with the operation information of the annotation operation superimposed on the explanation picture.

[0035] Here, for the annotation operation, the operation information may include the position of the annotation operation on the description picture and the operation form of the annotation operation, which is not specifically limited in the embodiment of the present invention.

[0036] It can be understood that, in the operation information, the position of the annotation operation on the explanatory picture can reflect the specific content identified by the annotation operation, that is, it can reflect the module or object that needs to be controlled through the annotation operation; the operation form of the annotation operation can be circling, clicking, arrow pointing, underlining, etc., and different operation forms can correspond to different operation intentions.

[0037] Step 220: Based on the large language model, the intention of the annotation operation in the annotated image is understood to obtain the instruction text corresponding to the annotation operation.

[0038] Specifically, after obtaining the annotated image, the annotated image can be input into a large language model. Here, a large language model (LLM) is referred to as a large model, which refers to a natural language processing (NLP) model with a large parameter scale. The number of model parameters and / or the complexity of the model structure exceeds a preset threshold. The model processes large-scale text data during training and has the ability to understand and generate natural language. For example, a large language model can include the Spark large model, etc.

[0039] Furthermore, the large language model here can be a large language model in a general field, or a large language model obtained by fine-tuning data in the field of device control. For example, a large language model can be obtained by fine-tuning a pre-trained large language model using various sample annotated images and corresponding label instruction texts, and the embodiments of the present invention do not specifically limit this.

[0040] In an embodiment of the present invention, the large language model is a large language model that can process data in multiple modes, and the multiple modes referred to here may specifically include text mode and picture mode. Therefore, for the annotated picture input into the large language model, the large language model can understand the intention of the annotation operation based on the information of the picture mode, thereby expressing the control intention reflected by the user's annotation operation on the explanatory picture in the text mode of natural language, that is, obtaining the instruction text. Accordingly, the system can obtain the instruction text corresponding to the annotation operation output by the large language model, for example, the instruction text may be "the user wants to adjust the seat leg rest."

[0041] Step 230: Control the device based on the control instruction corresponding to the instruction text.

[0042] Specifically, after obtaining the instruction text, the instruction text can be converted into control instructions that can be supported by the device, and the device can be controlled based on the control instructions, thereby completing the device control that meets the control intention reflected by the user's annotation operation.

[0043] In the method provided in the embodiment of the present invention, a large language model understands the intention of the marked operation based on the marked image, thereby obtaining the instruction text corresponding to the marked operation, and performing device control based on the control instruction corresponding to the instruction text. In this process, the user only needs to perform the marked operation on the image to achieve device control, and there is no need to find and locate the physical operating components by comparing the image, which greatly saves the time required for device control, avoids operation failure due to complex operation, and optimizes the user experience.

[0044] Based on the above embodiment, in step 220, the intention of the annotation operation in the annotated image is understood based on the large language model to obtain the instruction text corresponding to the annotation operation, including: Based on the large language model, combined with the annotated image, and the description text and / or the question text, the intention of the annotation operation is understood to obtain the instruction text corresponding to the annotation operation; The description text is a text corresponding to the description picture and used to describe the control description information of the device; The question text is the text that forms a question-answer pair with the explanatory image in a question-answer scenario.

[0045] Specifically, when the intent of a labeling operation is understood based on a large language model, in addition to applying the labeling images, description text and / or question text may also be applied to assist in the intent understanding.

[0046] The description text is a text that matches the description picture. The description text is a textual description of the control description information, and the control description information described in the description text is consistent with the control description information described in the description picture. That is, the description text and the description picture can be text and pictures that describe the control description information of the device in the form of a combination of text and pictures.

[0047] The question text is the question that forms a question-answer pair with the explanatory image in the question-answer scenario. That is, in the question-answer scenario, the intelligent question-answer system can answer the question text raised by the user and output the explanatory image. Here, the question text is the question in the question-answer pair, and the explanatory image is the answer in the question-answer pair. In addition, in some intelligent question-answering systems, the explanatory text and explanatory image can be output for the question text raised by the user, so as to answer the question in the form of both text and pictures.

[0048] It is understandable that when understanding the intention of the annotation operation, the annotation image, the description text and / or the question text can be input into the large language model together. Therefore, when the large language model understands the intention of the annotation operation, it can refer to the control description information in the description text and / or refer to the user needs reflected in the question text from the perspective of the user's question, thereby ensuring the richness of the information based on the intention understanding and improving the reliability and accuracy of the intention understanding.

[0049] Based on any of the above embodiments, in step 220, the intention of the annotation operation is understood based on the large language model, in combination with the annotated image, and the description text and / or the question text, to obtain the instruction text corresponding to the annotation operation, including: Combining the annotated image, the description text and / or the question text with a prompt template to obtain an intent understanding prompt, wherein the prompt template is used to instruct the large language model to perform intent understanding; The intention understanding prompt is input into the large language model to obtain the instruction text corresponding to the annotation operation output by the large language model.

[0050] Specifically, the prompt template is a template of a pre-constructed prompt (Promot) for realizing intent understanding through a large language model. The prompt template is an instruction in the form of a natural language model that instructs the large language model to understand the intent of the annotated operation to output the instruction text. For example, the prompt template can be: You are a professional user intent analysis expert. Please analyze what kind of operation the user wants to perform on the device based on the text description below and the annotations in the picture (give a clear predicted instruction, such as: "The user wants to close the window"), text description: "Quote the description text", picture: "Apply the annotated picture".

[0051] After obtaining the annotated image, the explanatory text and / or the question text, the above image and text can be combined with the prompt template to form an intention understanding prompt. The intention understanding prompt thus obtained is the input of the large voice model, and the intention understanding prompt can include the instructions reflected by the prompt template, and can also include the annotated image and the explanatory text and / or the question text.

[0052] By inputting the intention understanding prompt into the large language model, the large language model can understand the intention of the annotation operation according to the requirements of the prompt template in the intention understanding prompt, based on the annotated picture and the explanatory text and / or question text in the intention understanding prompt, and thus generate the instruction text in the output form required by the prompt template.

[0053] Based on any of the above embodiments, the prompt template includes a text describing the intended operation form of the annotation operation.

[0054] Specifically, there can be multiple forms of annotation operations, such as circling, clicking, arrow pointing, underlining, etc. Different forms of operation can correspond to different operation intentions. In order to help the large language model better understand the operation intentions reflected by different forms of operation, the intention description text for different forms of operation can be added to the prompt template.

[0055] Here, for any operation form, the intention description text of the operation form is used to describe the user's operation intention reflected by the operation form. For example, the intention description text for circle selection can be "the big circle on the picture is the module the user wants to operate", and the intention description text for click can be "the point on the picture is the specific action the user wants to perform".

[0056] In an embodiment of the present invention, intention description texts of different operation forms are added to the prompt template, so that when the large language model understands the intention of the marked operation, it can more accurately understand the operation intention reflected by the operation form of the marked operation, thereby more reasonably and reliably inferring the control intention reflected by the marked operation itself, thereby ensuring the accuracy of intention understanding.

[0057] For example, Figure 3 This is the labeled picture provided by the present invention. Figure 3 The red handwriting in the figure is the operation information superimposed on the description picture for the marked operation.

[0058] Based on this, the annotated image, the caption text and the prompt template can be combined to obtain the following intent understanding prompt: You are a professional user intent analysis expert. Please analyze what kind of operation the user wants to perform on the device based on the following text description and the annotations in the picture (give a clear prediction instruction, such as "the user wants to close the window"), where the picture: "reference the annotated picture"; Image Description: The big red circle in the picture is the module that the user wants to operate, and the red dot is the specific action that the user wants to perform; Text description: 1. Move the seat forward, backward or forward; 2. Adjust the inclination of the seat surface; 3. Raise or lower the seat height; 4. Adjust the inclination of the seat back; 5. Push the button up or down to adjust the seat.

[0059] Based on any of the above embodiments, before step 210, the method further includes: Acquire the question text, and determine relevant knowledge information matching the question text from each candidate knowledge information of the device; Based on the relevant knowledge information, the explanatory image and the explanatory text are generated and displayed.

[0060] Specifically, before performing device control, knowledge quiz can be performed first.

[0061] That is, the question text may be obtained first. The question text here may be a text directly entered by the user for asking a question, or may be a text obtained by voice transcription of the question voice entered by the user, which is not specifically limited in the embodiment of the present invention.

[0062] After obtaining the question text, the question text can be matched with each candidate knowledge information of the device acquired in advance. Here, for a device, there can be multiple candidate knowledge information for one device, and the candidate knowledge information here is the information of the knowledge points used to explain the use of the device, for example, it can be the knowledge points obtained by compiling the instruction manual. When matching the question text with each candidate knowledge information, the correlation between the vector representation of the question text and the vector representation of each candidate knowledge information can be calculated, thereby determining the relevant knowledge information matching the question text based on the correlation. For example, the candidate knowledge information with a correlation higher than a preset threshold can be used as the relevant knowledge information matching the question text, or the correlations can be arranged in order from high to low, and the candidate knowledge information corresponding to the first N correlations are taken as the relevant knowledge information matching the question text. The embodiment of the present invention does not specifically limit this.

[0063] After obtaining the relevant knowledge information, an answer to the question text can be generated based on the relevant knowledge information, and the answer here can include an explanatory picture and an explanatory text. In addition, the generation of the answer to the question text can be achieved through a question-answering model, or the question text and the relevant knowledge information can be input into a large language model, and the large language model refers to the relevant knowledge information to generate an explanatory picture and an explanatory text corresponding to the question text.

[0064] Based on any of the above embodiments, before step 230, the method further includes: Determine a target standard instruction matching the instruction text from each candidate standard instruction; The target standard instruction is semantically understood to obtain a control instruction corresponding to the instruction text.

[0065] Specifically, after obtaining the instruction text, the instruction text can be matched with the candidate standard instructions obtained in advance. The candidate standard instructions here are standardized instruction texts pre-set for the device. Compared with the text instructions output by the large language model, the candidate standard instructions are more standardized at the instruction level and easier to be understood by the semantic understanding system. Therefore, after obtaining the instruction text, the candidate standard instructions that match the instruction text can be determined from the candidate standard instructions, which are recorded here as target standard instructions.

[0066] Furthermore, when matching the instruction text with each candidate standard instruction, the correlation between the vector representation of the instruction text and the vector representation of each candidate standard instruction can be specifically calculated, thereby determining the target standard instruction that matches the instruction text based on the correlation. For example, the candidate standard instruction with a correlation higher than a preset threshold can be used as the target standard instruction that matches the instruction text, or the candidate standard instruction corresponding to the highest correlation can be used as the target standard instruction that matches the instruction text, and the embodiment of the present invention does not specifically limit this.

[0067] After obtaining the target standard instructions, the target standard instructions can be input into the semantic understanding model for semantic analysis, thereby obtaining control instructions that are output by the semantic understanding model and correspond to the target standard instructions and are executable by the device.

[0068] Based on any of the above embodiments, Figure 4 This is the second flow chart of the control method provided by the present invention, such as Figure 4 As shown, the method includes: For the device to be controlled, the user manual of the device can be compiled in advance, and the knowledge points involved in the user manual can be arranged in the form of candidate knowledge information, for example, as shown in the figure as candidate knowledge information 1, 2, 3, etc. Vector encoding can be performed on each candidate knowledge information, thereby obtaining a vector representation of each candidate knowledge information, and then constructing a vector knowledge base.

[0069] Furthermore, for the semantic understanding system, a set of candidate standard instructions supported by the semantic understanding system can be obtained, and each candidate standard instruction therein can be vector-encoded, thereby obtaining a vector representation of each candidate standard instruction, and then constructing a standard instruction vector library.

[0070] First, the question text input by the user can be received. Then, the question text can be vector-encoded to obtain the vector representation of the question text, and the vector representation of the question text can be input into the retrieval model. The retrieval model is applied to retrieve relevant knowledge information matching the question text from the vector knowledge base, and the relevant knowledge information and the question text are input into the reasoning response large model together, thereby obtaining the answer output by the reasoning response large model, which specifically includes an explanation picture and an explanation text. Here, the reasoning response large model is a large language model used for question scenarios.

[0071] Subsequently, the explanatory image and the explanatory text may be displayed. Furthermore, the user's annotation operation on the explanatory image may be received, thereby obtaining an annotated image. Then the annotated image and the explanatory text may be input into the multimodal large model together. The multimodal large model here is a large language model that can process data of multiple modalities. The multimodal large model can understand the intention of the annotation operation based on the annotated image and the explanatory text, thereby outputting the instruction text.

[0072] The instruction text can be input into the retrieval model, and the retrieval model can be used to retrieve candidate standard instructions that match the instruction text from the standard instruction vector library as target standard instructions. The target standard instructions are input into the semantic understanding system to obtain control instructions, and the control instructions are sent to the device, thereby realizing device control.

[0073] The control device provided by the present invention is described below. The control device described below and the control method described above can be referenced to each other.

[0074] Figure 5 Schematic diagram of the structure of the control device provided by the present invention. Figure 5 As shown, the device comprises: An operation response unit 510, configured to determine operation information of the annotation operation in response to an annotation operation on a picture, wherein the picture is used to describe control description information of a device; An intention understanding unit 520 is used to understand the intention of the annotation operation based on a large language model in combination with the operation information and the image, and obtain an instruction text corresponding to the annotation operation; The instruction control unit 530 is used to control the device based on the control instruction corresponding to the instruction text.

[0075] In the device provided by the embodiment of the present invention, a large language model understands the intention of the annotation operation based on the annotated picture, thereby obtaining the instruction text corresponding to the annotation operation, and performing device control based on the control instruction corresponding to the instruction text. In this process, the user only needs to perform the annotation operation on the picture to achieve device control, and there is no need to find and locate the physical operating components by comparing the picture, which greatly saves the time required for device control, avoids the operation failure due to complex operation, and optimizes the user experience.

[0076] Based on any of the above embodiments, the intention understanding unit is specifically used for: Based on the large language model, combined with the annotated image, and the description text and / or the question text, the intention of the annotation operation is understood to obtain the instruction text corresponding to the annotation operation; The description text is a text corresponding to the description picture and used to describe the control description information of the device; The question text is the text that forms a question-answer pair with the explanatory image in a question-answer scenario.

[0077] Based on any of the above embodiments, the intention understanding unit is specifically used for: Combining the annotated image, the description text and / or the question text with a prompt template to obtain an intent understanding prompt, wherein the prompt template is used to instruct the large language model to perform intent understanding; The intention understanding prompt is input into the large language model to obtain the instruction text corresponding to the annotation operation output by the large language model.

[0078] Based on any of the above embodiments, the prompt template includes a text describing the intended operation form of the annotation operation.

[0079] Based on any of the above embodiments, the device further includes a question-answering unit, which is used to: Acquire the question text, and determine relevant knowledge information matching the question text from each candidate knowledge information of the device; Based on the relevant knowledge information, the explanatory image and the explanatory text are generated and displayed.

[0080] Based on any of the above embodiments, the device further includes an instruction conversion unit, which is used to: Determine a target standard instruction matching the instruction text from each candidate standard instruction; The target standard instruction is semantically understood to obtain a control instruction corresponding to the instruction text.

[0081] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the control method, which includes: In response to a marking operation on the explanation picture, determining a marking picture corresponding to the marking operation, wherein the explanation picture is used to describe control explanation information of the device; Based on a large language model, the intention of the annotation operation in the annotated image is understood to obtain the instruction text corresponding to the annotation operation; The device is controlled based on the control instruction corresponding to the instruction text.

[0082] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the relevant technology or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0083] On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored in a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the control method provided by the above methods, the method includes: In response to a marking operation on the explanation picture, determining a marking picture corresponding to the marking operation, wherein the explanation picture is used to describe control explanation information of the device; Based on a large language model, the intention of the annotation operation in the annotated image is understood to obtain the instruction text corresponding to the annotation operation; The device is controlled based on the control instruction corresponding to the instruction text.

[0084] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the control method provided by the above methods is implemented, and the method includes: In response to a marking operation on the explanation picture, determining a marking picture corresponding to the marking operation, wherein the explanation picture is used to describe control explanation information of the device; Based on a large language model, the intention of the annotation operation in the annotated image is understood to obtain the instruction text corresponding to the annotation operation; The device is controlled based on the control instruction corresponding to the instruction text.

[0085] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0086] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A control method, characterized in that: include: In response to a marking operation on the explanation picture, determining a marking picture corresponding to the marking operation, wherein the explanation picture is used to describe control explanation information of the device; Based on a large language model, the intention of the annotation operation in the annotated image is understood to obtain the instruction text corresponding to the annotation operation; The device is controlled based on the control instruction corresponding to the instruction text.

2. The control method according to claim 1, characterized in that: The method of understanding the intention of the annotation operation in the annotated image based on the large language model to obtain the instruction text corresponding to the annotation operation includes: Based on the large language model, combined with the annotated image, and the description text and / or the question text, the intention of the annotation operation is understood to obtain the instruction text corresponding to the annotation operation; The description text is a text corresponding to the description picture and used to describe the control description information of the device; The question text is the text that forms a question-answer pair with the explanatory image in a question-answer scenario.

3. The control method according to claim 2, characterized in that: The method of understanding the intention of the annotation operation based on the large language model, in combination with the annotated image, and the description text and / or the question text, and obtaining the instruction text corresponding to the annotation operation includes: Combining the annotated image, the description text and / or the question text with a prompt template to obtain an intent understanding prompt, wherein the prompt template is used to instruct the large language model to perform intent understanding; The intention understanding prompt is input into the large language model to obtain the instruction text corresponding to the annotation operation output by the large language model.

4. The control method according to claim 3, characterized in that: The prompt template includes a text describing the intended operation form of the annotation operation.

5. The control method according to claim 2, characterized in that: The step of determining the annotated image corresponding to the annotating operation further includes: Acquire the question text, and determine relevant knowledge information matching the question text from each candidate knowledge information of the device; Based on the relevant knowledge information, the explanatory image and the explanatory text are generated and displayed.

6. The control method according to any one of claims 1 to 5, characterized in that: The device control is performed based on the control instruction corresponding to the instruction text, and the method further includes: Determine a target standard instruction matching the instruction text from each candidate standard instruction; The target standard instruction is semantically understood to obtain a control instruction corresponding to the instruction text.

7. A control device, characterized in that: include: an operation response unit, configured to determine operation information of the marking operation in response to a marking operation on the picture, wherein the picture is used to describe control description information of the device; An intention understanding unit, configured to understand the intention of the annotation operation based on a large language model in combination with the operation information and the image, and obtain an instruction text corresponding to the annotation operation; The instruction control unit is used to control the device based on the control instruction corresponding to the instruction text.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the control method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the control method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the control method according to any one of claims 1 to 6 is implemented.

Images