Equipment interaction method and related device

Abstract

The invention provides an equipment interaction method and a related device, and relates to the field of operation and maintenance. In the application, when a user has a problem related to operation and maintenance, a user intention can be determined based on user demand information, an operator having an association relationship with the user intention is called, and the user demand information is processed to obtain associated data corresponding to the user demand information; and utilizing at least one of the user demand information and the associated data to obtain an equipment interaction result, and providing an answer to the question for the user. In the application, when the user demand information is processed, the operator matched with the user intention is selected, so that the selected operator is matched with the user operation and maintenance demand scene, the accuracy of the obtained associated data is ensured, the accuracy of the obtained equipment interaction result is further improved, and the user experience is improved.

Description

Technical Field

[0001] This application relates to the field of operation and maintenance, and more specifically, to a device interaction method and related apparatus. Background Technology

[0002] With the continuous deepening of domestic industrialization and modernization, industrial equipment is widely used in various industries related to national economy and people's livelihood, such as petroleum, chemical, and power. The health status and operating condition of the equipment directly affect the safety and reliability of its operation. Therefore, the effective operation and maintenance of industrial equipment is becoming increasingly important.

[0003] During equipment operation and maintenance, users may encounter problems related to operation and maintenance. How to determine the corresponding answer to these problems is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] In view of this, this application provides a device interaction method and related apparatus to solve the problem of urgently needing to provide users with answers to questions related to operation and maintenance.

[0005] To solve the above-mentioned technical problems, this application adopts the following technical solution:

[0006] A device interaction method, comprising:

[0007] Determine the user intent corresponding to the user's needs information;

[0008] Invoke operators that are related to user intent to process user demand information and obtain associated data corresponding to user demand information;

[0009] The device interaction result is obtained by using at least one of the user demand information and related data.

[0010] Optionally, determining the user intent corresponding to the user demand information includes:

[0011] Determine user needs information;

[0012] Construct primary prompts corresponding to user needs information;

[0013] The multimodal large model is invoked to process the first prompt word and obtain the user intent.

[0014] Optionally, user requirement information may be determined, including:

[0015] Obtain initial requirements information;

[0016] If the initial requirement information is voice data, perform a text conversion operation on the voice data to obtain the target text, and use the target text as the user requirement information;

[0017] If the initial requirement information is video data, the video data is subjected to frame extraction to obtain an image sequence, which is then used as the user requirement information.

[0018] Optionally, if the user intent is related to operation and maintenance knowledge, an operator associated with the user intent is invoked to process the user request information and obtain the associated data corresponding to the user request information, including:

[0019] Transform user demand information into a target vector;

[0020] The target vector is input into the multimodal knowledge base operator in the operator library located on the device side, so that the multimodal knowledge base operator can retrieve the operation and maintenance knowledge fragment with the highest degree of correlation with the target vector;

[0021] Use operation and maintenance knowledge fragments as associated data.

[0022] Optionally, the device interaction result is obtained by utilizing at least one of the user demand information and related data, including:

[0023] Construct a second prompt word by utilizing user demand information and related data;

[0024] The multimodal large model is invoked to process the second prompt word, and the device interaction result is obtained.

[0025] Optionally, if the user intent is related to data analysis, an operator associated with the user intent is invoked to process the user's request information and obtain the associated data corresponding to the user's request information, including:

[0026] Select the target analysis operator that matches the user's needs from the data analysis operators corresponding to the user's intent in the operator library;

[0027] Invoke the target analysis operator to obtain the data to be analyzed that matches the user's intent, perform analysis operations on the data to be analyzed, and obtain the analysis results;

[0028] The analysis results are used as related data.

[0029] Optionally, the device interaction result is obtained by utilizing at least one of the user demand information and related data, including:

[0030] The associated data is displayed in charts to obtain the device interaction results.

[0031] A device interaction device, comprising:

[0032] The intent determination module is used to determine the user intent corresponding to the user's request information;

[0033] The data processing module is used to call operators that are related to user intent, process user demand information, and obtain related data corresponding to user demand information.

[0034] The result determination module is used to obtain the device interaction result by utilizing user demand information and at least one of the associated data.

[0035] An electronic device includes at least one processor and a memory connected to the processor, wherein:

[0036] Memory is used to store computer programs;

[0037] The processor is used to execute computer programs so that electronic devices can implement the device interaction methods described above.

[0038] A device interaction system includes an operator terminal and the aforementioned electronic device;

[0039] Operator terminals are used for control based on electronic devices, running operators invoked by the electronic devices to obtain associated data.

[0040] A computer storage medium is characterized in that it carries one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the aforementioned device interaction method.

[0041] This application provides a device interaction method and related apparatus. In this application, when a user has a problem related to operation and maintenance, the user's intent can be determined based on user demand information. An operator related to the user's intent is then invoked to process the user demand information, obtaining associated data corresponding to the user demand information. Using at least one of the user demand information and associated data, a device interaction result is obtained, providing the user with an answer to their problem. In this application, when processing user demand information, an operator matching the user's intent is selected, ensuring that the selected operator matches the user's operation and maintenance needs scenario. This guarantees the accuracy of the obtained associated data, thereby improving the accuracy of the device interaction result and enhancing the user experience. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0043] Figure 1 This is a schematic diagram of the structure of a device interaction system provided in an embodiment of this application;

[0044] Figure 2 A schematic diagram of an operator library provided for an embodiment of this application;

[0045] Figure 3 A schematic diagram of a user terminal provided in an embodiment of this application;

[0046] Figure 4 A flowchart illustrating a device interaction method provided in an embodiment of this application;

[0047] Figure 5 A flowchart illustrating a method for determining user intent provided in an embodiment of this application;

[0048] Figure 6 A flowchart illustrating a method for determining associated data provided in an embodiment of this application;

[0049] Figure 7 A schematic diagram illustrating a device interaction method provided in an embodiment of this application;

[0050] Figure 8 A flowchart illustrating another method for determining associated data provided in an embodiment of this application;

[0051] Figure 9 A schematic diagram illustrating another device interaction method provided in an embodiment of this application;

[0052] Figure 10 A schematic diagram illustrating another device interaction method provided in an embodiment of this application;

[0053] Figure 11 This is a schematic diagram of the structure of a device interaction apparatus provided in an embodiment of this application;

[0054] Figure 12 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0055] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0056] With the continuous deepening of domestic industrialization and modernization, industrial equipment is widely used in various industries related to national economy and people's livelihood, such as petroleum, chemical, and power. The health status and operating condition of the equipment directly affect the safety and reliability of operation, thereby directly affecting whether production plans can be successfully completed, and even more so, the safety of people's lives and property. Therefore, the effective operation and maintenance of industrial equipment is becoming increasingly important.

[0057] During equipment operation and maintenance, users may encounter problems related to operation and maintenance. How to determine the corresponding answer to these problems is a technical problem that urgently needs to be solved by those skilled in the art.

[0058] In order to answer users' questions in a timely manner, this application embodiment introduces a multimodal large model to answer user questions. The multimodal large model can process multimodal data such as text, images, videos, voice, and time series data, thereby fully mining the actual data at the site and providing applicability of operation and maintenance technologies.

[0059] In addition, this application supports users' operational and maintenance knowledge-related intents as well as data analysis-related intents, supporting a wider range of user needs and meeting different user problem scenarios.

[0060] To address this, this application provides a device interaction method and related apparatus. In this application, when a user has a maintenance-related problem, the user's intent can be determined based on user demand information. An operator related to the user's intent is then invoked to process the user demand information, obtaining associated data corresponding to the user demand information. Using at least one of the user demand information and associated data, a device interaction result is obtained, providing the user with an answer to their problem. In this application, when processing user demand information, an operator matching the user's intent is selected, ensuring that the selected operator matches the user's maintenance needs scenario. This guarantees the accuracy of the obtained associated data, thereby improving the accuracy of the device interaction result and enhancing the user experience.

[0061] Based on the above, this application provides a device interaction method, the execution subject of which can be an electronic device, such as the cloud.

[0062] When a user has a problem, they can output it to the cloud through the user terminal. After receiving the problem transmitted by the user terminal, the cloud processes it, obtains the corresponding answer, and outputs it to the user terminal, which then displays the answer to the user.

[0063] In this embodiment, the cloud and the user terminal are two components of the device interaction system. Furthermore, the device interaction system also includes an operator terminal, which can be an edge terminal. The structural diagram of the device interaction system in this application (also referred to as a cloud-edge-device collaborative operation and maintenance intelligent agent based on a multimodal large model) is shown below. Figure 1 .

[0064] Figure 1 In the cloud, a multimodal large model intelligent agent is configured with a multimodal large model. This multimodal large model can be an open-source multimodal large model. In different scenarios, the multimodal large model can be fine-tuned to improve the model's adaptability in different scenarios.

[0065] The operator terminal (edge) is internally configured with an operator library. In practical applications, the operation and maintenance scenario in this embodiment can be a field scenario for large industrial equipment or a user scenario. For the field scenario, the edge is deployed as a private server inside the field, and the operator library is only accessible within the field. For the user scenario, the edge is deployed embedded in the device itself. The operator library is composed as follows... Figure 2 As shown, it includes multimodal knowledge base operators and data mining tool operators.

[0066] The multimodal knowledge base operator stores historical operation and maintenance records accumulated during site visits or equipment testing and verification, including multimodal data such as text information, image information, and captured videos. Data mining tool operators are data analysis operators used for data analysis operations. Data analysis operators can be mining models for numerical data.

[0067] In real-world scenarios, both multimodal knowledge base operators and numerical mining tool operators are selected and configured by site personnel or on-site users themselves. The self-selection and configuration process is as follows:

[0068] 1. Configuration of operators for multimodal knowledge bases:

[0069] For the field site scenario: The multimodal knowledge base operator includes a local knowledge base maintained by field personnel. The local knowledge base stores multimodal data such as text and images uploaded by users that are related to the field's historical accumulated operation and maintenance cases or maintenance experience.

[0070] For residential applications: Multimodal knowledge base operators include relevant data accumulated from equipment historical maintenance records or testing and inspection.

[0071] 2. Configuration of operators for data mining tools:

[0072] For the field station scenario: Field station personnel can customize operator configurations according to actual needs. The edge operator library has corresponding configuration interfaces, and relevant data processing programs can be written based on the data interaction standard format to realize operator configuration. The configured operators can be such as load prediction models, current diagnosis and detection models, etc.

[0073] For residential scenarios: Data mining tools are data analysis tools designed for real-world business scenarios. These tools can be written in Python, and include functions such as electricity consumption statistics and electricity cost statistics. Users can configure operators via a PC (Personal Computer) terminal according to the actual characteristics and needs of the site. The operator library generates a set of data mining tool operators based on the user's configuration.

[0074] The user terminal consists of Figure 3 As shown, the system is divided into PC terminals and mobile terminals. The PC terminal can configure the aforementioned operator library according to the specific conditions of the site, including uploading the multimodal knowledge base and configuring data mining tools. Furthermore, to improve configuration security, the edge operator library has access control capabilities to manage the usage permissions of terminal users. Only authorized users can configure the operator library through the PC terminal.

[0075] The mobile terminal is used in the form of a mobile app (mobile software). Frontline maintenance personnel or on-site users can use the mobile terminal to upload problems to the cloud. The cloud pushes answers, such as recommended maintenance solutions or automatic data statistical analysis results, through the mobile terminal.

[0076] In practical applications, the interaction process of various components in a device interaction system is as follows:

[0077] Users submit their requests through user terminals, typically in the form of text or voice input. The cloud-based multimodal big data model intelligent agent recognizes the user's intent and then calls the edge operator library. The edge operator library performs calculations or queries to obtain processing results and feeds them back to the cloud-based multimodal big data model. The cloud-based big data model summarizes the knowledge based on the results given by the edge operator library and feeds it back to the user terminal.

[0078] In summary, the device interaction system is based on a cloud-edge-device framework with a microservice architecture. The modules are highly decoupled, enabling the operator library to be private, customized, and pluggable, thereby improving the adaptability of field stations and the security of private domain data.

[0079] In another implementation of this application, refer to Figure 4 Device interaction methods may include:

[0080] S11. Determine the user intent corresponding to the user demand information.

[0081] In practical applications, if users have questions that need answers during operation and maintenance, they can output their needs through the aforementioned mobile terminal.

[0082] For example, if a user discovers high voltage in equipment during an inspection, the system could output "What should I do if the voltage is high?" or "How much electricity have you used in the last month?".

[0083] The information output by the user can be multimodal data such as text, images, videos, audio, and time-series data. This information can be used directly as user requirement information, or it can be processed to obtain user requirement information.

[0084] The mobile terminal sends the user's output information to the cloud so that the cloud can determine the user's needs and perform user intent recognition. In this embodiment, user intent can be divided into two categories: operation and maintenance knowledge-related intent and data analysis-related intent. Specifically, if the user asks questions about fault phenomena or causes, this is an operation and maintenance knowledge-related intent. Data analysis-related intent refers to the user's intent to use data mining tools, such as when the user wants to display data statistical indicators or perform data prediction tasks.

[0085] S12. Call the operator that is related to the user's intent to process the user's demand information and obtain the associated data corresponding to the user's demand information.

[0086] In practical applications, the edge operator library contains two types of operators: the multimodal knowledge base operators mentioned above and data mining tool operators. After obtaining the user's intent, the corresponding operator can be invoked for data processing. If the user's intent is related to operation and maintenance knowledge, the multimodal knowledge base operator is invoked to retrieve operation and maintenance knowledge and obtain related data. If the user's intent is related to data analysis, the data mining tool operator is invoked to perform data analysis operations and obtain related data.

[0087] S13. Obtain the device interaction result by utilizing at least one of the user demand information and related data.

[0088] After obtaining user needs and related data, the cloud can access its internal multimodal big data model for analysis and obtain the device interaction results. These device interaction results are the answers to the user's questions.

[0089] If a user asks, "What should I do if the voltage is too high?", the device's interactive response could be, "A voltage reduction operation is required, such as shutting down some devices or performing a shutdown operation."

[0090] For example, if a user asks "How much electricity have you used in the last month?", the device's interactive result could be "1000 kilowatt-hours of electricity have been used in the last month".

[0091] The interaction results of this device can be displayed to the user via a mobile terminal. If the user is not satisfied with the interaction results, they can continue to ask questions until they receive a satisfactory answer.

[0092] In this embodiment, when a user has an operation and maintenance related question, the user's intent can be determined based on the user's demand information. An operator related to the user's intent is then invoked to process the user's demand information, obtaining associated data. Using at least one of the user's demand information and associated data, a device interaction result is obtained, providing the user with an answer to their question. In this embodiment, when processing the user's demand information, an operator matching the user's intent is selected, ensuring that the selected operator matches the user's operation and maintenance needs scenario. This guarantees the accuracy of the obtained associated data, thereby improving the accuracy of the device interaction result and enhancing the user experience.

[0093] The above embodiments illustrate the need to determine user intent during interaction. This application describes the specific process of determining the user intent corresponding to user request information, referring to... Figure 5 It can include:

[0094] S21. Determine user requirements information.

[0095] In this embodiment, the user demand information can be information directly output by the user through a mobile terminal, or it can be information obtained by processing the information output by the user through a mobile terminal.

[0096] In this embodiment, for ease of description, the information directly output by the user through the mobile terminal is referred to as initial requirement information. For example, if the user inputs text, images, voice, or video through the interactive interface of the mobile terminal, this data constitutes the initial requirement information.

[0097] Generally, large models can process data such as text and images. If the information output by the user directly through the mobile terminal is voice data, then the voice data can be converted into text to obtain the target text, which can then be used as the user's required information.

[0098] When converting speech to text, you can use a speech conversion tool. You can choose the appropriate speech conversion tool based on your needs.

[0099] If the initial requirement information is video data, the video data is subjected to frame extraction to obtain an image sequence, which is then used as the user requirement information.

[0100] If the user inputs text, images, time-series data, etc., the multimodal large model can process it directly. In this case, the initial requirement information is directly used as the user requirement information, and no further processing is required.

[0101] S22. Construct the first prompt word corresponding to the user's demand information.

[0102] In practical applications, when using a multimodal large model, it is necessary to construct a prompt word. Since this application requires multiple calls to the multimodal large model, multiple prompt words need to be constructed. Therefore, the prompt word constructed in this embodiment is called the first prompt word.

[0103] When constructing the prompt words, a few-shot approach can be used to construct the first prompt word. The template for the first prompt word is as follows:

[0104] "As an intent recognition expert, you need to identify user intent, which falls into two main categories:"

[0105] Operations and maintenance knowledge related intent, which mainly involves querying operations and maintenance solutions or solutions based on fault symptoms;

[0106] The data analysis intent is primarily to analyze or predict data from the field.

[0107] Please refer to the following examples:

[0108] Case 1, User Question: How should I handle a fan malfunction? This is an intent related to system maintenance knowledge.

[0109] Case 2, User Question: Please analyze and statistically analyze the monthly power generation data. This demonstrates an intent related to data analysis.

[0110] Case 3, User Question: (Image of a fault phenomenon), This is an intent related to maintenance knowledge.

[0111] The current user's question is: "query". Please analyze the user's intent behind this question.

[0112] When the first prompt word template is obtained, the query in the template can be replaced with the user demand information determined in this embodiment to obtain the first prompt word.

[0113] S23. Call the multimodal large model to process the first prompt word and obtain the user intent.

[0114] Specifically, the first prompt word is input into the multimodal big model, which can then analyze the user's needs information to obtain the user's intent. The user intent is either an intent related to operation and maintenance knowledge or an intent related to data analysis.

[0115] In this embodiment, the multimodal large model refers to a model that combines multimodal information such as text, images, videos, and audio for training.

[0116] Multimodal large models require training with a large number of training samples. Therefore, large models have high accuracy in recognizing user intent and can then call appropriate operators for subsequent data processing operations.

[0117] As mentioned above, the user intent can be either an operational knowledge-related intent or a data analysis-related intent. The subsequent operations performed differ depending on the user intent, which will be described separately below.

[0118] 1. The user intent is related to operation and maintenance knowledge.

[0119] When users want to query operation and maintenance knowledge, refer to Figure 6 It invokes operators related to user intent to process user demand information and obtain associated data corresponding to the user demand information, which may include:

[0120] S31. Convert user demand information into a target vector.

[0121] In practical applications, when users have operational and maintenance (O&M) knowledge needs, such as obtaining specific O&M knowledge or solutions, they need to retrieve information from a multimodal knowledge base. To improve retrieval efficiency, vector retrieval can be used. Therefore, user requirement information needs to be converted into a target vector for subsequent retrieval operations.

[0122] When performing vector transformation, a vector transformation model can be used to perform the vector transformation operation. The vector transformation model can be such as CLIP (Contrastive Language-Image Pre-training) multimodal vector model.

[0123] In addition to converting user demand information into vectors, the multimodal knowledge fragments in the multimodal knowledge base also need to be vectorized.

[0124] Specifically, refer to Figure 7 Users can upload historical operation and maintenance records of the site via PC terminals, including multimodal information such as pictures and text. The operator library receives and stores the multimodal operation and maintenance records to build a multimodal knowledge base.

[0125] The edge operator library is based on the CLIP multimodal vector model, which vectorizes multimodal knowledge fragments and constructs a vector pool for subsequent retrieval of the multimodal knowledge base.

[0126] S32. Input the target vector into the multimodal knowledge base operator in the operator library located on the device side, so that the multimodal knowledge base operator can retrieve the operation and maintenance knowledge fragment with the highest degree of correlation with the target vector.

[0127] Specifically, refer to Figure 7 The user query, i.e. the user's request information, is vectorized by CLIP and input into the vector pool constructed above. Then, the most relevant operation and maintenance knowledge fragments are retrieved from the multimodal knowledge base using Facebook AI Similarity Search (vector retrieval engine).

[0128] S33. Use operation and maintenance knowledge fragments as associated data.

[0129] The operation and maintenance knowledge fragments obtained through the vector pool are the associated data required in this embodiment.

[0130] It should be noted that, in addition to using the vector retrieval method mentioned above to obtain related data, the most relevant operation and maintenance knowledge fragments can also be obtained through keyword comparison and other methods, thereby obtaining related data.

[0131] After obtaining the associated data, we can continue to use the multimodal large model intelligent agent to summarize knowledge and obtain the device interaction results. Specifically, we can use user demand information and associated data to construct a second prompt word.

[0132] In this embodiment, since a multimodal large model is needed for question answering and knowledge summarization, the prompt word template used is different from the prompt word template for recognizing user intent mentioned above. The prompt word template for the large model for question answering and knowledge summarization in this embodiment is as follows:

[0133] "You are an operations and maintenance expert with extensive experience in multimodal data processing. Your current task is to provide operational and maintenance knowledge answers to user questions based on knowledge from a multimodal knowledge base. When constructing answers, please closely integrate the user's question with the extracted knowledge."

[0134] The multimodal knowledge fragment is: {content}

[0135] The user's question was: {query}

[0136] Please answer based on the context.

[0137] After obtaining the template, replace the query in the template with the user's needs information and replace the content in the template with the related data to obtain the second suggestion word.

[0138] After obtaining the second prompt word, the multimodal large model is invoked to process the second prompt word and obtain the device interaction result.

[0139] For example, the device interaction result could be something like, "The current voltage is too high. To ensure the safety of the device, an emergency shutdown is required."

[0140] In this embodiment, the multimodal large model agent combines the user's query and the retrieved operation and maintenance knowledge fragments to answer the question and summarize the knowledge, thus obtaining the answer required by the user.

[0141] In addition, this application pre-builds a multimodal operation and maintenance knowledge base, which can save the processing and query workload of operation and maintenance personnel or on-site users and improve efficiency.

[0142] 2. The user intent is related to data analysis.

[0143] When the user's intent is related to data analysis, it indicates that the user wants to perform data analysis operations, such as power consumption statistics or voltage statistics. In this case, it is necessary to invoke data mining tools and operators to process the user's needs. Specifically, refer to... Figure 8 It invokes operators related to user intent to process user demand information and obtain associated data corresponding to the user demand information, including:

[0144] S41. Select the target analysis operator that matches the user's needs information from the data analysis operators corresponding to the user's intent in the operator library.

[0145] Specifically, when the user's intent is related to data analysis, all data analysis operators corresponding to this intent are retrieved from the operator library. This includes various data mining tool operators such as electricity consumption statistics, electricity cost statistics, electricity cost prediction, and electricity consumption prediction. From these numerous operators, one operator needs to be selected for processing the user's information requirements in this specific case.

[0146] When selecting operators, user needs information can be referenced, and multimodal large models can be used to identify the user's specific data analysis intent in order to select the appropriate operators.

[0147] When using a multimodal large model, prompt words can be constructed using prompt word templates. User requirement information is then added to the prompt word templates to obtain the prompt words. Inputting these prompt words into the multimodal large model yields a target analysis operator that matches the user requirement information. In this embodiment, the prompt word template can be as follows:

[0148] "You are an expert specializing in identifying the user intent of data analysis tools. Your current task is to analyze which data analysis tool a user is using based on their questions. When identifying this intent, please closely consider both the user's question and the data analysis tools currently configured in the system."

[0149] The system's configured data analysis tool is: {methods_list}

[0150] The user's question was: {query}

[0151] Please answer based on the context.

[0152] Here, {methods_list} is the data mining toolset configured in the edge operator library, and {query} is the user requirement information.

[0153] S42. Call the target analysis operator so that the target analysis operator can obtain the data to be analyzed that matches the user's intent, perform analysis operations on the data to be analyzed, and obtain the analysis results.

[0154] After obtaining the target analysis operator, the multimodal large-scale intelligent agent performs operator proxying. That is, the cloud-based multimodal large-scale model sends proxy instructions to the edge operator library, calling the corresponding operator in the edge operator library. During runtime, the operator retrieves the corresponding data to be analyzed from the site database. For example, if the user's data analysis intent is to calculate the total electricity consumption over the past month, the operator will retrieve the daily electricity consumption values ​​for each day of the past month from the site database, calculate the total electricity consumption, and obtain the total electricity consumption. This total electricity consumption is the analysis result in this embodiment. In other words, after being called, the operator accesses the site database for computation to obtain the analysis result. For specific implementation details, refer to [link to implementation details]. Figure 9 .

[0155] S43. Use the analysis results as related data.

[0156] Specifically, in this embodiment, the analysis results are sent to the cloud via operators. The cloud uses the analysis results as associated data for subsequent statistical result display operations.

[0157] At this point, by utilizing at least one of the user's needs information and related data, the device interaction result is obtained, which is:

[0158] The associated data is displayed graphically to obtain the device interaction results. For specific implementation details, please refer to [link / reference]. Figure 9 .

[0159] When displaying statistical results in the cloud, numerical values ​​or statistical charts can be used for graphical representation. The specific display method chosen can be determined using a multimodal large-scale model. This involves constructing prompts using prompt word templates, inputting these prompts into the multimodal large-scale model to determine the display method, and then performing the display operation accordingly. In this embodiment, the prompt word template can be as follows:

[0160] "You are an expert specializing in data analysis and chart visualization. Your current task is to select the best way to present the data analysis results. When presenting the results, please closely integrate user questions with the data analysis findings."

[0161] The data analysis results are: {result_list}

[0162] The user's question was: {query}

[0163] Please display the results according to the context.

[0164] Here, {result_list} represents the analysis results and also the associated data, while {query} represents the user's request information.

[0165] Once the display method is determined, the cloud can send the data to be displayed to the mobile terminal according to that method for the user to view. If the user is not satisfied with the pushed data, they can continue to ask questions to the cloud.

[0166] If the cloud sends the total electricity consumption for a month to the user and the user finds that the total electricity consumption for that month is abnormally high, the user can ask the cloud to send the electricity consumption for each day of that month. The cloud can then perform further data processing operations based on the user's needs.

[0167] In this embodiment, a data mining toolset is constructed, and a multimodal large model intelligent agent is used for operator proxying, thereby achieving automated data analysis and statistics using the corresponding operators.

[0168] As can be seen from the above, referring to Figure 10 The interaction process of each component in the device interaction system in this application embodiment is as follows:

[0169] The user terminal asks questions, and the user inputs information such as text, images, videos, voice, and time-series data. A multimodal large-scale intelligent agent performs intent recognition after receiving the user's question. There are two main categories of intents: operational knowledge-related intents and data analysis-related intents.

[0170] If the intent is identified as related to operations and maintenance knowledge, a multimodal knowledge base search is performed. If the intent is identified as related to data analysis, a data mining tool (or data mining model) proxy is used to obtain related data.

[0171] Finally, the multimodal big data agent summarizes the knowledge. If the intent is related to operation and maintenance, the retrieved multimodal knowledge fragments will be combined with the user's question and input into the multimodal big data agent to answer the question; if the intent is related to data analysis, the corresponding data mining tools will be called to perform statistical analysis or prediction on the numerical data, and the results will be combined with the user's question and input into the multimodal big data agent to answer the question.

[0172] In this embodiment, images, text, video, voice, and time-series data are used as signal inputs, realizing the unified integration and application of multi-source heterogeneous data, improving the application scope and practicality of this application in industrial scenarios.

[0173] In addition, this application constructs a multimodal large model intelligent agent agent technology and multi-source signal operator processing technology to fully mine the actual data at the site. It can directly provide users with the data they need, such as the operation and maintenance plan or data analysis results required by users, without requiring users to perform some data processing or data filtering and determination work (such as not requiring users to select an operation and maintenance plan based on the model output results). This improves the applicability of operation and maintenance technology and reduces the workload of users.

[0174] In addition, this solution adopts a cloud-edge-device framework based on microservice architecture, with each module highly decoupled, realizing the privatization, customization and pluggability of the operator library, improving the adaptability of field sites and the security of private domain data.

[0175] Based on the embodiments of the above-described device interaction method, another embodiment of this application provides a device interaction apparatus, referring to... Figure 11 ,include:

[0176] The intent determination module 11 is used to determine the user intent corresponding to the user demand information;

[0177] The data processing module 12 is used to call operators that are related to the user's intent, process the user's demand information, and obtain the associated data corresponding to the user's demand information.

[0178] The result determination module 13 is used to obtain the device interaction result by utilizing at least one of the user demand information and related data.

[0179] In one implementation, the intent determination module 11 includes:

[0180] The information determination submodule is used to determine user requirement information;

[0181] The first prompt word construction submodule is used to construct the first prompt word corresponding to the user's request information;

[0182] The first prompt word processing submodule is used to call the multimodal large model to process the first prompt word and obtain the user intent.

[0183] In one implementation, the information determination submodule includes:

[0184] The information acquisition unit is used to acquire initial requirement information;

[0185] The first information processing unit is used to perform text conversion on the voice data if the initial demand information is voice data, to obtain the target text, and to use the target text as the user demand information.

[0186] The second information processing unit is used to perform frame extraction on the video data to obtain an image sequence if the initial demand information is video data, and then use the image sequence as the user demand information.

[0187] In one implementation, if the user's intent is related to operation and maintenance knowledge, the data processing module 12 includes:

[0188] The information conversion submodule is used to convert user requirement information into target vectors;

[0189] The knowledge determination submodule is used to input the target vector into the multimodal knowledge base operator in the operator library located on the device side, so that the multimodal knowledge base operator can retrieve the operation and maintenance knowledge fragment with the highest degree of correlation with the target vector;

[0190] The first data determination submodule is used to associate operation and maintenance knowledge fragments as related data.

[0191] In one implementation, the result determination module 13 includes:

[0192] The second prompt word construction submodule is used to construct second prompt words using user demand information and related data;

[0193] The second prompt word processing submodule is used to call the multimodal large model to process the second prompt word and obtain the device interaction result.

[0194] In one implementation, if the user's intent is related to data analysis, the data processing module 12 includes:

[0195] The operator selection submodule is used to select the target analysis operator that matches the user's needs from the data analysis operators corresponding to the user's intent in the operator library;

[0196] The operator invocation submodule is used to invoke the target analysis operator so that the target analysis operator can obtain the data to be analyzed that matches the user's intent, perform analysis operations on the data to be analyzed, and obtain the analysis results;

[0197] The second data determination submodule is used to use the analysis results as related data.

[0198] In one implementation, the result determination module 13 includes:

[0199] The charting submodule is used to perform charting operations on related data to obtain device interaction results.

[0200] In this embodiment, when a user has an operation and maintenance related question, the user's intent can be determined based on the user's demand information. An operator related to the user's intent is then invoked to process the user's demand information, obtaining associated data. Using at least one of the user's demand information and associated data, a device interaction result is obtained, providing the user with an answer to their question. In this embodiment, when processing the user's demand information, an operator matching the user's intent is selected, ensuring that the selected operator matches the user's operation and maintenance needs scenario. This guarantees the accuracy of the obtained associated data, thereby improving the accuracy of the device interaction result and enhancing the user experience.

[0201] It should be noted that the working process of each module, submodule and unit in this embodiment is described in the corresponding description in the above embodiment, and will not be repeated here.

[0202] Based on the embodiments of the above-described device interaction methods and apparatus, another embodiment of this application provides an electronic device, including at least one processor and a memory connected to the processor, wherein:

[0203] Memory is used to store computer programs;

[0204] The processor is used to execute computer programs so that electronic devices can implement the device interaction methods described above.

[0205] refer to Figure 12 The diagram illustrates a structural schematic suitable for implementing the electronic device in the embodiments of this application. The electronic device in the embodiments of this application may include, but is not limited to, fixed terminals such as the cloud, mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 12 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0206] like Figure 12As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. When the electronic device is powered on, the RAM 603 also stores various programs and data required for the operation of the electronic device. The processing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0207] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, memory cards, hard drives, etc.; and communication devices 609. Communication device 609 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0208] This application also provides a device interaction system, which is detailed in the embodiments below. Figure 1 This includes operator terminals and the aforementioned electronic devices;

[0209] The operator terminal is used for control based on electronic devices, running operators invoked by the electronic devices to obtain associated data. For specific implementation details, please refer to the corresponding descriptions above.

[0210] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the device interaction methods provided in this application.

[0211] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the device interaction methods provided in this application.

[0212] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A device interaction method, characterized in that, include: Determine the user intent corresponding to the user's needs information; Invoke operators that are associated with the user intent to process the user demand information and obtain associated data corresponding to the user demand information; The device interaction result is obtained by using at least one of the user demand information and the associated data.

2. The device interaction method according to claim 1, characterized in that, Determine the user intent corresponding to the user's needs information, including: Determine user needs information; Construct the first prompt word corresponding to the user's demand information; The first prompt word is processed by calling a multimodal large model to obtain the user intent.

3. The device interaction method according to claim 2, characterized in that, Determine user needs information, including: Obtain initial requirements information; If the initial requirement information is voice data, perform a text conversion operation on the voice data to obtain the target text, and use the target text as the user requirement information; If the initial requirement information is video data, the video data is subjected to frame extraction to obtain an image sequence, and the image sequence is used as the user requirement information.

4. The device interaction method according to claim 1, characterized in that, If the user's intent is related to operation and maintenance knowledge; Invoking operators associated with the user intent to process the user demand information and obtain associated data corresponding to the user demand information, including: Convert the user demand information into a target vector; The target vector is input into a multimodal knowledge base operator in the operator library located on the device side, so that the multimodal knowledge base operator retrieves the operation and maintenance knowledge fragment with the highest degree of correlation with the target vector; The aforementioned operation and maintenance knowledge fragments are used as associated data.

5. The device interaction method according to claim 4, characterized in that, Using at least one of the user demand information and the associated data, the device interaction result is obtained, including: Using the user demand information and the associated data, a second prompt word is constructed; The second prompt word is processed by calling a multimodal large model to obtain the device interaction result.

6. The device interaction method according to claim 1, characterized in that, If the user intent is related to data analysis; Invoking operators associated with the user intent to process the user demand information and obtain associated data corresponding to the user demand information, including: Select the target analysis operator that matches the user's demand information from the data analysis operators corresponding to the user's intent in the operator library; The target analysis operator is invoked to obtain data to be analyzed that matches the user intent, and the data to be analyzed is analyzed to obtain the analysis results. The analysis results are used as related data.

7. The device interaction method according to claim 6, characterized in that, Using at least one of the user demand information and the associated data, the device interaction result is obtained, including: The associated data is then displayed graphically to obtain the device interaction results.

8. A device interaction apparatus, characterized in that, include: The intent determination module is used to determine the user intent corresponding to the user's request information; The data processing module is used to call operators that are related to the user intent to process the user demand information and obtain the associated data corresponding to the user demand information. The result determination module is used to obtain the device interaction result by utilizing at least one of the user demand information and the associated data.

9. An electronic device, characterized in that, It includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to enable the electronic device to implement the device interaction method as described in any one of claims 1 to 7.

10. A device interaction system, characterized in that, Includes an operator terminal and the electronic device as described in claim 9; The operator terminal is used to run the operators invoked by the electronic device based on the control of the electronic device, and obtain associated data.

11. A computer storage medium, characterized in that, The storage medium carries one or more computer programs that, when executed by an electronic device, enable the electronic device to implement the device interaction method as described in any one of claims 1 to 7.

Images