Data processing method, apparatus, and electronic device

Abstract

Embodiments of the present application disclose a data processing method and device and electronic equipment. In the method, after obtaining first input data, second input data can be obtained by using the first input data and a first recognition result, and then feedback content corresponding to the second input data is obtained. Because the second input data can more accurately express the information acquisition intention of the user, the obtained feedback content can be more accurate. Moreover, the first recognition result used to obtain the second input data is obtained when obtaining feedback content of historical input data, and thus it is not necessary to obtain the feedback content by using real-time recognition after obtaining the first input data, thereby improving the efficiency of obtaining the feedback content.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a data processing method, apparatus, and electronic device. Background Technology

[0002] In human-computer dialogue scenarios, dialogue systems can provide corresponding responses based on user input. For example, if a user inputs "What's the weather like tomorrow?", the dialogue system can provide weather information. However, the response efficiency of such systems still needs improvement. Summary of the Invention

[0003] In view of the above problems, this application proposes a data processing method, apparatus, and electronic device to improve the above problems.

[0004] In a first aspect, this application provides a data processing method, the method comprising: obtaining first input data; obtaining second input data through a first identification result and the first input data, wherein the first identification result is used to update a portion of the content in the first input data, the first identification result being obtained when acquiring feedback content of historical input data, the historical input data being input data prior to the first input data; and obtaining corresponding feedback content through the second input data.

[0005] Secondly, this application provides a data processing apparatus, the apparatus comprising: a data acquisition unit for acquiring first input data; a data processing unit for acquiring second input data through a first identification result and the first input data, wherein the first identification result is used to update a portion of the first input data, the first identification result being obtained when acquiring feedback content of historical input data, the historical input data being input data prior to the first input data; and a feedback unit for acquiring corresponding feedback content through the second input data.

[0006] Thirdly, this application provides an electronic device including a processor and a memory; one or more programs are stored in the memory and configured to be executed by the processor to implement the above-described method.

[0007] Fourthly, this application provides a computer-readable storage medium storing program code, wherein the above-described method is executed when the program code is run by a starter controller.

[0008] Fifthly, this application provides a computer program product comprising computer instructions stored in a computer-readable storage medium; the computer program product is used to implement the above-described method.

[0009] This application provides a data processing method, apparatus, and electronic device. In the method, after obtaining first input data, second input data can be obtained through a first recognition result and the first input data. Then, corresponding feedback content can be obtained through the second input data. Since the second input data can more accurately express the user's information acquisition intention, the obtained feedback content can be more accurate. Furthermore, the first recognition result used to obtain the second input data is obtained when acquiring the feedback content of historical input data, thus eliminating the need to obtain it through real-time recognition after acquiring the first input data, which also helps to improve the efficiency of obtaining feedback content. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 A schematic diagram of an operating environment for the data processing method proposed in an embodiment of this application is shown;

[0012] Figure 2 A schematic diagram of another operating environment for the data processing method proposed in the embodiments of this application is shown;

[0013] Figure 3 A flowchart of a data processing method proposed in an embodiment of this application is shown;

[0014] Figure 4 This illustration shows a schematic diagram of the first input data and the corresponding associated data in an embodiment of this application;

[0015] Figure 5 A schematic diagram of multiple input data in an embodiment of this application is shown;

[0016] Figure 6 A flowchart of a data processing method according to another embodiment of this application is shown;

[0017] Figure 7 A flowchart of a data processing method according to another embodiment of this application is shown;

[0018] Figure 8A flowchart of a data processing method according to another embodiment of this application is shown;

[0019] Figure 9 A flowchart of a data processing method according to another embodiment of this application is shown;

[0020] Figure 10 This paper shows a structural block diagram of a data processing apparatus according to an embodiment of the present application;

[0021] Figure 11 This invention illustrates a structural block diagram of an electronic device for performing a data processing method according to an embodiment of the invention;

[0022] Figure 12 This is a storage unit in this application embodiment for storing or carrying program code that implements the data processing method according to this application embodiment. Detailed Implementation

[0023] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0024] In human-computer dialogue scenarios, dialogue systems can provide corresponding responses based on user input. This interaction method greatly improves the efficiency and convenience of information acquisition. For example, if a user inputs "What will the weather be like tomorrow?" in a human-computer dialogue scenario, the dialogue system can parse this request and provide detailed weather information, including temperature, humidity, and whether it will rain. However, despite the significant progress made in dialogue systems, there is still room for improvement in the efficiency and accuracy of their responses in practical applications.

[0025] Therefore, the inventors have proposed a data processing method, apparatus, and electronic device that can improve the above-mentioned problems. In this method, after obtaining the first input data, the second input data can be obtained through the first recognition result and the first input data, and then the corresponding feedback content can be obtained through the second input data. Since the second input data can more accurately express the user's information acquisition intention, the obtained feedback content can be more accurate. Furthermore, the first recognition result used to obtain the second input data is obtained when obtaining the feedback content of historical input data, so it is not necessary to obtain it through real-time recognition after obtaining the first input data, which also helps to improve the efficiency of obtaining feedback content.

[0026] The application scenarios involved in the embodiments of this application will be introduced below.

[0027] In the embodiments of this application, the provided data processing method can be executed by an electronic device. In this manner, all steps of the data processing method provided in the embodiments of this application can be executed by the electronic device. For example, as Figure 1 As shown, all steps in the data processing method provided in this application embodiment can be executed by the voice assistant in the electronic device 100.

[0028] Alternatively, the data processing method provided in this application embodiment can also be executed by a server. Correspondingly, in this server-executed method, the server can begin executing the steps of the data processing method provided in this application embodiment in response to a triggering instruction. This triggering instruction can be sent by an electronic device used by a user, or it can be triggered locally by the server in response to some automated event.

[0029] Furthermore, the data processing method provided in this application embodiment can also be executed collaboratively by an electronic device and a server. In this method, some steps of the data processing method provided in this application embodiment are executed by the electronic device, while other steps are executed by the server. For example, as shown... Figure 2 As shown, the electronic device 100 can perform a data processing method including: obtaining first input data. Then, the electronic device 100 transmits the first input data to the server 200. After receiving the first input data, the server 200 can perform subsequent steps to obtain feedback content. After obtaining the feedback content, the server 200 can return the feedback content to the electronic device 100 for display.

[0030] It should be noted that in this method where electronic devices and servers work together, the steps performed by the electronic devices and servers are not limited to those described in the examples above. In practical applications, the steps performed by the electronic devices and servers can be dynamically adjusted according to the actual situation.

[0031] It should be noted that the electronic equipment 100, in addition to being for Figure 1 and Figure 2Besides smartphones, the device can also be a tablet, wearable device, etc. Server 200 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as an artificial intelligence platform. In the case where the data processing method provided in this embodiment is executed by a server cluster or distributed system composed of multiple physical servers, different steps in the data processing method can be executed by different physical servers, or can be executed in a distributed manner by servers built on a distributed system.

[0032] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0033] Please see Figure 3 This application provides a data processing method, which includes:

[0034] S110: Obtain the first input data.

[0035] In this embodiment of the application, the first input data is the data currently input by the user. Correspondingly, the first input data can also be understood as data representing the user's current operational needs.

[0036] The first input data can fall into several categories. For example, it can be instruction-type input data or question-and-answer type input data. Instruction-type input data is used to control the device. For example, it could include commands like "turn on Bluetooth," "turn on the speaker," or "start WiFi." Question-and-answer type input data is used to request information from the device. For example, it could include questions like "What's the weather like today?", "Who is this person?", "Where are we?", or "Please recommend some dishes at this restaurant."

[0037] Optionally, the first input data can be text input data, or the first input data can also be voice input data.

[0038] S120: Obtain second input data through the first recognition result and the first input data, wherein the first recognition result is used to update part of the content in the first input data, the first recognition result is obtained when obtaining the feedback content of historical input data, and the historical input data is the input data before the first input data.

[0039] As mentioned above, the categories and content of the first input data can be diverse. Therefore, in some cases, the obtained first input data can clearly express the user's intent. For example, if the first input data is "What's the weather like today?", it can be clearly determined that the user wants to know today's weather. However, in other cases, the obtained first input data may not be clear enough in expressing intent, thus making it impossible to clearly determine the user's actual intent. For example, the obtained first input data could be "What albums has this person released?", where the expression "this person" does not actually determine who "this person" is, thus making it impossible to determine the object the user wants to query. Therefore, in this application, when it is determined that the first input data cannot clearly express the user's intent, the first input data can be updated using the first recognition result.

[0040] As another approach, the input data can also have corresponding associated data. In the embodiments of this application, the associated data corresponding to the input data can be understood as data related to the intent expressed by the input data. For example, when the first input data is question-and-answer type input data, the associated data corresponding to the first input data represents the question-and-answer object of the first input data. For example, when the first input data is instruction type input data, the associated data corresponding to the first input data represents the control object of the first input data.

[0041] For example, such as Figure 4 As shown, in Figure 4 The electronic device displays an image 10, and then the user enters "What delicious food is available in this place?" in the input control 20 of the electronic device. In this case, the user's input "What delicious food is available in this place?" can be identified as the first input data, and the image 10 can be the associated data of the first input data.

[0042] In this embodiment, the associated data of the first input data may be input together with the first input data, or it may be input before the first input data. For example, before inputting the first input data, the user has already input other input data, and while inputting the other input data, associated data is also input simultaneously. This associated data may correspond not only to the other input data, but also to the first input data input later. In other words, the first recognition result can be understood as the recognition of associated data corresponding to historical input data, and the associated data of the historical input data is the same as the associated data of the first input data obtained.

[0043] For example, such as Figure 5 As shown, in the display Figure 10In this case, the initial input data is the user's question 1. That is, when question 1 is entered, the corresponding related data (i.e., ...) has already been input. Figure 10 Subsequently, the user entered question 2. When receiving question 2, it can be used as the first input data. However, the associated data corresponding to question 2 can be image 10, which was already entered when question 1 was input. In other words, the associated data for both question 1 and question 2 are... Figure 10 .exist Figure 5 In the example shown, the electronic device has already recognized image 10 when it responds to the user's input question Q1, and it stores the recognition result as the first recognition result. For example, the first recognition result obtained from recognizing image 10 is: Image 10 is Shanghai.

[0044] As one approach, upon obtaining the first input data, it can be determined whether the intent expressed by the first input data is clear. If the intent expressed by the first input data is unclear, a first recognition result can be obtained, and the first input data can be updated using this first recognition result to obtain second input data with a clear intent. For example, in conjunction with the aforementioned... Figure 5 For example, when question Q2 is obtained, the first identification result can be directly used to update question Q2. For instance, the first identification result includes... Figure 10 Representing Shanghai, the content of question Q2 can be updated from "What food is available in this place?" to "What food is available in Shanghai?" The updated "What food is available in Shanghai?" can be understood as the second input data.

[0045] S130: Obtain the corresponding feedback content through the second input data.

[0046] Since the second input data is explicitly targeted, the corresponding feedback can be obtained directly from it. For example, based on the aforementioned... Figure 5 In the example shown, with question Q2 as the first input data, the second input data can be "What are some delicious foods in Shanghai?", and the feedback can be "There are pan-fried buns, fresh meat mooncakes, potstickers, etc."

[0047] In the embodiments of this application, there can be multiple ways to obtain feedback content.

[0048] One approach is to use a large language model to obtain feedback content corresponding to the second input data. In this method, the second input data is fed into the large language model, which then obtains the corresponding feedback content based on its capabilities. It's important to note that a large language model is a language model built from a deep neural network with numerous parameters. It can be trained using self-supervised learning methods on a large amount of unlabeled text, enabling it to predict and generate text and other content through training on large datasets. One of its core functions is its generative ability, which allows it to automatically generate content based on the input. Furthermore, through deep learning and training on large amounts of text data, the large language model can quickly understand and generate natural language text, allowing it to obtain feedback content corresponding to the input content (the content fed into the large language model) in a relatively short time.

[0049] In one scenario, multiple large language models can be pre-configured. After receiving the second input data, a suitable target large language model can be selected from these models based on the second input data. This allows for the generation of corresponding feedback content using the second input data and the target large language model. Optionally, the multiple large language models may include a large language model for educational question answering, a solution supporting document question answering, a document question answering large language model with built-in ultra-long input, a multimodal question answering large language model with search functionality, and a large language model supporting question answering across various vertical domains.

[0050] Alternatively, after obtaining the second input data, one can directly retrieve corresponding feedback content through a web search. In this method, the second input data can be entered into a search engine to obtain the relevant feedback.

[0051] As a method, the feedback content is subjected to security verification. If the security verification is passed, the feedback content is displayed.

[0052] The purpose of security verification includes checking the generated feedback content to determine whether it needs to be blocked, thus avoiding negative impacts from errors in the feedback content. Once the security verification is passed, the feedback content can be output and / or stored. Outputting the feedback content can be understood as displaying it.

[0053] It should be noted that when obtaining feedback content through a large language model, some potentially security-hazardous content may be output due to the model's limitations. This potentially security-hazardous content can include discriminatory language, privacy violations, or content that violates social ethics. In such cases, if the feedback content does not contain any potentially security-hazardous content, it is determined that the feedback content has passed security verification. If potentially security-hazardous content is detected in the feedback content, it is determined that the feedback content has failed security verification. In cases where the feedback content is incorrect, it can be regenerated using a second input data.

[0054] This embodiment provides a data processing method that makes the obtained feedback content more accurate. Furthermore, the first recognition result used to obtain the second input data is obtained when acquiring the feedback content of historical input data, thus eliminating the need to obtain it through real-time recognition after acquiring the first input data, which also helps to improve the efficiency of obtaining feedback content.

[0055] Please see Figure 6 This application provides a data processing method, which includes:

[0056] S210: Obtain the first input data.

[0057] S220: If the first identification result contains target content corresponding to the first input data, then the first input data is updated using the target content to obtain the second input data. The first identification result is used to update part of the content in the first input data. The first identification result is obtained when obtaining the feedback content of historical input data. The historical input data is the input data before the first input data.

[0058] As described above, in this embodiment, the content included in the first identification result may be varied. In this case, in order to determine whether there is content in the first identification result that can be used to update the first input data, it is possible to first detect whether there is target content in the first identification result that corresponds to the first input data. This target content can be understood as content that can be used to update a portion of the content in the first input data.

[0059] One approach is to first detect the part to be updated in the first input data after obtaining it, and then, based on the content of the part to be updated, detect whether there is corresponding target content in the first recognition result. The part to be updated can be a part whose semantic expression is not clear enough. For example, the part to be updated can be a personal pronoun (e.g., I, he, etc.), a location pronoun (e.g., there, this place, that place), etc.

[0060] Optionally, a content library can be pre-established, which can record content that needs to be updated. In this case, after receiving the first input data, it can be first checked whether the first input data includes content from the content library. If it does, the included content can be identified as the part to be updated. For example, the content recorded in the content library can be shown in the following table:

[0061] ID content 1 AAA 2 BBB 3 CCC

[0062] Based on the table above, if the content of the first input data is "AAAXXXYYY", it can be determined that "AAA" in the first input data is the content in the content library, and thus "AAA" is the part to be updated. Therefore, the corresponding target content can be obtained from the first recognition result to replace "AAA".

[0063] In one approach, according to this embodiment, the identification results of all historical input data and their associated data can be centrally stored. During the acquisition of target content, it may be detected that multiple contents within the first identification result can be used as target content. For example, if the first input data is "What kind of food is available in this place?", it can be determined that "this place" belongs to the part to be updated. In some cases, it can be detected that contents such as "Shanghai," "Chengdu," and "Shenzhen" included in the first identification result can all be used to update "this place." Optionally, if it is detected that multiple contents within the first identification result can be used to update the part to be updated in the first input data, then the content whose generation time is closest to the current time among these multiple contents can be used as the target content.

[0064] In one approach, as described in this embodiment, the identification results of the same associated data can be stored separately, or each identification result can be stored in relation to its corresponding associated data. In this approach, during the acquisition of the target content, the associated data corresponding to the first input data (e.g., the first associated data) can be determined first, and then the target content can be obtained from the identification results corresponding to the first associated data included in the first identification result.

[0065] In one approach, in this embodiment of the application, the recognition result of only the most recently input associated data can be stored. In this case, the first recognition result can be understood as the recognition result of the most recently input associated data, and the first associated data corresponding to the first input data is also the most recently input associated data. For example, please refer to [the relevant documentation]. Figure 5When the user inputs question Q1, they have already input image 10. To obtain feedback on question Q1, image 10 is recognized, and the recognition result is stored. The recognition result of image 10 can be understood as the first recognition result. When the user inputs question Q2, question Q2 can be used as the first input data, and the first associated data corresponding to question Q2 is still image 10.

[0066] S221: Obtain the corresponding feedback content through the second input data.

[0067] S230: If the target content is not found in the first identification result or if the first identification result does not exist, the first associated data is identified to obtain the second identification result, wherein the first associated data corresponds to the first input data.

[0068] In this case, the user may not have entered any other input data before entering the first input data. This means that there is no historical input data for the first input data, and therefore, there may be no first recognition result. For an example, please refer to [link to previous document]. Figure 5 When a user inputs question Q1, that is the first input data. However, the user has not input any other questions before Q1, so there is no historical input data for Q1, and therefore, there may be no first recognition result. But when the user subsequently inputs question Q2, because question Q1 has already been input, a first recognition result already exists when Q2 is used as the first input data.

[0069] S231: Using the second recognition result and the first input data, a third input data is obtained, and the second recognition result is used to update part of the content in the first input data.

[0070] S232: The feedback content is obtained through the third input data.

[0071] In one approach, the first input data includes a question entered by the user, the first recognition result is obtained by recognizing second associated data, the second associated data corresponds to the historical input data, and the second associated data is an image or a document.

[0072] This embodiment provides a data processing method that makes the obtained feedback content more accurate. Furthermore, the first recognition result used to obtain the second input data is obtained when acquiring the feedback content of historical input data, thus eliminating the need for real-time recognition after acquiring the first input data, thereby improving the efficiency of obtaining feedback content. In this embodiment, after obtaining the first input data, it first checks whether there is target content corresponding to the first input data in the first recognition result. If target content exists, it is directly used to update the first input data. If the target content does not exist, the second recognition result is obtained by recognizing the first associated data, which in turn yields the third input data, making the data processing method provided by this application more flexible.

[0073] Please see Figure 7 This application provides a data processing method, which includes:

[0074] S310: Obtain the first input data.

[0075] S320: If the first input data is question-and-answer type input data, then the second input data is obtained through the first recognition result and the first input data, wherein the first recognition result is used to update part of the content in the first input data, and the first recognition result is obtained when obtaining the feedback content of historical input data, wherein the historical input data is the input data before the first input data.

[0076] S321: Obtain the corresponding feedback content through the second input data.

[0077] S330: If the first input data is instruction type input data, then execute the control instruction corresponding to the first input data.

[0078] In this embodiment of the application, the category of the first input data can be determined by the semantics of the first input data.

[0079] This embodiment provides a data processing method that makes the obtained feedback more accurate. Furthermore, the first recognition result used to obtain the second input data is obtained when acquiring the feedback content of historical input data, thus eliminating the need for real-time recognition after acquiring the first input data, thereby improving the efficiency of obtaining feedback content. In this embodiment, after obtaining the first input data, the category of the first input data can be determined first, and different processing methods will correspond to different categories of first input data.

[0080] Please see Figure 8This application provides a data processing method, which includes:

[0081] S410: Display the first associated data.

[0082] In different situations, the method for determining the first associated data can be different.

[0083] In one scenario, the currently captured preview image of the electronic device can be used as the first associated data. Specifically, after the image acquisition device of the electronic device is activated, it can acquire images in real time and display the real-time captured preview image in the currently displayed image preview interface. Therefore, the first associated data can be the currently displayed preview image in the image preview interface. Optionally, when displaying the real-time captured preview image of the electronic device, the currently displayed preview image can be used as the first associated data in response to a first preset operation. This first preset operation can be a gesture operation on the screen. The gesture operation can be a pressing operation, a swiping operation, etc. Alternatively, the first preset operation can be an operation on a button in the electronic device. This button can be a virtual button or a physical button. The physical button can be a power button, a volume up button, a volume down button, or an image capture button, etc. For example, when the electronic device is equipped with a physical image capture button, the first preset operation can be a first pressing operation on that image capture button. Furthermore, when a second pressing operation on the image capture button is detected, a shooting operation can be triggered. The difference between the first and second pressing operations lies in the degree of pressure applied to the case for image capture. This degree of pressure can include the intensity of the pressure applied and the duration of the pressure.

[0084] In one scenario, an image selected by the user from a photo album application can be used as the first associated data. After the photo album application is launched on the electronic device, the images stored on the device can be viewed through it. Upon launch, the photo album application can display the stored images as thumbnails. When images are displayed as thumbnails, if an image is selected by the user (e.g., clicked), the selected image can be enlarged and displayed (e.g., in full-screen mode). Optionally, when an image is enlarged and displayed, the enlarged image can be used as the first associated data. Optionally, when an image is enlarged and displayed, the enlarged image can be used as the first image in response to a second preset operation. This second preset operation can be a gesture operation on the screen, such as a press or swipe. Alternatively, the second preset operation can be an operation on a button on the electronic device, which can be a virtual button or a physical button. The physical button can be a power button, volume up button, volume down button, or image capture button, etc.

[0085] In one scenario, the captured image can be used as the first image. After the image acquisition device of the electronic device is activated, if a shooting operation is triggered, the electronic device can store the image captured at the time of the shooting operation in the photo album. Optionally, when the electronic device triggers a shooting operation, in addition to storing the image captured at the time of the shooting operation, it can also display the image captured at the time of the shooting operation. Optionally, if the electronic device is configured with a preset shooting mode, when this preset shooting mode is activated, if a shooting operation is performed, the captured image will be stored and displayed synchronously. Therefore, if the preset shooting mode is activated, the image displayed after the shooting operation can be used as the first associated data.

[0086] S420: Obtain first input data, which corresponds to the first associated data.

[0087] When displaying the first associated data, the user can input data related to the first associated data as the first input data. The first input data may include text data entered by the user, or it may include voice data entered by the user.

[0088] S430: Obtain second input data through the first recognition result and the first input data, wherein the first recognition result is used to update part of the content in the first input data, the first recognition result is obtained when obtaining the feedback content of historical input data, and the historical input data is the input data before the first input data after the first associated data is displayed.

[0089] The first identification result is obtained when the feedback content of the historical input data is acquired. This can be understood as follows: after the first associated data is displayed, the user first inputs the historical input data. In order to obtain the feedback content related to the historical input data, the first associated data is identified and the identification result is stored.

[0090] S440: Obtain the corresponding feedback content through the second input data.

[0091] The method provided in this application embodiment can be executed by an application running in an electronic device. For example, the application can be a voice assistant. Optionally, the structural diagram of the application can be as follows: Figure 9 As shown. Figure 9 As shown, the application can be divided into the following modules: Modal Understanding Layer, Operational Intervention Layer, Function Call Layer, Task Orchestration Layer, Multimodal Dialogue Management Layer, Security Verification Layer, Service Storage Layer, and Terminal Output Layer. The information flow between these modules can be referenced... Figure 9 The direction indicated by the middle arrow.

[0092] The terminal input layer supports users to input data in various interactive formats, such as voice, plain text, preview images, photos, documents, and other unstructured data from multiple scenarios. After the user inputs data, the terminal input layer can uniformly convert it into structured data, making it easier for downstream modules to understand and process it.

[0093] The modality-based understanding layer can invoke different understanding modules to interpret the associated data based on the modality of the user-input data. For example, when the associated data is an image, the image understanding module can be invoked for image understanding. Image understanding can include extracting text information, subject categories, image classification results, and image descriptions from the image. Similarly, when the associated data is a document, the document understanding module can be invoked for document understanding, which can include document parsing, segmentation, and feature extraction. The modality-based understanding layer can then pass the understood information to downstream modules, facilitating their understanding of the module information and recording it in the multimodal dialogue management layer.

[0094] The operational intervention layer is designed to quickly address operational needs and bad cases at each stage of the execution chain. It bypasses individual service modules and directly outputs results to the final response side. The operational intervention layer can also be understood as the platform primarily used by product and operations teams for the rapid and stable implementation of product marketing scenarios, directly specifying the end-to-end response module through intervention.

[0095] The function call layer is used to understand the application programming interface (API) corresponding to the current user dialogue. For example, for "turn on Bluetooth," the instruction module generates the instruction `enable_bluetooth`, which is then executed by the application execution layer. For multimodal question-and-answer questions like "What are this person's works?", the instruction module generates the instruction `get_qa_response`, which is then distributed to the task decomposition and orchestration layer for execution by the agent. In other words, the function call layer can determine the category of the first input data from the user. For example, it can determine whether the first input data is a complex task (e.g., question-and-answer) or an application execution task (e.g., instruction). The instruction result is written to the multi-talk management layer and then passed to downstream modules. Optionally, the function call layer can be implemented using a large model's function call (FC).

[0096] The task decomposition and orchestration layer can decompose and orchestrate tasks by combining information recorded in the multimodal dialogue management layer. For example, if a user inputs question Q1: "What are this person's works?", and if the modal understanding layer does not have a specific name, the task decomposition and orchestration layer will decompose question Q1 into two steps. The first step is "Who is this person?", which can then call the multimodel module to obtain the name, for example, Zhang San. The second step is to rewrite the user's original question Q1 as "What are Zhang San's works?", and can call bot-llm-agent to answer the question. All this intermediate information is recorded in the multimodal dialogue layer. If the user inputs question Q2: "Who is his wife?", the task decomposition and orchestration layer, combining information from the multimodal dialogue management layer, will decompose question Q2 into one step: rewrite the user's original question Q2 as "Who is Zhang San's wife?", and call bot-llm-agent to answer the question. Figure 9 As shown, the task decomposition and orchestration layer currently supports the following Agents: bot-education-plugin – for educational Q&A, it integrates a third-party interface specifically for educational Q&A; bot-doc-plugin – a self-built solution supporting document Q&A, with an internal large-scale document Q&A model for extremely long inputs; multimodel – a large-scale multimodal Q&A model with search capabilities; bot-llm-agent – ​​a self-built large-scale model solution supporting Q&A across various vertical domains: tourism, gaming, personalization, general branding, and general Q&A; and Tools Function Call – used to support multi-task question orchestration for execution (control) and question-answering tasks.

[0097] The multimodal dialogue management layer is used to record various events that occur on the link (and the various results generated by the services of each module on the link). For example, after a user captures a picture through an electronic device, the input data is "What are the local delicacies?", and the electronic device displays the feedback content "pan-fried buns, xiaolongbao, and fresh meat mooncakes".

[0098] In this scenario, the information recorded in the multimodal dialogue management layer includes:

[0099] user_requset: "What are some good foods to eat in this area?"

[0100] assistant_request:{“Tools Function Call”:“What are some good local dishes?”}

[0101] `assitant_response:{"get_cui_response"--screen recognition command}`

[0102] image_upload: {"User uploaded an image"}

[0103] image_request: {"Get image information"}

[0104] image_response: {Image text, main information; Image classification result;}

[0105] assistant_request:{“Agent&Planning”:“What are some good restaurants in this area?”}

[0106] assistant_response:{

[0107] “Agent&Planning”:{

[0108] "query": "What are some good foods to eat in this area?"

[0109] “agent”:”multimodel”,

[0110] “show_type”:False

[0111] }}

[0112] assistant_request:{“multimodel”:“What are some good local dishes?”}

[0113] assitant_response:{“Shanghai}

[0114] assistant_request:{“Agent&Planning”:“What are some good restaurants in Shanghai?”}

[0115] assistant_response:{

[0116] “Agent&Planning”:{

[0117] The query is: "What are some good foods in Shanghai?".

[0118] "agent":"bot-lllm-agent",

[0119] “show_type”:True

[0120] }}

[0121] assistant_request:{“bot-llm-agent”:”What are some delicious foods in Shanghai?”}

[0122] `assitant_response:{"pan-fried buns, xiaolongbao, fresh meat mooncakes, etc.}"`

[0123] Optionally, the multimodal dialogue management layer can dynamically record the most recent N rounds of dialogue (dialogue between the user and the application), as well as details of each event that occurred along the chain, where N can be 4, 5, 6, etc. Each module along the chain can read the content in the multimodal dialogue management layer.

[0124] The security verification layer primarily performs security checks on user requests (e.g., initial input data) and generated content (e.g., feedback content) to determine whether interception is necessary. This prevents negative impacts from errors or illusions in the large language model when feedback is obtained. Once security verification is passed, data can be stored through the service storage layer. The service storage layer primarily stores or caches some of the user's personalized information. The terminal output layer mainly streams the generated and verified feedback content to the screen and broadcasts it via voice.

[0125] based on Figure 9The method provided in this application embodiment, as shown, may include: obtaining first input data through a terminal input layer; obtaining second input data through a task decomposition and orchestration layer, a first recognition result, and the first input data, wherein the first recognition result is used to update a portion of the first input data, and the first recognition result is obtained when acquiring feedback content of historical input data, wherein the historical input data is input data prior to the first input data, and the first recognition result is obtained through a modal understanding layer. Corresponding feedback content is obtained through the task decomposition and orchestration layer and the second input data, and the feedback content is output through a terminal output layer.

[0126] based on Figure 9 The content shown is:

[0127] By designing a multimodal dialogue management layer to record events occurring on the blockchain, the advantages include:

[0128] 1. **Preservation of Multimodal Information:** The solution records text, images, and related events and results, ensuring the integrity of multimodal information. 2. **Focus on Inter-module Collaboration:** Each layer can access the content in the multimodal dialogue management layer, facilitating collaboration and information sharing between different dialogue rounds. 3. **Dynamic Recording:** The multimodal dialogue management layer dynamically records multiple rounds of dialogue, helping to capture changes in user needs and contextual information. 4. **Enhanced Dialogue Understanding and Precise Responses:** By recording multimodal information and various events along the dialogue path, the assistant can better understand the conversation and provide precise responses to user needs.

[0129] In addition, by designing a task decomposition and orchestration layer to handle complex dialogue requests, the advantages include:

[0130] 1. Modularity: Abstracting Agent sets facilitates modular design, improving decoupling and replaceability between components. 2. Consistent Interface: By decomposing and orchestrating tasks, interaction with other layers can be achieved through a unified interface, simplifying the integration process. 3. Ease of Extensibility: Abstracting functionality into Agent sets makes adding new features or updating existing ones easier, allowing the assistant to adapt to new needs more quickly. 4. Maintainability: Abstracting into Agent sets improves the maintainability of the voice assistant; independent updates or repairs to Agent sets reduce the impact on the entire system. 5. Multi-Scenario Adaptability: Through agent invocation, the voice assistant's capabilities can be flexibly adjusted and extended in different scenarios, achieving multi-scenario adaptation.

[0131] Please see Figure 10 This application provides a data processing apparatus 500, which includes:

[0132] The data acquisition unit 510 is used to obtain the first input data.

[0133] The data processing unit 520 is used to obtain second input data through the first recognition result and the first input data, wherein the first recognition result is used to update part of the content in the first input data, and the first recognition result is obtained when obtaining the feedback content of historical input data, wherein the historical input data is the input data before the first input data.

[0134] Feedback unit 530 is used to obtain corresponding feedback content through the second input data.

[0135] In one approach, the data processing unit 520 is specifically configured to update the first input data using the target content if the first identification result contains target content corresponding to the first input data, thereby obtaining second input data. Optionally, the data processing unit 520 is specifically configured to identify first associated data to obtain a second identification result if the first identification result does not contain the target content or if the first identification result does not exist, wherein the first associated data corresponds to the first input data; and obtain third input data using the second identification result and the first input data, wherein the second identification result is used to update a portion of the content in the first input data. In this case, the feedback unit 530 is configured to obtain feedback content using the third input data.

[0136] In one manner, the data processing unit 520 is specifically configured to, if the first input data is question-and-answer type input data, obtain second input data through the first recognition result and the first input data; if the first input data is instruction type input data, execute the control instruction corresponding to the first input data.

[0137] In one approach, the feedback unit 530 is specifically used to obtain corresponding feedback content through the second input data and the target large language model, wherein the target large language model is determined through the second input data.

[0138] As one method, the feedback unit 530 is specifically used to perform security verification on the feedback content; if the security verification is passed, the current feedback content is displayed.

[0139] The data processing device provided in this embodiment enables the obtained feedback content to be more accurate. Furthermore, the first recognition result used to obtain the second input data is obtained when acquiring the feedback content of historical input data, thus eliminating the need to obtain it through real-time recognition after acquiring the first input data, which also helps to improve the efficiency of obtaining feedback content.

[0140] It should be noted that the device embodiments in this application correspond to the aforementioned method embodiments. The specific principles in the device embodiments can be found in the content of the aforementioned method embodiments, and will not be repeated here.

[0141] The following will combine Figure 11 This application describes an electronic device.

[0142] Please see Figure 11 Based on the aforementioned data processing method and apparatus, this application embodiment also provides another electronic device 1000 capable of executing the aforementioned data processing method. The electronic device 1000 includes one or more (only one shown in the figure) processors 102, a memory 104, and a network module 106 coupled together. The memory 104 stores programs capable of executing the contents of the aforementioned embodiments, and the processors 102 can execute the programs stored in the memory 104.

[0143] The processor 102 may include one or more processing cores. The processor 102 connects to various parts within the electronic device 1000 using various interfaces and lines, and performs various functions and processes data of the electronic device 1000 by running or executing instructions, programs, code sets, or instruction sets stored in the memory 104, and by calling data stored in the memory 104. Optionally, the processor 102 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA), or the processor 102 may be an application processor. The processor 102 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 102, but may be implemented separately using a communication chip.

[0144] The memory 104 may include random access memory (RAM) or read-only memory (ROM). The memory 104 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 104 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), and instructions for implementing the various method embodiments described below.

[0145] The network module 106 is used to enable information interaction between the electronic device 1000 and other devices, such as transmitting device control commands, manipulation request commands, and status information acquisition commands. However, the network module 106 may differ depending on the specific device being used.

[0146] In another case, the electronic device 1000 may also include a screen, an image acquisition device, a sensor module, and an audio acquisition device.

[0147] The screen can be used for displaying images. For example, when a game program is running, it can be used to display the game screen corresponding to the game program. In one approach, the electronic device 1000 can take screenshots of multiple frames continuously displayed on the screen to obtain the multiple frames needed to detect whether the electronic device 100 is displaying an event animation.

[0148] An image acquisition device can be used for image acquisition, and the preview image acquired by the image acquisition device in real time can be displayed on a screen. The displayed preview image can be used as associated data (e.g., first associated data or second associated data) in the embodiments of this application.

[0149] The sensor module may include at least one sensor. Specifically, the sensor module may include, but is not limited to, light sensors, motion sensors, pressure sensors, infrared thermal sensors, distance sensors, accelerometers, and other sensors. Among them, the pressure sensor is a sensor that can detect pressure generated by pressing on the electronic device 1000. That is, the pressure sensor detects pressure generated by contact or pressing between the user and the audio playback device, such as pressure generated by contact or pressing between the user's ear and the mobile terminal. Therefore, the pressure sensor can be used to determine whether contact or pressing has occurred between the user and the electronic device 1000, and the magnitude of the pressure. The accelerometer can detect the magnitude of acceleration in various directions (generally three axes), and when stationary, it can detect the magnitude and direction of gravity. It can be used for applications that identify the posture of the electronic device 1000 (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition-related functions (such as pedometers, taps), etc. In addition, the electronic device 1000 may also be equipped with other sensors such as gyroscopes, barometers, hygrometers, and thermometers, which will not be elaborated here.

[0150] An audio acquisition device is used to acquire audio signals. Optionally, the audio acquisition device may include multiple audio acquisition components. These audio acquisition components may be microphones.

[0151] In one approach, the network module of the electronic device 1000 is a radio frequency (RF) module. This RF module is used to receive and transmit electromagnetic waves, converting electromagnetic waves into electrical signals, thereby enabling communication with a communication network or other devices. The RF module may include various existing circuit elements used to perform these functions, such as antennas, RF transceivers, digital signal processors, encryption / decryption chips, user identity module (SIM) cards, memory, etc. For example, the RF module can interact with external devices by transmitting or receiving electromagnetic waves, thereby receiving audio signals transmitted by external devices.

[0152] Please refer to Figure 12 This diagram illustrates a structural block diagram of a computer-readable storage medium provided in an embodiment of this application. The computer-readable medium 1100 stores program code that can be called by a processor to execute the methods described in the above method embodiments.

[0153] The computer-readable storage medium 1100 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Optionally, the computer-readable storage medium 1100 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 1100 has storage space for program code 1110 that performs any of the method steps described above. This program code can be read from or written to one or more computer program products. The program code 1110 may, for example, be compressed in a suitable form.

[0154] In summary, the data processing method, apparatus, and electronic device provided in this application, in which, after obtaining the first input data, a second input data can be obtained through the first recognition result and the first input data, and then the corresponding feedback content can be obtained through the second input data. In this method, because the second input data can more accurately express the user's information acquisition intention, the obtained feedback content can be more accurate. Furthermore, the first recognition result used to obtain the second input data is obtained when obtaining the feedback content of historical input data, thus eliminating the need to obtain it through real-time recognition after obtaining the first input data, which also helps to improve the efficiency of obtaining feedback content.

[0155] In the case where the data processing method is executed by the voice assistant, this solution can improve the accuracy and efficiency of the voice assistant in multimodal question answering, solve the defects of repetitive arrangement for complex question answering tasks for users, and reduce the cost of multimodal question answering by the voice assistant.

[0156] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical feature diagrams. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A data processing method, characterized in that, The method includes: Obtain the first input data; The second input data is obtained by using the first recognition result and the first input data, wherein the first recognition result is used to update part of the content in the first input data, and the first recognition result is obtained when obtaining the feedback content of historical input data, wherein the historical input data is the input data before the first input data; The corresponding feedback content is obtained by using the second input data.

2. The method according to claim 1, characterized in that, The process of obtaining the second input data using the first recognition result and the first input data includes: If the first recognition result contains target content corresponding to the first input data, then the first input data is updated using the target content to obtain the second input data.

3. The method according to claim 2, characterized in that, The method further includes: If the target content is not found in the first identification result or if the first identification result does not exist, the first associated data is identified to obtain the second identification result, wherein the first associated data corresponds to the first input data; The third input data is obtained by combining the second recognition result and the first input data, and the second recognition result is used to update part of the content in the first input data. Feedback content is obtained through the third input data.

4. The method according to claim 1, characterized in that, The process of obtaining the second input data using the first recognition result and the first input data includes: If the first input data is question-and-answer type input data, then the second input data is obtained through the first recognition result and the first input data; If the first input data is instruction type input data, then the control instruction corresponding to the first input data is executed.

5. The method according to claim 1, characterized in that, The first input data includes a question entered by the user, and the first recognition result is obtained by recognizing the second associated data, which corresponds to the historical input data. The second associated data is an image or a document.

6. The method according to claim 1, characterized in that, The step of obtaining the corresponding feedback content through the second input data includes: The corresponding feedback content is obtained by using the second input data and the target large language model, wherein the target large language model is determined by the second input data.

7. The method according to any one of claims 1-6, characterized in that, The method further includes: The feedback content is then subjected to security verification. If the security verification is successful, the feedback content will be displayed.

8. A data processing apparatus, characterized in that, The device includes: The data acquisition unit is used to obtain the first input data; A data processing unit is configured to obtain second input data through a first identification result and the first input data, wherein the first identification result is used to update a portion of the first input data, and the first identification result is obtained when acquiring feedback content of historical input data, wherein the historical input data is input data prior to the first input data; The feedback unit is used to obtain corresponding feedback content through the second input data.

9. An electronic device, characterized in that, Including the processor and memory; One or more programs are stored in the memory and configured to be executed by the processor to implement the method of any one of claims 1-7.

10. A computer program product, characterized in that, The computer program product includes computer instructions stored in a computer-readable storage medium; the computer program product is used to implement the method as described in any one of claims 1-7.

Images